A utility for running external programs
A utility for running external programs, built on UIOP.
Cmd is designed to:
- Be natural to use.
- Protect against shell interpolation.
- Be usable from multi-threaded programs.
- Support Windows.
cmd are never passed to a shell for interpretation.
Arguments are usually:
- lists of strings and keywords
Some other types get special handling. Nested lists are not allowed.
Arguments are handled as follows:
A string is tokenized (using cl-shlex) and added to the list of arguments.
(cmd "ls -al") ≡ (uiop:run-program '("ls" "-al")) (cmd "echo 'hello world'") ≡ (uiop:run-program '("echo" "hello world"))
Redirection operators in the tokenized string (such as
|) are translated into keywords (see below).
(cmd "echo 'hello world' > myfile") ≡ (cmd '("echo" "hello world" :> "myfile")) ≡ (uiop:run-program '("echo" "hello world") :output "myfile")
A list is added directly to the list of arguments (not tokenized). (Putting a string in a list is “escaping” it.)
(cmd "bash -c 'exit 1'") ≡ (cmd "bash -c" '("exit 1"))
Keywords in the list are treated exactly like keywords as arguments.
Keywords that are subcommand dividers (like
|) are handled internally by
cmd. Otherwise, a keyword, along with the next value, is used as a keyword argument to UIOP.
(cmd "bash -c 'exit 1'" :ignore-error-status t) ≡ (cmd :ignore-error-status t "bash -c 'exit 1'") ≡ (cmd "bash -c" :ignore-error-status t '("exit 1")) ≡ (uiop:run-program '("bash" "-c" "exit 1") :ignore-error-status t)
Note that unlike normal Lisp functions, keyword arguments can appear anywhere, not just at the end.
Any character, integer, or pathname is directly added to the list of arguments, as if it were an escaped string. (It is an error if a pathname begins with
Cmd supports a basic form of process substitution, running processes as input to commands that expect files. To construct a process substitution, use the
(cmd? "diff" (psub "echo x") (psub "echo x")) => T (cmd? "diff" (psub "echo x") (psub "echo y")) => NIL
The external program’s working directory
Cmd is designed with multi-threaded programs in mind. It always runs
programs with their working directory relative to
*default-pathname-defaults*. This is because the OS-level
working directory a program, on both Windows and Unix, is the working
directory of the entire process, not the individual thread, and
changing it changes it for all threads.
You can also specify the directory for a particular command with the
(cmd "ls" :in #p"/") (cmd :in #p"/" "ls") => /bin /home /tmp /usr ...
For convenience Cmd supplies the macro
(with-working-directory (dir) (cmd ...) (cmd ...)) ≡ (progn (cmd :in dir ...) (cmd :in dir ...))
The external program’s environment
For Unix users only, the variable
*cmd-env* holds an alist of extra
environment variables to set for each call to
(let ((*cmd-env* (acons "GIT_PAGER" "cat" *cmd-env*))) (cmd "git diff" ...))
We are currently very restrictive about what we consider a valid environment variable name.
cmd package offers several entry points:
cmdruns an external program synchronously, returning the exit code. By default, on a non-zero exit it signals an error.
(cmd "cat /etc/os-release") NAME="Ubuntu" [...] => 0
$cmdreturns the output of the external program as a string, stripping any trailing newline. (Much like
$(cmd)in a shell.)
($cmd "date") => "Sun Sep 27 15:43:01 CDT 2020"
cmd!runs an external program purely for side effects, discarding all output and returning nothing. If the program exits non-zero, however, it will still signal an error.
tif the external program returned
nilotherwise, with the exit code as a second value. As other variants by default signal an error if the process exists non-zero,
cmd?is useful for programs that are expected to fail.
(cmd? "kill -0" pid) => T # PID is a live process => NIL # PID is not a live process
cmd&runs an external program asynchronously (with
uiop:launch-program) and returns a UIOP
(cmd& "cp -a" src dest) => #<PROCESS-INFO ...>
Redirection is accomplished via either tokenized strings or keyword arguments. These should be self-explanatory to anyone who has used a shell.
;;; Using keyword arguments. (cmd "echo 'hello world'" :> "hello.txt") (cmd "cat hello.txt") => hello world ;; Append (cmd "echo 'goodbye world'" :>> "hello.txt") (cmd "cat hello.txt") => hello world goodbye world (cmd "tar cf - hello.txt" :> #p"hello.tar") (cmd "rm hello.txt") (cmd "tar xf hello.tar") (cmd "cat hello.txt") => hello world goodbye world ;;; Using tokenized strings. (cmd "echo 'hello world' > hello.txt") (cmd "cat hello.txt") => hello world ;; Append (cmd "echo 'goodbye world' >> hello.txt") (cmd "cat hello.txt") => hello world goodbye world (cmd "tar cf - hello.txt > hello.tar") (cmd "rm hello.txt") (cmd "tar xf hello.tar") (cmd "cat hello.txt") => hello world goodbye world
Redirection with keyword arguments is usually more readable when the arguments are computed.
Supported directions include:
:>&Redirect stdout and stderr.
:>>&Append stdout and stderr.
:<<<Provide input from a “here string”.
Note that redirections are interpreted according to the rules for Lisp keywords (only the first occurrence of a keyword argument matters), not the side-effecting rules for redirections in POSIX shells.
The simplest way to set up pipelines is to use tokenized strings:
(cmd "cat /usr/share/dict/words | sort | uniq -c | sort -nrs | head -3") => 1 a 1 A 1 Aachen
Alternately you can use keywords. While
:|\|| is acceptable, you can write
"|" instead. (Remember
"|" will be tokenized to
(cmd "cat /usr/share/dict/words" "|" '("sort") "|" '("uniq" "-c") "|" '("sort" "-nrs") "|" '("head" "-3")) => 1 a 1 A 1 Aachen
Again, separating out the pipeline symbols is usually more readable when the subcommands are computed.
Controlling cmd with hooks
There are two hooks you can use to control
cmd. These are exported from the
cmd/hooks package (so you can
:use :cmd without having to worry about them.) Both hooks expect a list of functions of one argument.
*message-hook* is called with the external program and its arguments, quoted as a shell command line. This can be useful for logging commands as they are run.
*proc-hook* is called with the process object (as returned by
uiop:launch-program). This can be useful if you want to be able to track what is being run in a particular dynamic extent.
On Windows only, the first argument (the program name) has
.exe appended to it automatically if it doesn’t already have a file extension.
cmd does not use a shell to interpret its arguments, it does still have to run a shell (
sh on Unix,
cmd.exe on Windows) in order to change the working directory of the program.
How inefficient this is depends on what your distribution uses as a shell; it is faster when
sh is, say,
dash, than when it is
Recent versions of GNU
env support a
-C switch to do this directly. When that is supported (support is detected dynamically) then
env -C is used in place of a shell and overhead is negligible.
Cmd is a spinoff of Overlord, a Common Lisp build system, and was
inspired by the
cmd function in Shake, a Haskell build system,
as well as the Julia language’s shell command
psub function is inspired by the
builtin of the same name in the Fish shell.
- Pipelines should have “pipefail” behavior.
- Pipelines should support stderr as well (
- Efferent process substitution should also be supported.
- There should be a special variable holding an alist of extra environment variables to set when running a command. (The problem here is Windows.)