• Simplified API/implementation, achieved by REMOVING some features in run-program
• Declarative process handling
• Subprocesses spawned by POSIX fork&exec, no impl-specific interface
• Memory / CPU-time management of the child processes
• Compatibility layer to the existing State-of-the-Art libraries
  • trivial-shell
  • inferior-shell
  • sb-ext:run-program

1Usage

(defvar pipe (pipe))
(defvar in "t/test-input")
(defvar out "t/test-output")

(defvar p1 (shell '("cat") `(,in  ,pipe))) ; fd 0:pathname, 1:pipe
(defvar p2 (shell '("cat") `(,pipe ,out))) ; fd 0:pipe, 1:pathname

(wait p1)
(wait p2)

Processes are async by default. Even if you forget waiting the process, it is handled by trivial-garbage --- when a process object (here, p1 and p2) is GC-ed, it is automatically finalized -- killed and waited.

To ensure the process is finalized, with-process might help you. It ensures the subprocess is terminated and no zombie remains. However, it does not synchronize with the subprocess by itself --- synchronization should be done manually.

(with-process (p1 '("sleep" "3"))
  (print :x))
;; p1 is killed right after the execution of (print :x),
;; which means that the subprocess does not actually run for 3 seconds.

(with-process (p1 '("sleep" "3"))
  (print :x)
  (wait p1)) ;; synchronization. Waits for 3 sec

Note that if you leave too many processes alive, the total number of file descriptors might hit the system-provided limit (around 1000 by default).

1.1Implicit Piping

If you do not specify pipes nor pathnames, a new pipe is created for each fd, and the other end of the pipe is accessible with (fd process fdnum). When nil is specified, it means /dev/null.

    (let* ((in "t/test-input")
           (p1 (shell '("cat") `(,in :output))) ; fd 0,1 where 1 is an implicit pipe
           (p2 (shell '("cat") `(,(fd p1 1) :out :o :input :in :i nil)))) ; fd 0-6
      (wait p1)
      (wait p2))

1.2Reading the output

The output of the subprocess is NOT directly accessible with EAZY-PROCESS. Instead, open the pathname returned by fd-as-pathname, which is /dev/fd/[fd].

(test read
  (let ((p1 (shell `("hostname"))))
    (with-open-file (s (fd-as-pathname p1 1))
      (is (string= (machine-instance)
                   (read-line s))))))

This has greatly simplified the API of eazy-process.

1.3Resource management

Macro with-rlimit dynamically binds the current rlimit resource limitation. As noted in *rlimit-resources* docstring, this does not affect the lisp process itself. --- It only affect the new child processes spawned by shell.

Example below shows the usecase where *spendtime* contains a path to a simple C program that busy-waits for 10 seconds. The execution is terminated in 3 seconds. TERMSIG is set to 24 because the program is killed by SIGXCPU.

 (with-rlimit ((+rlimit-cpu-time+ 3)) ; 3 sec
   (let ((p (shell `(,(namestring *spendtime*))))) 
     (multiple-value-bind
            (exited exitstatus ifsignalled termsig ...)
            (wait p)
       (is-false exited)
       (is-false exitstatus)
       (is-true ifsignalled)
       (is (= 24 termsig)))))

This macro can be nested, and the new subprocess reflects the inntermost limitation.

(with-rlimit ((+rlimit-cpu-time+ 3))
  (shell ...) ; 3 sec
  (with-rlimit ((+rlimit-cpu-time+ 5)
                (+rlimit-as+ 500000))
    (shell ...))) ; 5 sec, 500 MB

2Somewhat Longer Description

In `run-program` interface in the popular implementations, piping between subprocesses are hard. It requires either reading the entire output stream and packing the contents as a new string-input-stream, or using some other implementation-specific functions. Also, compatibility libraries e.g. trivial-shell or inferior-shell, often depend on these functions, implying the same problem.

Iolib also has `run-program` that allows easy piping, but it is restricted to 3 fds: `input,output,error`.

Eazy-process provides a clean, declarative and thin layer for the processes. It depends on the concept of "everything is a file" and do not provide interfaces to streams.

3Tested Impl

This library is at least tested on implementation listed below:

• SBCL 1.2.1 on X86-64 Linux 3.13.0-39-generic (author's environment)
• SBCL 1.1.14 on X86 Linux 3.13.0-44-generic (author's environment)
• CCL 1.10 on linux currently has a problem reading /dev/fd/[fd], whichis actually a symlink to /proc/[pid]/fd/[fd], and the test does notpass. Do not use (fd-as-pathname process fd) and use temoraryfiles instead.
• ECL opens /dev/fd/[fd] correctly, but it fails to load CFFI...
• ABCL has more problems than CCL. It fails to open /proc/[pid]/fd/[fd]and also have problems with CFFI.

Test reports on other OS'es/impls are greatly appreciated. Run ./simple-build-test.sh, assuming it already loads quicklisp in your init files.

4Dependencies

It depends on the latest libfixposix available at https://github.com/sionescu/libfixposix .

Also, it depends on the following libraries:

• iterate by Jonathan Amsterdam :Jonathan Amsterdam's iterator/gatherer/accumulator facility
• Alexandria by ** :Alexandria is a collection of portable public domain utilities.
• cffi by James Bielman <jamesjb@jamesjb.com> :The Common Foreign Function Interface
• trivia by Masataro Asai
• iolib
• trivial-garbage
• cl-rlimit

5Syntax

(defun shell (argv &optional
               (fdspecs '(:input :output :output))
               (environments nil env-p)
               (search t))
    ...)

When search is nil, it disables the pathname resolving using PATH.

5.1Fdspecs

fdspecs := {fdspec}*
fdspec  := output | input | fd | path-or-pipe | openspec
output  := :output | :out | :o
input   := :input | :in | :i
fd      := <fixnum>
openspec := (path-or-pipe &key direction if-exists if-does-not-exist)
path-or-pipe := <pipe object> | <pathname>
direction := :input | :output | :io | :probe
if-exists := :overwrite | :supersede | :append | :error
if-does-not-exist := :create | :error

• output form and input form implicitly create a new pipe.
• The fixnum fd should be a value of function (fd process fdnum).
• Openfilespec is almost identical to the argument list of OPEN in ANSIspec, however :rename, :rename-and-delete, :new-version are notsupported and signals an error.
• Function pipe generates a new pipe object that can be used in an fdspec.
• If a <pipe object> or a <pathname> are given without options, it usesa default direction, which is :input for fd 0 and :output for fd 1and fd 2. For fd > 2, missing direction signals an error.
• Be careful when you open a fifo, the process will be blocked.

5.2Environments

environments := {environment}*
environment  := env-pair | env-string
env-pair     := (name . value)
env-string   := "name=value"
name, value -- string

If we omit the second argument environments, the subprocess inherits the environment of the parent lisp process. unset -ting the environment value is not available.

6Compatibility Layers

There are compatibility layers for trivial-shell and inferior-shell. For documentation, see compat/README.org .

6.1run-program compatibility

Abandoned , since the design of run-program is not thoroughly delibelated, and any form of its descendants is not acceptable. It is a mistake to combine processes with streams in a tightly coupled manner.

7Author & Copyright

Copyright (c) 2014 Masataro Asai (guicho2.71828@gmail.com)