Yet Another Portable Library for Process Handling / Subshell Invokation

eazy-process


#+startup: showall

+ Simplified API/implementation, achieved by *REMOVING* some features in run-program
+ Declarative process handling
+ Subprocesses spawned by POSIX fork&exec, no impl-specific interface
+ Memory / CPU-time management of the child processes
+ [-] Compatibility layer to the existing State-of-the-Art libraries 
  + [X] trivial-shell
  + [ ] inferior-shell
  + [ ] sb-ext:run-program

* Usage

#+BEGIN_SRC lisp
(defvar pipe (pipe))
(defvar in "t/test-input")
(defvar out "t/test-output")

(defvar p1 (shell '("cat") `(,in  ,pipe))) ; fd 0:pathname, 1:pipe
(defvar p2 (shell '("cat") `(,pipe ,out))) ; fd 0:pipe, 1:pathname

(wait p1)
(wait p2)
#+END_SRC

Processes are async by default.
Even if you forget waiting the process, it is handled by trivial-garbage
--- when a process object (here, p1 and p2) is GC-ed, it is automatically
finalized -- killed and waited.

To ensure the process is finalized, =with-process= might help you.
It ensures the subprocess is terminated and no zombie
remains. However, it does not synchronize with the subprocess by itself ---
synchronization should be done manually.

#+BEGIN_SRC lisp
(with-process (p1 '("sleep" "3"))
  (print :x))
;; p1 is killed right after the execution of (print :x),
;; which means that the subprocess does not actually run for 3 seconds.

(with-process (p1 '("sleep" "3"))
  (print :x)
  (wait p1)) ;; synchronization. Waits for 3 sec
#+END_SRC

Note that if you leave too many processes alive, the total number of file
descriptors might hit the system-provided limit (around 1000 by default).

** Implicit Piping

If you do not specify pipes nor pathnames, a new pipe is created for each
fd, and the /other/ end of the pipe is accessible with =(fd process
fdnum)=. When =nil= is specified, it means /dev/null.

#+BEGIN_SRC lisp
    (let* ((in "t/test-input")
           (p1 (shell '("cat") `(,in :output))) ; fd 0,1 where 1 is an implicit pipe
           (p2 (shell '("cat") `(,(fd p1 1) :out :o :input :in :i nil)))) ; fd 0-6
      (wait p1)
      (wait p2))
#+END_SRC

** Reading the output

The output of the subprocess is *NOT directly accessible with EAZY-PROCESS*.
Instead, open the pathname returned by =fd-as-pathname=, which is =/dev/fd/[fd]=.

#+BEGIN_SRC lisp
(test read
  (let ((p1 (shell `("hostname"))))
    (with-open-file (s (fd-as-pathname p1 1))
      (is (string= (machine-instance)
                   (read-line s))))))
#+END_SRC

This has greatly simplified the API of eazy-process.

** Resource management

Macro =with-rlimit= dynamically binds the current rlimit resource limitation. 
As noted in =*rlimit-resources*= docstring, this does not affect the lisp process itself.
--- It only affect the new child processes spawned by =shell=.

Example below shows the usecase where =*spendtime*= contains a path to a
simple C program that busy-waits for 10 seconds. The execution is
terminated in 3 seconds. TERMSIG is set to 24 because the program
is killed by SIGXCPU.

#+BEGIN_SRC lisp
 (with-rlimit ((+rlimit-cpu-time+ 3)) ; 3 sec
   (let ((p (shell `(,(namestring *spendtime*))))) 
     (multiple-value-bind
            (exited exitstatus ifsignalled termsig ...)
            (wait p)
       (is-false exited)
       (is-false exitstatus)
       (is-true ifsignalled)
       (is (= 24 termsig)))))
#+END_SRC

This macro can be nested, and the new subprocess reflects the inntermost limitation.

#+BEGIN_SRC lisp
(with-rlimit ((+rlimit-cpu-time+ 3))
  (shell ...) ; 3 sec
  (with-rlimit ((+rlimit-cpu-time+ 5)
                (+rlimit-as+ 500000))
    (shell ...))) ; 5 sec, 500 MB
#+END_SRC

* Somewhat Longer Description

In `run-program` interface in the popular implementations, piping between
subprocesses are hard. It requires either reading the entire output stream
and packing the contents as a new string-input-stream, or using some other
implementation-specific functions. Also, compatibility libraries e.g.
trivial-shell or inferior-shell, often depend on these functions, implying
the same problem.

Iolib also has `run-program` that allows easy piping, but it is restricted
to 3 fds: `input,output,error`.

Eazy-process provides a clean, declarative and thin layer for the
processes. It depends on the concept of "everything is a file" and do not
provide interfaces to streams.

* Tested Impl

This library is at least tested on implementation listed below:

+ SBCL 1.2.1 on X86-64 Linux 3.13.0-39-generic (author's environment)
+ SBCL 1.1.14 on X86 Linux 3.13.0-44-generic (author's environment)
+ CCL 1.10 on linux currently has a problem reading =/dev/fd/[fd]=, which
  is actually a symlink to =/proc/[pid]/fd/[fd]=, and the test does not
  pass. Do not use =(fd-as-pathname process fd)= and use temorary
  files instead.
+ ECL opens =/dev/fd/[fd]= correctly, but it fails to load CFFI...
+ ABCL has more problems than CCL. It fails to open =/proc/[pid]/fd/[fd]=
  and also have problems with CFFI.

Test reports on other OS'es/impls are greatly appreciated.
Run =./simple-build-test.sh=, assuming it already loads quicklisp in your
init files.

* Dependencies

It depends on the latest libfixposix available at
https://github.com/sionescu/libfixposix .

Also, it depends on the following libraries:

+ iterate by *Jonathan Amsterdam* :
    Jonathan Amsterdam's iterator/gatherer/accumulator facility
+ Alexandria by ** :
    Alexandria is a collection of portable public domain utilities.
+ cffi by *James Bielman  <jamesjb@jamesjb.com>* :
    The Common Foreign Function Interface
+ trivia by *Masataro Asai*
+ iolib
+ trivial-garbage
+ cl-rlimit

* Syntax

#+BEGIN_SRC lisp
(defun shell (argv &optional
               (fdspecs '(:input :output :output))
               (environments nil env-p)
               (search t))
    ...)
#+END_SRC

When =search= is nil, it disables the pathname resolving using PATH.

** Fdspecs

#+BEGIN_SRC 
fdspecs := {fdspec}*
fdspec  := output | input | fd | path-or-pipe | openspec
output  := :output | :out | :o
input   := :input | :in | :i
fd      := <fixnum>
openspec := (path-or-pipe &key direction if-exists if-does-not-exist)
path-or-pipe := <pipe object> | <pathname>
direction := :input | :output | :io | :probe
if-exists := :overwrite | :supersede | :append | :error
if-does-not-exist := :create | :error
#+END_SRC

+ =output= form and =input= form implicitly create a new pipe.
+ The fixnum =fd= should be a value of function =(fd process fdnum)=.
+ Openfilespec is almost identical to the argument list of =OPEN= in ANSI
  spec, however =:rename=, =:rename-and-delete=, =:new-version= are not
  supported and signals an error.
+ Function =pipe= generates a new pipe object that can be used in an fdspec.
+ If a =<pipe object>= or a =<pathname>= are given without options, it uses
  a default direction, which is =:input= for fd 0 and =:output= for fd 1
  and fd 2. For fd > 2, missing direction signals an error.
+ Be careful when you open a fifo, the process will be blocked.

** Environments

: environments := {environment}*
: environment  := env-pair | env-string
: env-pair     := (name . value)
: env-string   := "name=value"
: name, value -- string

If we omit the second argument =environments=,
the subprocess inherits the environment of the parent lisp process.
=unset= -ting the environment value is not available.

* Compatibility Layers

There are compatibility layers for /trivial-shell/ and /inferior-shell/.
For documentation, see [[compat/README.org]] .

** /run-program/ compatibility

*Abandoned* , since the design of =run-program= is not thoroughly
delibelated, and any form of its descendants is not acceptable.
It is a mistake to combine processes with streams in a
tightly coupled manner.

* Author & Copyright

Copyright (c) 2014 Masataro Asai (guicho2.71828@gmail.com)



alexandria

cffi

cl-ppcre

cl-rlimit

iolib/syscalls

iterate

trivia

trivial-garbage

trivia.ppcre

In `run-program` interface in the popular
  implementations, piping between subprocesses are hard. It requires either
  reading the entire output stream and packing the contents as a new
  string-input-stream, or using some other implementation-specific
  functions. Also, compatibility libraries e.g.  trivial-shell or
  inferior-shell, often depend on these functions, implying the same
  problem. Iolib also has `run-program` that allows easy piping, but it is
  restricted to 3 fds: `input,output,error`.

  Eazy-process provides a clean, declarative and thin layer to the
  processes. It depends on the concept of 'everything is a file' and do not
  provide interfaces to streams.

fiveam

eazy-process.test

eazy-process

Upstream URL

Author

License

eazy-process

Version:

License:

Authors:

Dependencies:

eazy-process.test

License:

Authors:

Dependencies:

Dependencies (9)

Dependents (0)