Shellpool

Shellpool is a way to run external programs from within a Common Lisp program. It features:

• Forking control. Sub-programs are launched with a separate shell, so you can avoid forking your Lisp image, which may be unreliable you have dozens of GB of memory allocated or are running multiple threads.

• Output handling. You can easily stream, filter, or collect output from the command. You can access both stdout and stderr lines as they are produced, and can tell the difference between them.

• Exit code. You get it.

• Interruption. Interrupts are handled gracefully. After you interrupt (e.g., Control C), you can :continue to keep running the program, or :q to terminate the sub-program and return to Lisp as normal.

• Multithreading. You can safely launch multiple programs from multiple threads. Threading support is based on the bordeaux-threads library.

• Portability. It runs on many Lisps and operating systems, installs via Quicklisp, and has few external dependencies.

Note: Shellpool is not suitable for running sub-commands that need access to command-line input from the user or for TTY-based programs.

Resources

• Github Project -- This is the main shellpool homepage. Please feel free to use the issue tracker, etc. Note that the master branch is the stable version, while the devel branch is the under-development version and may be unstable.

• Installation

• API Documentation

Related Lisp Libraries

If shellpool isn't quite right for you, here are some related Lisp libraries.

• inferior-shell is allegedly very complete and portable for synchronous shells. It has fancy features like support for remote execution (via ssh) and a domain specific language for constructing pipelines of shell commands.

• trivial-shell is less full featured than inferior-shell, but is apparently highly portable.

• ASDF has run-program shell-command with many options.

• external-program is a wrapper for run-program functionality.

License and History

Shellpool is Copyright (C) 2014-2015 Kookamara LLC. It has an MIT/X11 style license. It was written by Jared Davis.

Shellpool is a successor to "tshell", a mechanism for running external programs from Clozure Common Lisp. Tshell was developed by Centaur Technology and was distributed as a library for ACL2; it was also released under the MIT license. It was mainly used for running SAT solvers, which are programs that can easily get "stuck" on hard problems, and hence often need to be interrupted. It worked very well for this purpose.

Unfortunately tshell was inherently single-threaded and wasn't portable beyond CCL on Linux. Shellpool is a complete rewrite that solves these problems, by explicitly having a pool of shells (to support multiple threads), by adding compatibility wrappers (to support multiple Lisps), and by reworking the core Bash code (to support multiple OSes).

Shellpool has now replaced Tshell in the ACL2 books and is now being used, in production, on an everyday basis.