Overlord

Overlord is an experimental build system for Common Lisp, inspired by Redo.

Overlord addresses two problems which might seem unrelated, but which, on closer examination, turn out to the same problem:

1. It lets you reproducibly specify the desired state of a Lisp system which is to be saved as an image.

2. It provides a general-purpose build system (a superset of Make, inspired by Redo).

Overlord expects to be used alongside ASDF, with ASDF responsible for compiling and loading Lisp code, and Overlord doing everything else.

For more discussion of the thinking behind Overlord, how it relates to Redo and other build systems, consult the wiki.

Advice for users

Before loading Overlord, it would be a good idea to make sure you are running the latest version of ASDF.

Note that, to run the test suite, you will need to download Core Lisp, and, if not on Windows, you must have the touch program in your search path. (On Windows, Powershell is used instead).

Overlord stores its persistent data structures in a cache directory. On Linux, this is $XDG_CACHE_HOME/overlord. The data structures stored there are versioned. It might worth checking the cache directory from time to time to delete obsolete files.

Overlord is developed and tested on Clozure and SBCL. In the future it may officially support other Lisp implementations, but that is not a priority. Lisp implementations that do not support image-based persistence (e.g. ECL) are unlikely to receive support.

Examples

Here are some examples of how to make direct use of Overlord:

1. cl-https-everywhere. In-process HTTPS Everywhere rulesets, automatically fetched from the HTTPS Everywhere repository and compiled into Lisp code.

2. Proctor. Proctor treats tests as build targets, allowing you to precisely specify their dependencies and re-run tests only when necessary.

3. Vernacular. Provides a module system for embedding languages, with arbitrary syntaxes, into Common Lisp systems.

(Non-)Parallelism

One thing that might not be obvious about Redo-style build systems is that they afford unusually good opportunities for parallelism.

Overlord does not currently support parallelism, but even without parallelism, it tries to discourage reliance on side effects by, whenever possible, randomizing the order in which targets are built.

Freezing the Lisp image

During development, as targets are defined and re-defined, and rebuilt or not rebuilt, the actual state of the Lisp world will drift away from the one specified by Overlord?s dependency graph. Before an image is saved, Overlord needs to do two things to resolve such discrepancies:

1. Finalize the state of the image by making sure that all defined targets have been built.

2. Disable itself.

If you use uiop:dump-image to save the image, you don?t need to do anything else; Overlord will finalize the state of the image, and disable itself, automatically.

If you are using implementation-specific means to save an image, however, you will need to arrange to call overlord:freeze before the image is saved.

The default policy is to allow the restored image to be unfrozen, and development to be resumed, by calling overlord:unfreeze. This is probably what you want when, say, saving an image on a server. In other scenarios, however ? like delivering a binary ? you may want to strip the build system from the image entirely. This is possible by changing Overlord?s ?freeze policy?, using the freeze-policy accessor.

;;; The default: can be reversed after
;;; loading the image by calling
;;; `overlord:unfreeze`.
(setf (overlord:freeze-policy) t)

;;; Irreversible: before saving the
;;; image, Overlord should destroy its
;;; internal state.
(setf (overlord:freeze-policy) :hard)