A conduit package is a package which sits between one or more implementation packages and users of those packages. As an example imagine you have a system which has two implementation packages:
b. But you want users of your system to simply be able to use a single package to get all the functionality of the system. Then a conduit package, say
c, can import and then reexport all the external symbols from both
b. Or it could reexport only some subset of those symbols. This is something which is perfectly possible just using normal package definitions, but it's a pain to manage as you have to keep lists of exports up to date. It's also not very dynamic: if the underlying packages change you need to update and redefine the conduits manually.
The conduit packages system provides tools which help you define and manage conduit packages like this easily as well as two other things.
- Conduit packages themselves are supported by a variant version of
defpackagetogether with variants of some of the CL package functions which are aware of conduits and deal with keeping them up to date. Assuming you modify packages using the functionality defined in this system, then conduit packages will notice changes you make and recompute themselves.
- The variant
defpackageis itself extensible. This extensibility is used to support the new options needed for conduit packages, but it can be further extended by other systems.
- Finally there is a way of making clones of packages which try to be as identical to them as possible. Clones are not dynamic, but rather point-in-time copies.
Conduits and the extensible
defpackage variant are independently useful, or I think so. Package cloning probably is not: it was once useful on Lisp Machines where you really didn't want to break things too much because it took half an hour to reboot the machine, so instead you might work by making a clone of a package, doing some work in the clone and then deleting it and starting again. Today, I can restart a lisp and rebuild everything from cold in a few seconds: cloning is not very useful any more. The extensible
defpackagevariant, together with the reengineering of conduits to use it, is recent: see further down for some history.
Systems and packages
There are two systems which provide various mixes of features.
This is the guts of everything. It defines three packages[^1].
org.tfeb.conduit-packagesis the implementation package. It exports everything, including symbols which have names which conflict with symbols in
CL. Generally, you want to interface to this package by defining packages which are themselves conduits of it.
org.tfeb.clc). This is a package which is a conduit for both
org.tfeb.conduit-packages: it reexports all the symbols in
CL, except that its
defpackageand some other package-related names come from
org.tfeb.cl/conduits. If you define a package which uses this (you can't use both it and
CL) you will get a
defpackageunderstands the conduit (and package cloning) options as well as the normal ones.
org.tfeb.clc-user) is such a package: it is a package which uses only
org.tfeb.cl/conduitsand can be used for experimentation and small programs the way
CL-USERis. One of its main uses in practice is as a package in which to place package declarations for other packages.
Note that it will not (and can not) work, in any package which uses
CL, to simply say
(use-package :org.tfeb.conduit-packages), as there are inevitable name clashes. Instead, you should either define a package which uses
org.tfeb.clc, define your own conduit package (see below), or use the
org.tfeb.conduit-packages/define-package package described below.
This is a shim system which provides all the conduits functionality under names which do not clash with names in
It provides one package:
org.tfeb.conduit-packages/define-package: this package exports the variant
defpackage under its native name,
define-package, some shims for the conduit packages variants of package functionality, as well as all the names associated with managing conduit packages, extending
define-package itself and so on.
Almost certainly this is the package to use if you want to use conduit packages and are happy to say
define-package rather than
Conduits by example
The old way using
These examples assume that the current package is
org.tfeb.clc-user, so that
org.tfeb.conduit-packages:defpackage for instance.
Let's say I'm writing a program which consists of several implementation packages, say
com.cley.my-great-prog.outright-misfeatures. The definitions of these three packages might be:
(defpackage :com.cley.my-great-prog.clever-hacks (:use :cl) (:export #:cause-fire)) (defpackage :com.cley.my-great-prog.not-so-clever-hacks (:use :cl) (:export #:cause-serious-fire)) (defpackage :com.cley.my-great-prog.outright-misfeatures (:use :cl) (:export #:fail-to-put-out-fire))
Now I want to provide a single package,
com.cley.my-great-program which combines the functionality of the three packages:
(defpackage :com.cley.my-great-prog (:use) (:extends :com.cley.my-great-prog.clever-hacks) (:extends :com.cley.my-great-prog.not-so-clever-hacks) (:extends :com.cley.my-great-prog.outright-misfeatures))
> (do-external-symbols (s (find-package :com.cley.my-great-prog)) (format t "~&~A from ~A~%" (symbol-name s) (package-name (symbol-package s)))) FAIL-TO-PUT-OUT-FIRE from COM.CLEY.MY-GREAT-PROG.OUTRIGHT-MISFEATURES CAUSE-FIRE from COM.CLEY.MY-GREAT-PROG.CLEVER-HACKS CAUSE-SERIOUS-FIRE from COM.CLEY.MY-GREAT-PROG.NOT-SO-CLEVER-HACKS
com.cley.my-great-prog serves as a conduit for the various implementation packages of the program, which means that users of the program don't have to worry about what the implementation packages are.
Conduits are dynamic. If I now decide that the
com.cley.my-great-prog.clever-hacks package should export some other symbols, I can simply redefine it:
(defpackage :com.cley.my-great-prog.clever-hacks (:use :cl) (:export #:cause-fire #:light-match #:burn-petrol))
(do-external-symbols (s (find-package :com.cley.my-great-prog)) (format t "~&~A from ~A~%" (symbol-name s) (package-name (symbol-package s)))) FAIL-TO-PUT-OUT-FIRE from COM.CLEY.MY-GREAT-PROG.OUTRIGHT-MISFEATURES BURN-PETROL from COM.CLEY.MY-GREAT-PROG.CLEVER-HACKS CAUSE-FIRE from COM.CLEY.MY-GREAT-PROG.CLEVER-HACKS LIGHT-MATCH from COM.CLEY.MY-GREAT-PROG.CLEVER-HACKS CAUSE-SERIOUS-FIRE from COM.CLEY.MY-GREAT-PROG.NOT-SO-CLEVER-HACKS
The conduit package noticed the redefinition of a package for which it was a conduit, and changed its exports appropriately.
The dynamic behaviour of conduit packages is a little fragile: it will work so long as you use the conduits-provided versions of
export and so on, but it won't work if you use the standard CL ones, either explicitly or implicitly, during error recovery say. If the system gets itself in a mess, you can always call
org.tfeb.conduit-packages:recompute-conduits to recompute everything. The dynamic behaviour of conduit packages is meant to be a help for when writing a program, during which process packages often get reorganised fairly frequently: it's not something that should be relied on for production.
Conduits can export only part of the packages for which they are conduits. For instance, perhaps I don't want the
(defpackage :com.cley.my-great-prog (:use) (:extends/excluding :com.cley.my-great-prog.clever-hacks #:burn-petrol) (:extends :com.cley.my-great-prog.not-so-clever-hacks) (:extends :com.cley.my-great-prog.outright-misfeatures))
Or perhaps I only want to burn petrol:
(defpackage :com.cley.my-great-prog (:use) (:extends/including :com.cley.my-great-prog.clever-hacks #:burn-petrol) (:extends :com.cley.my-great-prog.not-so-clever-hacks) (:extends :com.cley.my-great-prog.outright-misfeatures))
In these two latter cases the symbols you are excluding or including need to exist in the package being extended: you can't, for instance, add an exclusion for a symbol which only may be there, or which may be there in future. That might be a nice feature to add.
Using inclusions and exclusions like this allows you to construct conduit packages which are 'like' underlying packages but have some small differences, such as replacing some functionality.
org.tfeb.clc is such a package: it is a conduit which extends
CL but replaces some of its functionality. Here is its definition[^2]:
(defpackage :org.tfeb.cl/conduits (:use) (:nicknames :org.tfeb.clc) (:extends/excluding :cl #:export #:unexport #:defpackage #:delete-package #:rename-package) (:extends/including :org.tfeb.conduit-packages #:export #:unexport #:defpackage #:delete-package #:rename-package))
You can see that it just replaces a set of exports from
CL with names from
org.tfeb.clc-user is not a conduit: it's just a package which uses
org.tfeb.clc rather than
(defpackage :org.tfeb.cl-user/conduits (:nicknames :org.tfeb.clc-user) (:use :org.tfeb.clc))
The new way using
These examples assume the package is
CL-USER. Assume there are two packages,
com.cley.program-2 both of which export a class called
onion. We want to define a conduit which exports only one of these. Here is how to do that:
(in-package :cl-user) (use-package :org.tfeb.conduit-packages/define-package) (define-package :org.tfeb.program (:use) (:extends :org.tfeb.program-1) (:extends/excluding :org.tfeb.program-2 #:onion))
Here's a slightly larger example where four packages are involved:
(in-package :cl-user) (use-package :org.tfeb.conduit-packages/define-package) (define-package :org.tfeb.substrate (:use :cl) (:export #:vegetable)) (define-package :org.tfeb.program-1 (:use :cl) (:use :org.tfeb.substrate) (:export #:onion #:shallot)) (define-package :org.tfeb.program-2 (:use :cl) (:use :org.tfeb.substrate) (:export #:onion #:leek)) (define-package :org.tfeb.program (:use) (:extends/including :org.tfeb.substrate #:vegetable) (:extends :org.tfeb.program-1) (:extends/excluding :org.tfeb.program-2 #:onion))
Cloning a package is making a package which is 'like' it: all of its internal, external, and shadowing symbols, as well as its used packages will be the same as the package it clones, but in addition any other things asked for by the
defpackage form will be done. Once a clone is made it lives an independent life to the package it clones: clones are not dynamic, and don't remember what package they were cloned from. Clones can't also be conduits.
Here is an example of making a clone:
(defpackage :org.tfeb.foo (:use :org.tfeb.clc) (:export #:spot)) (intern "FUG" ':org.tfeb.foo) (defpackage :org.tfeb.bar (:use :org.tfeb.clc) (:clones :org.tfeb.foo) (:export #:spit))
> (eq 'org.tfeb.foo:spot 'org.tfeb.bar:spot) t > (eq 'org.tfeb.foo::fug 'org.tfeb.bar::fug) t > (eq 'org.tfeb.foo::fog 'org.tfeb.bar::fog) nil > 'org.tfeb.bar:spit org.tfeb.bar:spit > 'org.tfeb.foo:spit Error: Symbol "SPIT" not found at all in the ORG.TFEB.FOO package. [...]
The idea behind package clones was to allow you to make a quick-and-dirty point-in-time copy of a package in which you could then experiment without contaminating the namespace of the cloned package. Their intended use was on LispMs which took a long time to reboot: in practice I think I have almost never used them and I don't think they are useful now.
Here is at least some documentation of the exports of
These two macros are the same (
defpackage is now a shim for
define-package) and provide an extensible variant of
defpackage. The extensions take the form of new clauses which the macros understand. The pre provided nonstandard clauses are:
(:extends <package>)to reexport all the external symbols in
(:extends/including <package> [<name> ...])to reexport only the external symbols designated by the specified names from
(:extends/excluding <package> [<name> ...])to reexport all the external symbols except those designated by the specified names from
(:clones <package>)to make a point-in-time clone of
All of these clauses can also be spelled in the singular:
(extend <package>) for historical reasons.
These clauses are all implemented by a predefined mechanism for
define-package: see below.
In addition the semi-standard
:local-nicknames clause is passed down to the underlying
defsystem by default: other clauses like this can be added (see below).
Variant package functions, and a utility
These functions call their CL equivalents but also do conduits maintenance.
See below on how the CL equivalents are called and how you can replace them.
In addition there is a single utility function,
recompute-conduits. This takes no arguments and will recompute all conduit packages, causing them to take notice of any changes in the packages they are conduits for. This is useful if, for instance, something has called one of the underlying CL package functions directly.
defpackage extension mechanism
To define an extension for
define-package you need to pick a name for it, and add that name to the list of known 'mechanisms' for the macro. You then write
eql methods for three generic functions keyed on the name of the extension.
*define-package-mechanisms* is the list of known mechanisms for
define-package. It is a list of symbols, which by default contains two entries: one for the conduits mechanism and one for the basic CL mechanism (which itself is slightly configurable, see below).
initial-define-package-state is a generic function called with three arguments for each mechanism in the list:
- the mechanism name, a symbol;
- the name of the package being defined (first argument to
- all the clauses to
It is called at the start of clause processing and its return value is a 'state' which is some object handed to and returned from the other generic functions in the protocol. A state can be any object at all.
*define-package-mechanisms* should define an
eql method for their name on the first argument of this function. There is no fallback method, so any missing methods will cause an error.
process-define-package-clause is called for each mechanism and clause in the body of the
define-package form. Its arguments are:
- the mechanism name, a symbol;
- the keyword of the
define-packageclause, a keyword (for instance
- the whole
define-packageclause being processed, a list (for instance
- the current state for the mechanism;
- the name of the package being defined;
- all the
It should return two values:
- a state object which may incorporate information about the clause;
nilif it handled the clause.
Note that each mechanism gets each clause: more than one mechanism can handle a given clause, but each clause needs to be handled by at least one mechanism. Also note that the state object returned by a call to
process-define-package-clause is what it passed to the next call. The initial state comes from
initial-define-package-state for that mechanism and the final state will be passed to
Again, mechanisms should define
eql methods for their name on this generic function. If you wanted to, you could define methods which have two
eql specializers: one for the mechanism name and one for the clause key. In practice, I've done the second part by a
case clause in the body.
compute-define-package-form is the final generic function in the protocol. Its job is to compute and return parts of the expansion of
define-class. It is called for each mechanism with four arguments:
- the mechanism name, a symbol;
- the final state for this mechanism;
- the name of the package being defined;
- all the
It should return three values:
- a list of forms which will be evaluated before the final
defpackageform resulting from the expansion;
- a list of clauses to add to the body of the
- a list of forms which will be evaluated after the
Some or all of these values may be
() of course.
The result of all this is that the defined mechanisms all get to contribute to the expansion of the
- There is limited control over the relative ordering of the various mechanisms, but the forms before the
defpackageform will be in the reverse order of the mechanisms in
*define-package-mechanisms*and the clauses in the body and forms after it in the order of the mechanisms. This is a little like around methods.
- All the forms before and after the main body of the expansion are wrapped in
(eval-when (:compile-toplevel :load-toplevel :execute)forms.
- All the generic functions get handed the package name and all the clauses just in case they want to, for instance, check for incompatible clauses, and so they can report errors and warnings in useful ways.
An example: the common lisp mechanism
This predefined mechanism listens to one variable.
*extended-cl-define-package-clause-keys* contains clause keys which the common lisp mechanism should consider to be standard as well as those defined by the language. By default, it contains
:local-nicknames and perhaps some implementation-specific clause keys. You can add clause keys to this to let the mechanism know it should pass them down to the underlying macro.
Here is the entire mechanism:
(defmethod initial-define-package-state ((mechanism (eql 'common-lisp)) name clauses) (declare (ignore name clauses)) '()) (defmethod process-define-package-clause ((mechanism (eql 'common-lisp)) key clause state name clauses) (declare (ignore name clauses)) (case key ((:nicknames :documentation :use :shadow :shadowing-import-from :import-from :export :intern :size) (values `(,@state ,clause) t)) (otherwise (if (member key *extended-cl-define-package-clause-keys*) (values `(,@state ,clause) t) (values state nil))))) (defmethod compute-define-package-forms ((mechanism (eql 'common-lisp)) state name clauses) (declare (ignore name clauses)) (values '() state '()))
An older extension mechanism
Before the current extensible
define-system I wanted to allow the possibility that things might sit on top of functions besides the CL ones. So
org.tfeb.clc:export might not just call
cl:export but some other function (which then presumably would call
cl:export somehow). This mechanism is controlled by the following variable:
*underlying-package-implementations* is an alist which maps between operation names and their underlying implementations. The operation names are keywords. It only maps operations for which conduit packages defines variant versions. If you wanted to cause
define-package to expand into
my-special:defpackage then you would push
(:defpackage . my-special:defpackage) onto this list. Its default value simply points everything at the CL functions.
Note that this variable is used during the macroexpansion of
define-package as well as by the variant versions of the CL package functions.
org.tfeb.conduit-packages/define-package, in the system with the same name, exports several shims with names which include the word
conduit and so will not clash with names from
CL. It also exports all the other conduits functionality, omitting only names that would clash with
The shim functions are:
delete-conduit-packageis a shim around the conduits
rename-conduit-packageis a shim around the conduits
export-from-conduit-packageis a shim around the conduits
unexport-from-conduit-packageis a shim around the conduits
You can use this package in a package which also uses
cl as no names now clash.
Conduit packages should generally be defined with
(:use) so the used-package list is empty rather than an implementation-default. A conduit really is just that: it doesn't need to use any packages because it's never a package where anything is defined.
defpackage macros use
*underlying-package-implementations* to know what the underlying
defpackage is, and so this variable matters at macro-expansion time as well as at runtime.
I believe that UIOP's
define-package can do at least some of what conduit packages can do: conduit packages predates it by at least a decade however. For a long time,
define-package was called
define-conduit-package to avoid clashing with the UIOP version, but now it is extensible I have given up on that.
Until 2023 this was all just one file, with a trivial ASDF system definition, That was a pain because you needed all sorts of annoying
eval-whens[^3], so now it is in multiple files with a system declaration which is less trivial. Apart from the shim system depending on conduit packages itself, there are no dependencies on other systems.
All of this system should be portable CL: if it's not, then that's a bug.
Conduit packages is an old system and has been through several changes, not all of them backward compatible.
Long ago (1998-2002)
When I first wrote it, it was entangled with a version of the Franz Allegro local package nicknames which was made publicly available some time in the late 1990s or early 2000s and which I made work in at least LispWorks and perhaps CMUCL and Genera at that time. That entanglement went away when I revived it in 2020 (astonishingly, it still worked in LW at that point).
Two approaches to conduits (2021)
When I first wrote conduit packages I was concerned about fasl size, so the expansion of
defpackage first used
cl:defpackage to define a simple package, and then walked over the packages it was extending and modified it appropriately. This made for small fasls, but meant that you often got warnings when compiling & loading files because
cl:defpackage would redefine a package in a way which was incompatible with its current state when the file was loaded.
I'm less concerned with fasl sizes now, so in 2021 I changed the implementation to compute an enormous
cl:defpackage form instead. For a file containing only, say
(defpackage :foo (:use) (:extends :cl)) this can cause a factor of ten increase in fasl size (from about 3kB to about 22kB in one implementation), but it means you don't get annoying warnings. Given that everything else around CL has bloated by hugely more than a factor of ten since the late 1990s, I think this is a price worth paying.
Substantial incompatible changes (2023)
In 2023, I realised that I needed a way of extending
defpackage for another purpose, but that conduit packages didn't make this really easy. I didn't want to implement yet another
defpackage version, so I bit the bullet and made several changes, not all of them compatible.
- I renamed
define-conduit-package, previously provided as a shim, to
define-packageand made it be the underlying macro, with
defpackagenow a shim for it.
- I implemented the extension mechanism for
define-packageand used it to replace the old hardwired macro.
- I renamed some variables.
- I split the system into several files to get rid of the old
- I renamed the shim system.
Much of this is not entirely compatible: I suspect it hurts nobody but me, however. It all smells better after the changes, I think.
Conduit packages is copyright 1998-2002, 2020-2023 by Tim Bradshaw. See
LICENSE for the license.
[^1]: Here and below I have given package names in lower case: they are, of course, really upper case strings unless you are using Allegro in its incompatible 'modern' mode. The exceptions are the
CL-USER packages which I've given in upper case: this is slightly inconsistent, sorry. Symbol names are always given in lower case, although they too are all upper case of course.
[^2]: In fact the definition now uses
define-package rather than
[^3]: And, once upon a time, you discover all sorts of exciting bugs in various implementation's handling of
eval-when. I expect these are now all long fixed though.