Module system for language embeddings.
Vernacular is a build and module system for languages that compile to Common Lisp. It allows languages to compile to Lisp while remaining part of the Common Lisp ecosystem. Vernacular languages interoperate with Common Lisp and one another.
Vernacular handles locating files, compiling files into fasls, tracking dependencies and rebuilding, and export and import between your new language, Lisp, and any other language Vernacular supports.
Here are some example languages:
Using a Vernacular language
Let’s use use CL-Yesql as an example.
In your system definition, add
"cl-yesql/sqlite" as a dependency.
Create a file named
fact.sql in your system, and add the following:
#lang cl-yesql/sqlite -- name: create-facts-table @execute CREATE TABLE fact (subject text, fact text) -- name: add-fact @execute INSERT INTO fact (subject, fact) VALUES (?, ?) -- name: facts-about @column SELECT fact FROM fact WHERE subject = ?
In a Lisp file in your system, add:
(vernacular:import facts :from "fact.sql" :binding :all-functions)
You can then do:
(sqlite:with-open-database (db ":memory:") (create-facts-table db) (add-fact db "Lisp" "Lisp is a programmable programming language.") (add-fact db "Lisp" "Lisp is fun") (facts-about db "Lisp")) ;; => '("Lisp is a programmable programming language." "Lisp is fun")
Two caveats. Vernacular needs to know what system your package belongs to. If your package has a different name from your system (why?), you may need to tell Vernacular what system it belongs to. In the package, evaluate:
Also, note that import paths are given relative to the base of the system, not the file with the import form. Whether you are importing from
all-in-one.lisp at the base of your project, or
deeply/nested/component/impl.lisp, the path does not change.
The language of a file can be specified in three ways.
The preferred way is to use a special first line (or second line, if the first line starts with
#lang my-lang ....
This is called a hash lang. A hash lang takes precedence over all other ways of specifying a language.
Sometimes, however, you will be re-using an existing syntax. This lets you employ existing tooling like editors, linters, etc. In this case we use Emacs’s syntax for specifying modes, using a special syntax anywhere in the first line:
# -*- mode: my-lang -*-
(You can consult the Emacs manual for the details of the syntax.)
The advantage of this approach is that the sequence between
-*- markers does not have to appear at the beginning; it can be commented out using the appropriate comment syntax. (If the file starts with
#!, the mode can also be specified in the second line.)
Lastly, the language of a module can be specified as part of the import syntax. This lets you use files as modules without having to edit them at all, which may be useful for shared files you cannot edit.
(vernacular:import facts :as :cl-yesql/sqlite ...)
You can use
import-as-package to import a Vernacular module as a Lisp package:
(vernacular:import-as-package :facts :from "sqlite.sql" :binding :all-functions)
This will bind
(If your Lisp supports hierarchical packages, you might find
import-as-subpackage more useful.)
Vernacular supports local imports using the
(vernacular:with-imports (facts :from "sqlite.sql" :binding :all-functions) ...)
This is local in extent, but it supports all the same options as
binding clause of an import form is actually a DSL for import sets, loosely based on R6RS.
A list of names is just a list of imports:
(x y z) ; x, y, and z as variables
Names can be aliased:
((x :as eks) (y :as why) (z :as zed))
You can import the same binding more than once under different aliases:
((x :as x1) (x :as x2))
Names can be namespaced:
;; x as a variable, y as a function, z as a macro (x #'y (macro-function z))
What this does depends on whether the language uses namespaces. If it exports a function named
y, that is what you get. If it doesn’t use namespaces, then you get the variable
y in the function namespace.
Namespaces and renaming can be combined:
((y :as #'why) (#'z :as zed))
Now the variable named
y is being imported as a function named
why, and the function named
z is being imported as a variable named
There are shorthands for importing everything:
:all ; all the variables, as variables :all-functions ; all the functions, as functions :all-setters ; all the setf functions :all-as-functions ; all the exports, as functions :all-as-setters ; all the exports, as setters
These can be combined, with each other and with regular names:
(:all-functions :all-setters x y z)
You can limit these using
(:except :all x y) ; All variables except x and y.
You can add a prefix to everything:
; Import all functions, adding sql- as a prefix. (:prefix :all-functions sql-)
Or you can drop a prefix:
; Import all functions, removing any get- prefix. (:drop-prefix :all-functions get-)
In Vernacular, a language is just a package. The package exports a reader and an expander. The function bound to the symbol named
read-module is the package reader. The macro bound to the symbol named
module-progn is the package expander.
The important thing: when the package’s reader is called, that same package is also bound as the current package. It is then the responsibility of the reader to make sure any symbols it reads in, or inserts into the expansion, are interned in the correct package. (There is a utility for this,
(There is one exception to the rule of language=package. If another package exists, having the same name, but ending in
-user, and this other package inherits from the original package, then this user package is the package that is made current while reading (and expanding). E.g. a file beginning with
#lang cl would actually be read in using the
cl-user package, not the
cl package itself.)
Note that the reader is responsible for returning a single form, which is the module. That is, the form returned by the package reader should already be wrapped in the appropriate
module-progn. The exported binding for
module-progn is only looked up when the language is being used as the expander for a meta-language.
(Meta-languages are for language authors who want to reuse an existing syntax.)
Any package can be used as a language, but it can only be used as a hash lang if its name is limited to certain characters (
[a-zA-Z0-9/_+-]). (Of course this name can also be a (global) nickname.)
(Note that package names are absolute, even on a Lisp that supports package-local nicknames.)
It is recommended, although not required, that your language package inherit from
vernacular/cl rather than from
cl. The result is the same, except that
vernacular/cl shadows Common Lisp’s binding and definition forms so they can be locally rebound by language implementations.
The package must export a binding for both
read-module, a function, and
module-progn, a macro.
If the syntax of your language makes it possible to determine exports statically, you should also define and export
static-exports. If your language defines
static-exports, then Vernacular can statically check the validity of import forms.
(This also has implications for phasing. If your language doesn’t provide a
static-exports binding, then the only way Vernacular can expand a request to import everything from a module is by loading that module at compile time to get a list of its exports.)
Most of the time, your language’s package expander will return a
(vernacular:simple-module (#'moo) (defun make-moo (o) (concat "M" (make-string o :initial-element #\o))) (defun moo (&optional (o 2)) (print (make-moo o))))
This exports a single name,
moo, bound to a function that says “Moo” with a varying amount of “oo”.
You can define and use macros in a simple module (with
defmacro), but you can’t export them, and they have to come first. These are limitations you can get around, but you’ll need something more complex (see below).
Vernacular’s module system is based on the generic functions
Export names and packages
Vernacular allows importing and exporting macros. The ability to export macros only becomes useful in the presence of macro hygiene. After experimenting, I have concluded that the right thing, if you want your language to support macros, is to embed a hygiene-compatible language in Lisp, and then compile your language to that.
I’m not being flippant. Embedding a hygiene-compatible language in CL is not just doable; it’s already been done. As a proof of concept, I have adapted Pascal’s Costanza’s hygiene-compatible implementation of ISLISP in Common Lisp (“Core Lisp”) to work with Vernacular’s module system. This version of Core Lisp lives in its own repository.