40Ants-CI - Github Workflow Generator

This is a small utility, which can generate GitHub workflows for Common Lisp projects.

It generates workflow for running tests and building docs. These workflows use 40ants/run-tests and 40ants/build-docs actions and SBLint to check code for compilation errors.

40ANTS-CI ASDF System Details

• Description: A tool simplify continuous deployment for Common Lisp projects.
• Licence: BSD
• Author: Alexander Artemenko
• Homepage: https://40ants.com/ci/
• Source control: GIT
• Depends on: alexandria, serapeum, str, yason

Reasons to Use

• This system hides all entrails related to caching.
• Includes a few ready to use job types.
• Custom job types can be defined and distributed as separate ASDF systems.
• You don't have to write YAML anymore!

Quickstart

This system allows you to define workflows in the lisp code. The best way is to make these definitions a part of your ASDF system. This way 40ants-ci (1 2) will be able to automatically understand for which system it builds a workflow.

Each workflow consists of jobs and each job is a number of steps.

There are three predefine types of jobs and you can create your own. Predefined jobs allows to reuse steps in multiple CL libraries.

In next examples, I'll presume you are writing code in a file which is the part of the package inferred ASDF system EXAMPLE/CI. A file should have the following header:

(defpackage #:example/ci
  (:use #:cl)
  (:import-from #:40ants-ci/workflow
                #:defworkflow)
  (:import-from #:40ants-ci/jobs/linter)
  (:import-from #:40ants-ci/jobs/run-tests)
  (:import-from #:40ants-ci/jobs/docs))

Job Types

Autotag

This job is automates git tag placement on the commit where you have changed the ChangeLog.md.

This can be a useful to automate package deployment and releases. You update the changelog, a job pushes a new git tag and the next action triggers on this tag and build a release.

Or you if you publish your library at Quicklisp distribution, then you can change it's source type to the latest-github-tag to provide more stable releases to your users. This way you commits into master will be ignored until you change the changelog and git tag will be pushed. Here is an example how to setup this kind of quicklisp project source.

(defworkflow release
  :on-push-to "master"
  :jobs ((40ants-ci/jobs/autotag:autotag)))

function 40ants-ci/jobs/autotag:autotag &key (filename *default-filename*) (regex *default-regex*) (tag-prefix *default-tag-prefix*) (token-pattern *default-token-pattern*) env

Creates a job which will run autotagger to create a new git tag for release.

class 40ants-ci/jobs/autotag:autotag (job)

This type of the job created a git tag when finds a new tag in specified file.

Linter

The simplest job type is linter. It loads a

(defworkflow linter
  :on-pull-request t
  :jobs ((40ants-ci/jobs/linter:linter)))

When you'll hit C-c C-c on this definition, it will generate .github/workflows/linter.yml with following content:

{
  "name": "LINTER",
  "on": {
    "pull_request": null
  },
  "jobs": {
    "linter": {
      "runs-on": "ubuntu-latest",
      "env": {
        "OS": "ubuntu-latest",
        "QUICKLISP_DIST": "quicklisp",
        "LISP": "sbcl-bin"
      },
      "steps": [
        {
          "name": "Checkout Code",
          "uses": "actions/checkout@v4"
        },
        {
          "name": "Setup Common Lisp Environment",
          "uses": "40ants/setup-lisp@v4",
          "with": {
            "asdf-system": "example"
          }
        },
        {
          "name": "Install SBLint",
          "run": "qlot exec ros install cxxxr/sblint",
          "shell": "bash"
        },
        {
          "name": "Run Linter",
          "run": "qlot exec sblint example.asd",
          "shell": "bash"
        }
      ]
    }
  }
}

Here you can see, a few steps in the job:

1. Checkout the code.
2. Install Roswell & Qlot using 40ants/setup-lisp action.
3. Install SBLint.
4. Run linter for example.asd.

Another interesting thing is that this workflow automatically uses ubuntu-latest OS, Quicklisp and sbcl-bin Lisp implementation. Later I'll show you how to redefine these settings.

class 40ants-ci/jobs/linter:linter (lisp-job)

Critic

This job is similar to linter, but instead of SBLint it runs Lisp Critic.

Lisp Critic is a program which advices how to make you Common Lisp code more idiomatic, readable and performant. Also, sometimes it might catch logical errors in the code.

Here is how you can add this job type in your workflow:

(defworkflow ci
  :on-pull-request t
  :jobs ((40ants-ci/jobs/critic:critic)))

Also, you might combine this job together with others, for example, with linter:

(defworkflow ci
  :on-pull-request t
  :jobs ((40ants-ci/jobs/linter:linter)
         (40ants-ci/jobs/critic:critic)))

and they will be executed in parallel. See docs on 40ants-ci/jobs/critic:critic function to learn about supported arguments.

Running Tests

Another interesting job type is 40ants-ci/jobs/run-tests:run-tests (1 2).

When using this job type, make sure, your system runs tests on (ASDF:TEST-SYSTEM :system-name) call and signals error if something went wrong.

(defworkflow ci
  :on-push-to "master"
  :by-cron "0 10 * * 1"
  :on-pull-request t
  :jobs ((40ants-ci/jobs/run-tests:run-tests
          :coverage t)))

Here I've added a few options to the workflow:

• by-cron - sets a schedule.
• on-push-to - defines a branch or branches to track.

It will generate .github/workflows/ci.yml with following content:

{
  "name": "CI",
  "on": {
    "push": {
      "branches": [
        "master"
      ]
    },
    "pull_request": null,
    "schedule": [
      {
        "cron": "0 10 * * 1"
      }
    ]
  },
  "jobs": {

    "run-tests": {
      "runs-on": "ubuntu-latest",
      "env": {
        "OS": "ubuntu-latest",
        "QUICKLISP_DIST": "quicklisp",
        "LISP": "sbcl-bin"
      },
      "steps": [
        {
          "name": "Checkout Code",
          "uses": "actions/checkout@v4"
        },
        {
          "name": "Setup Common Lisp Environment",
          "uses": "40ants/setup-lisp@v4",
          "with": {
            "asdf-system": "example"
          }
        },
        {
          "name": "Run Tests",
          "uses": "40ants/run-tests@v2",
          "with": {
            "asdf-system": "example",
            "coveralls-token": "${{ secrets.github_token }}"
          }
        }
      ]
    }
  }
}

The result is similar to the workflow generated for Linter, but uses 40ants/setup-lisp action at the final step.

Also, I've passed an option :coverage t to the job. Thus coverage report will be uploaded to Coveralls.io automatically.

Defining a test Matrix

Lisp has many implementations and can be used on multiple platforms. Thus it is a good idea to test our software on many combinations of OS and lisp implementations. Workflow generator makes this very easy.

Here is an example of workflow definition with three dimentional matrix. It not only tests a library under different lisps and OS, but also checks if it works with the latest Quicklisp and Ultralisp distributions:

(defworkflow ci
  :on-pull-request t
  :jobs ((run-tests
          :os ("ubuntu-latest"
               "macos-latest")
          :quicklisp ("quicklisp"
                      "ultralisp")
          :lisp ("sbcl-bin"
                 "ccl-bin"
                 "allegro"
                 "clisp"
                 "cmucl")
          :exclude (;; Seems allegro is does not support 64bit OSX.
                    ;; Unable to install it using Roswell:
                    ;; alisp is not executable. Missing 32bit glibc?
                    (:os "macos-latest" :lisp "allegro")))))

Multiple jobs

Besides a build matrix, you might specify a multiple jobs of the same type, but with different parameters:

(defworkflow ci
  :on-push-to "master"
  :on-pull-request t
  :jobs ((run-tests
          :lisp "sbcl-bin")
         (run-tests
          :lisp "ccl-bin")
         (run-tests
          :lisp "allegro")))

This will generate a workflow with three jobs: "run-tests", "run-tests-2" and "run-tests-3".

Meaningful names might be specified as well:

(defworkflow ci
  :on-push-to "master"
  :on-pull-request t
  :jobs ((run-tests
          :name "test-on-sbcl"
          :lisp "sbcl-bin")
         (run-tests
          :name "test-on-ccl"
          :lisp "ccl-bin")
         (run-tests
          :name "test-on-allegro"
          :lisp "allegro")))

Here is how these jobs will look like in the GitHub interface:

Building Docs

Third predefined job type is 40ants-ci/jobs/docs:build-docs (1 2). It uses 40ants/build-docs action and will work only if your ASDF system uses a documentation builder supported by 40ants/docs-builder.

To build docs on every push to master, just use this code:

(defworkflow docs
  :on-push-to "master"
  :jobs ((40ants-ci/jobs/docs:build-docs)))

It will generate .github/workflows/docs.yml with following content:

{
  "name": "DOCS",
  "on": {
    "push": {
      "branches": [
        "master"
      ]
    }
  },
  "jobs": {
    "build-docs": {
      "runs-on": "ubuntu-latest",
      "env": {
        "OS": "ubuntu-latest",
        "QUICKLISP_DIST": "quicklisp",
        "LISP": "sbcl-bin"
      },
      "steps": [
        {
          "name": "Checkout Code",
          "uses": "actions/checkout@v4"
        },
        {
          "name": "Setup Common Lisp Environment",
          "uses": "40ants/setup-lisp@v4",
          "with": {
            "asdf-system": "example",
            "qlfile-template": ""
          }
        },
        {
          "name": "Build Docs",
          "uses": "40ants/build-docs@v1",
          "with": {
            "asdf-system": "example"
          }
        }
      ]
    }
  }
}

Caching

To significantly speed up our tests, we can cache installed Roswell, Qlot and Common Lisp fasl files.

To accomplish this task, you don't need to dig into GitHub's docs anymore! Just add one line :cache t to your workflow definition:

(defworkflow docs
  :on-push-to "master"
  :cache t
  :jobs ((40ants-ci/jobs/docs:build-docs)))

Here is the diff of the generated workflow file. It shows steps, added automatically:

modified   .github/workflows/docs.yml
@@ -20,13 +20,40 @@
           "name": "Checkout Code",
           "uses": "actions/checkout@v4"
         },
+        {
+          "name": "Grant All Perms to Make Cache Restoring Possible",
+          "run": "sudo mkdir -p /usr/local/etc/roswell\n                 sudo chown \"${USER}\" /usr/local/etc/roswell\n                 # Here the ros binary will be restored:\n                 sudo chown \"${USER}\" /usr/local/bin",
+          "shell": "bash"
+        },
+        {
+          "name": "Get Current Month",
+          "id": "current-month",
+          "run": "echo \"::set-output name=value::$(date -u \"+%Y-%m\")\"",
+          "shell": "bash"
+        },
+        {
+          "name": "Cache Roswell Setup",
+          "id": "cache",
+          "uses": "actions/cache@v3",
+          "with": {
+            "path": "qlfile\n                           qlfile.lock\n                           /usr/local/bin/ros\n                           ~/.cache/common-lisp/\n                           ~/.roswell\n                           /usr/local/etc/roswell\n                           .qlot",
+            "key": "${{ steps.current-month.outputs.value }}-${{ env.cache-name }}-ubuntu-latest-quicklisp-sbcl-bin-${{ hashFiles('qlfile.lock') }}"
+          }
+        },
+        {
+          "name": "Restore Path To Cached Files",
+          "run": "echo $HOME/.roswell/bin >> $GITHUB_PATH\n                 echo .qlot/bin >> $GITHUB_PATH",
+          "shell": "bash",
+          "if": "steps.cache.outputs.cache-hit == 'true'"
+        },
         {
           "name": "Setup Common Lisp Environment",
           "uses": "40ants/setup-lisp@v4",
           "with": {
             "asdf-system": "40ants-ci",
             "qlfile-template": ""
-          }
+          },
+          "if": "steps.cache.outputs.cache-hit != 'true'"
         },
         {

Details

TODO: I have to write a few chapters with details on additional job's parameters and a way how to create new job types.

But for now, I want to show a small example, how to define a workflow with a job which takes care about lisp installation and then calls a custom step:

(defworkflow ci
  :on-push-to "master"
  :by-cron "0 10 * * 1"
  :on-pull-request t
  :cache t
  :jobs ((40ants-ci/jobs/lisp-job:lisp-job :name "check-ros-config"
                                           :lisp "ccl-bin"
                                           :steps ((40ants-ci/steps/sh:sh "Show Roswell Config"
                                                                          "ros config")))))

Here we are using the class 40ants-ci/jobs/lisp-job:lisp-job which is base for most classes in this ASDF system and pass a custom 40ants-ci/steps/sh:sh (1 2) step to it. This step will be called after the repostory checkout and CCL-BIN lisp installation. so, thus when this step will run ros config command, it will output something like that:

asdf.version=3.3.5.3
ccl-bin.version=1.12.2
setup.time=3918000017
sbcl-bin.version=2.4.1
default.lisp=ccl-bin

Possible subcommands:
set
show

Pay attention to the NAME argument of 40ants-ci/jobs/lisp-job:lisp-job class. If you omit it, then default "lisp-job" name will be used.

API

40ANTS-CI

package 40ants-ci

Functions

function 40ants-ci:generate system &key path

Generates GitHub workflow for given ASDF system.

This function searches workflow definitions in all packages of the given ASDF system.

If PATH argument is not given, workflow files will be written to .github/workflow/ relarive to the SYSTEM.

40ANTS-CI/GITHUB

package 40ants-ci/github

Generics

generic-function 40ants-ci/github:generate obj path

generic-function 40ants-ci/github:prepare-data obj

Variables

variable 40ants-ci/vars:*current-system* -unbound-

When workflow is generated for ASDF system, this variable will contain a primary ASDF system.

40ANTS-CI/JOBS/AUTOTAG

package 40ants-ci/jobs/autotag

Classes

AUTOTAG

class 40ants-ci/jobs/autotag:autotag (job)

This type of the job created a git tag when finds a new tag in specified file.

Readers

reader 40ants-ci/jobs/autotag:filename (autotag) (:filename = *default-filename*)

File where to search for version numbers.

reader 40ants-ci/jobs/autotag:regex (autotag) (:regex = *default-regex*)

Regexp used to extract version numbers.

reader 40ants-ci/jobs/autotag:tag-prefix (autotag) (:tag-prefix = *default-tag-prefix*)

Tag prefix.

reader 40ants-ci/jobs/autotag:token-pattern (autotag) (:token-pattern = *default-token-pattern*)

Auth token pattern.

Functions

function 40ants-ci/jobs/autotag:autotag &key (filename *default-filename*) (regex *default-regex*) (tag-prefix *default-tag-prefix*) (token-pattern *default-token-pattern*) env

Creates a job which will run autotagger to create a new git tag for release.

40ANTS-CI/JOBS/CRITIC

package 40ants-ci/jobs/critic

Classes

CRITIC

class 40ants-ci/jobs/critic:critic (lisp-job)

Readers

reader 40ants-ci/jobs/critic:asdf-systems (critic) (:asdf-systems)

Critic can validate more than one system, but for the base class we need provide only one.

reader 40ants-ci/jobs/critic:ignore-critiques (critic) (:ignore-critiques)

A list strigns with names of critiques to ignore.

Functions

function 40ants-ci/jobs/critic:critic &key asdf-systems asdf-version ignore-critiques env

Creates a job which will run Lisp Critic for given ASDF systems.

If argument ASDF-SYSTEMS is NIL, it will use ASDF system to which current lisp file is belong.

You may also provide ASDF-VERSION argument. It should be a string. By default, the latest ASDF version will be used.

40ANTS-CI/JOBS/DOCS

package 40ants-ci/jobs/docs

Classes

BUILD-DOCS

class 40ants-ci/jobs/docs:build-docs (lisp-job)

Builds documentation and uploads it to GitHub using "40ants/build-docs" github action.

Readers

reader 40ants-ci/jobs/docs:error-on-warnings (build-docs) (:error-on-warnings = t)

Functions

function 40ants-ci/jobs/docs:build-docs &key asdf-system asdf-version (error-on-warnings t) env

Creates a job of class build-docs.

40ANTS-CI/JOBS/JOB

package 40ants-ci/jobs/job

Classes

JOB

class 40ants-ci/jobs/job:job ()

Readers

reader 40ants-ci/jobs/job:exclude (job) (:exclude = nil)

A list of plists denoting matrix combinations to be excluded.

reader 40ants-ci/jobs/job:explicit-steps (job) (:steps = nil)

This slot holds steps given as a STEPS argument to a job constructor. Depending on a job class, it might add additional steps around these explicit steps.

reader 40ants-ci/jobs/job:job-env (job) (:env = nil)

An alist of environment variables and their values to be added on job level. Values are evaluated in runtime.

reader 40ants-ci/jobs/job:name (job) (:name)

If this name was not given in constructor, then name will be lowercased name of the job class.

reader 40ants-ci/jobs/job:os (job) (:OS = "ubuntu-latest")

reader 40ants-ci/jobs/job:permissions (job) (:permissions = nil)

A plist of permissions need for running the job.

These permissions will be bound to secrets.GITHUB_TOKEN variable. Use default-initargs to override permissions in subclasses:

(:default-initargs
 :permissions '(:content "write"))

Generics

generic-function 40ants-ci/jobs/job:make-env job

generic-function 40ants-ci/jobs/job:make-matrix job

generic-function 40ants-ci/jobs/job:make-permissions job

Should return an alist with mapping from string to string where keys are scopes and values are permission names. Default method generates this alist from the plist of job's "permissions" slot.

generic-function 40ants-ci/jobs/job:steps job

generic-function 40ants-ci/jobs/job:use-matrix-p job

40ANTS-CI/JOBS/LINTER

package 40ants-ci/jobs/linter

Classes

LINTER

class 40ants-ci/jobs/linter:linter (lisp-job)

Readers

reader 40ants-ci/jobs/linter:asdf-systems (linter) (:asdf-systems = nil)

Linter can validate more than one system, but for the base class we need provide only one.

reader 40ants-ci/jobs/linter:check-imports (linter) (:check-imports = nil)

Linter will check for missing or unused imports of package-inferred systems.

Functions

function 40ants-ci/jobs/linter:linter &key asdf-systems asdf-version check-imports env

Creates a job which will run SBLint for given ASDF systems.

If no ASD files given, it will use all ASD files from the current ASDF system.

40ANTS-CI/JOBS/LISP-JOB

package 40ants-ci/jobs/lisp-job

Classes

LISP-JOB

class 40ants-ci/jobs/lisp-job:lisp-job (job)

This job checkouts the sources, installs Roswell and Qlot. Also, it caches results between runs.

Readers

reader 40ants-ci/jobs/lisp-job:asdf-system (lisp-job) (:asdf-system = nil)

reader 40ants-ci/jobs/lisp-job:asdf-version (lisp-job) (:asdf-version = nil)

ASDF version to use when setting up Lisp environment. If NIL, then the latest will be used.

reader 40ants-ci/jobs/lisp-job:lisp (lisp-job) (:LISP = "sbcl-bin")

reader 40ants-ci/jobs/lisp-job:qlfile (lisp-job) (:qlfile = nil)

reader 40ants-ci/jobs/lisp-job:qlot-version (lisp-job) (:qlot-version = nil)

Qlot version to use when setting up Lisp environment. If NIL, then will be used version, pinned in setup-lisp github action.

reader 40ants-ci/jobs/lisp-job:quicklisp (lisp-job) (:QUICKLISP = "quicklisp")

reader 40ants-ci/jobs/lisp-job:roswell-version (lisp-job) (:roswell-version = nil)

Roswell version to use when setting up Lisp environment. If NIL, then will be used version, pinned in setup-lisp github action.

40ANTS-CI/JOBS/RUN-TESTS

package 40ants-ci/jobs/run-tests

Classes

RUN-TESTS

class 40ants-ci/jobs/run-tests:run-tests (lisp-job)

This job test runs tests for a given ASDF system.

Readers

reader 40ants-ci/jobs/run-tests:coverage (run-tests) (:coverage = nil)

reader 40ants-ci/jobs/run-tests:custom (run-tests) (:custom = nil)

Functions

function 40ants-ci/jobs/run-tests:run-tests &rest rest &key coverage qlfile asdf-system asdf-version os quicklisp lisp exclude custom env

Creates a job step of class run-tests.

40ANTS-CI/STEPS/ACTION

package 40ants-ci/steps/action

Classes

ACTION

class 40ants-ci/steps/action:action (step)

Readers

reader 40ants-ci/steps/action:action-args (action) (:args)

A plist to be passed as "with" dictionary to the action.

reader 40ants-ci/steps/action:uses (action) (:uses)

Functions

function 40ants-ci/steps/action:action name uses &rest args &key id if env &allow-other-keys

40ANTS-CI/STEPS/SH

package 40ants-ci/steps/sh

Classes

SH

class 40ants-ci/steps/sh:sh (step)

Readers

reader 40ants-ci/steps/sh:command (sh) (:command)

reader 40ants-ci/steps/sh:shell (sh) (:shell = *default-shell*)

Functions

function 40ants-ci/steps/sh:sh name command &key id if (shell *default-shell*) env

Macros

macro 40ants-ci/steps/sh:sections &body body

Returns a string with a bash script where some parts are grouped.

In this example we have 3 sections:

(sections
      ("Help Argument"
       "qlot exec cl-info --help")
      ("Version Argument"
       "qlot exec cl-info --version")
      ("Lisp Systems Info"
       "qlot exec cl-info"
       "qlot exec cl-info cl-info defmain"))

It will be compiled into:

echo ::group::Help Argument
qlot exec cl-info --help
echo ::endgroup::
echo ::group::Version Argument
qlot exec cl-info --version
echo ::endgroup::
echo ::group::Lisp Systems Info
qlot exec cl-info
qlot exec cl-info cl-info defmain
echo ::endgroup::

40ANTS-CI/STEPS/STEP

package 40ants-ci/steps/step

Classes

STEP

class 40ants-ci/steps/step:step ()

Readers

reader 40ants-ci/steps/step:env (step) (:env = nil)

An alist of environment variables.

reader 40ants-ci/steps/step:step-id (step) (:id = nil)

reader 40ants-ci/steps/step:step-if (step) (:if = nil)

reader 40ants-ci/steps/step:step-name (step) (:name = nil)

40ANTS-CI/UTILS

package 40ants-ci/utils

Generics

generic-function 40ants-ci/utils:system-packages system

Returns a list of packages created by ASDF system.

Default implementation returns a package having the same name as a system and all packages matched to package-inferred subsystems:

CL-USER> (docs-builder/utils:system-packages :docs-builder)
(#<PACKAGE "DOCS-BUILDER">
 #<PACKAGE "DOCS-BUILDER/UTILS">
 #<PACKAGE "DOCS-BUILDER/GUESSER">
 #<PACKAGE "DOCS-BUILDER/BUILDERS/GENEVA/GUESSER">
 #<PACKAGE "DOCS-BUILDER/BUILDER">
 #<PACKAGE "DOCS-BUILDER/BUILDERS/MGL-PAX/GUESSER">
 #<PACKAGE "DOCS-BUILDER/DOCS">
 #<PACKAGE "DOCS-BUILDER/BUILDERS/MGL-PAX/BUILDER">)

Functions

function 40ants-ci/utils:alistp list

Test wheather LIST argument is a properly formed alist.

In this library, alist has always a string as a key. Because we need them to have this form to serialize to JSON propertly.

(alistp '(("cron" . "0 10 * * 1"))) -> T (alistp '((("cron" . "0 10 * * 1")))) -> NIL

function 40ants-ci/utils:current-system-name

function 40ants-ci/utils:dedent text

Removes common leading whitespace from each string.

A few examples:

(dedent "Hello
          World
          and all Lispers!")

"Hello
World
and all Lispers!"

(dedent "
    Hello
    World
    and all Lispers!")

"Hello
World
and all Lispers!"

(dedent "This is a code:

              (symbol-name :hello-world)

          it will output HELLO-WORLD.")

"This is a code:

    (symbol-name :hello-world)

it will output HELLO-WORLD."

function 40ants-ci/utils:ensure-list-of-plists data

function 40ants-ci/utils:ensure-primary-system system

function 40ants-ci/utils:make-github-workflows-path system

function 40ants-ci/utils:plist-to-alist plist &key (string-keys t) (lowercase t)

Make an alist from a plist PLIST.

By default, transforms keys to lowercased strings

function 40ants-ci/utils:plistp list

Test wheather LIST is a properly formed plist.

function 40ants-ci/utils:single list

Test wheather LIST contains exactly 1 element.

function 40ants-ci/utils:to-json data

40ANTS-CI/VARS

package 40ants-ci/vars

Variables

variable 40ants-ci/vars:*current-system* -unbound-

When workflow is generated for ASDF system, this variable will contain a primary ASDF system.

variable 40ants-ci/vars:*use-cache* nil

Workflow will set this variable when preparing the data or YAML generation.

---

[generated by 40ANTS-DOC]