Lispy fixtures

Have you ever wanted to test a function with all possible combination of some arguments? Or have you even wanted to create fixtures that should be initialized with different values? cl-fixtures tries to enable you to do these things.

1Usage

The primary feature of cl-fixtures are its fixtures and parameters.

1.1Fixtures

Fixtures are objects that are used to test a piece of software. The simplest wayto use a fixture would be doing a LET with a certain value. However after sometime you might want to use a certain fixture in multiple places so you define aglobal variable. But the fixture might have some state and you need a fresh onefor each test, how to do this?

This is were cl-fixtures can help. Lets first define some fixtures.

1.1.1Defining fixtures

The easiest way to define a fixture is with the DEFINE-SIMPLE-FIXTURE macro.

However it might happen that a certain fixture can have multiple values, and each test should be ran with all options. In this case it is useful to use the DEFINE-SEQUENCE-FIXTURE macro.

All these macro's build on a 'master' macro: DEFINE-FIXTURE (surprise). With this macro you can create difficult fixtures that do setup and teardown between options instead of just before and after yielding values.

DEFINE-SIMPLE-FIXTURE

Define a simple fixture that returns a single value.

Define a new simple fixture with the name NAME. This fixture will execute the BODY and CLEANUP as described in DEFINE-SEQUENCE-FIXTURE. However this time the body does not have to return a sequence but any value. This value is used as the only value of the fixture.

CL-FIXTURES> (list
               (flet ((simple-func () :item))
                      (define-simple-fixture simpl1 () nil (simple-func)))
               (collecting (with-fixtures (simpl1) (collect simpl1))))
 (SIMPL1 #(:item))

DEFINE-SEQUENCE-FIXTURE

Define a sequence fixture that returns a sequence.

Define a new sequence fixture with the name NAME. This fixture will execute BODY with the given fixtures FIXTURES (which should be a list in the format described in WITH-FIXTURES) and it should return a sequence. Each element in this sequence is used as if it was yielded from the fixture. This means that if a sequence fixture that returns #(1 2) is used the fixture will have two values: 1 and 2.

After the fixtures is used the function given in CLEANUP is called, if CLEANUP is NIL it will be ignored. This function should take exactly one argument that is the result returned by the body. If there was a condition in the body the cleanup function is not called. The cleanup is always called if the body did return.

CL-FIXTURES> (list
               (define-sequence-fixture seq1 () nil (list 1 2 3))
               (collecting (with-fixtures (seq1) (collect seq1))))
 (SEQ1 #(1 2 3))

CL-FIXTURES> (let ((exec 0))
               (define-sequence-fixture seq2 ((item seq1))
                   (lambda (_) (declare (ignore _)) (incf exec))
                 (list item 4 5))
               (list (collecting (with-fixtures (seq2) (collect seq2)))
                     exec))
 (#(1 4 5 2 4 5 3 4 5) 3)

DEFINE-FIXTURE

Define a new normal fixture.

This is the basic function to define a new fixture named NAME with the given body BODY. This fixture can use the given fixtures FIXTURES which should be a list in the same format as specified in the WITH-FIXTURES macro.

Within the given body a variable is bound by the name given in MAPPER. This is the continuation of the fixtures. So to yield a value one must call the function in this variable (by doing a funcall or something). This function accepts exactly one argument: the value that should be yielded.

The return value of BODY is ignored.

CL-FIXTURES> (define-fixture test-fixture cont () (funcall cont 5))
 TEST-FIXTURE

CL-FIXTURES> (list (define-fixture test-fixture cont () (funcall cont 5))
                   (define-fixture test-fixture cont () (funcall cont 6))
                   (collecting (with-fixtures (test-fixture) (collect test-fixture))))
 (TEST-FIXTURE TEST-FIXTURE #(6))

1.1.2Using fixtures

So now you have a few fixtures, but how to use them? The main way to use thefixtures is by using the WITH-FIXTURES macro. It takes a list of fixtures anda body and executes the body with all possible combinations of the givenfixtures (it makes calculates the cartesian product).

However this means that every fixture will get recalculated every time you use it. Lets say we have a fixture DATABASE that is used by the USERS fixture. We might a database and users in our body, but users should use the same database as the USERS fixture. This will NOT happen if you use the WITH-FIXTURES macro, but fortunately there is the WITH-CACHED-FIXTURES. If we use this fixture like (with-cached-fixture (database users) ...) we will have the desired behavior.

WITH-FIXTURES

Execute the given BODY with the given FIXTURES bound in a implicit progn.

The FIXTURES variable should be a list where each item can be of the following forms:

• A symbol designating a fixture. This means bind the result of the fixture tothe variable with the same name as the fixture.
• A list with two elements like (BIND NAME) where BIND will be bound tothe result of the fixture designated by NAME

CL-FIXTURES> (progn (define-sequence-fixture simple () nil '(1 2))
                    (equalp (collecting (with-fixtures (simple)
                                          (collect simple)))
                            (collecting (with-fixtures ((a simple))
                                          (collect a)))))
 T

Multiple fixtures can occur multiple times in the fixture list, and this behaves like one would suspect. However note that you have to alias at least one of them:

CL-FIXTURES> (progn (define-sequence-fixture simple () nil '(1 2))
                    (collecting (with-fixtures ((a simple) simple)
                                  (collect a simple))))
 #((1 1) (1 2) (2 1) (2 2))

This means that every fixture is evaluated every time it is being used. Take for example this example:

CL-FIXTURES> (progn
               (define-fixture rand mapper () (funcall mapper (random 10.0)))
               (define-fixture rand2 mapper (rand) (funcall mapper rand))
               (collecting (with-fixtures (rand rand2) (collect (= rand rand2)))))
 #(NIL)

If this is not the behavior you want look at the WITH-CACHED-FIXTURES macro.

The return value of the BODY is ignored.

WITH-CACHED-FIXTURES

Execute the given BODY with the given FIXTURES with the values of the fixtures cached.

The structure of FIXTURES is the same as in the WITH-FIXTURES macro. However this macro does caching of the named fixtures in order. There are few things to consider:

The first is that fixtures are evaluated, and therefore cached, in order. For example:

CL-FIXTURES> (progn
               (define-fixture first mapper () (funcall mapper (random 10.0)))
               (define-fixture second mapper (first) (funcall mapper first))
               (list (collecting (with-cached-fixtures (first second) (collect (= first second))))
                     (collecting (with-cached-fixtures (second first) (collect (= first second))))
                     (collecting (with-fixtures (second first) (collect (= first second))))))
 (#(T) #(NIL) #(NIL))

In the first case the FIRST fixture is cached and replaced by the cached version when we are executing the SECOND fixture. However as the fixture list when defining the fixtures is a shorthand for WITH-FIXTURES it is not cached when we reverse the order, so it has the same result as if WITH-FIXTURES was used. So if only executing the fixture once is important for more than just performance simply wrap all the code in WITH-CACHED-FIXTURES.

The second thing to consider is the using the same fixture multiple times will NOT result a Cartesian product.

CL-FIXTURES> (progn
               (define-sequence-fixture test () nil '(1 2 3))
               (list (collecting (with-cached-fixtures ((a test) (b test))
                                   (collect a b)))
                     (collecting (with-cached-fixtures ((a test))
                                   (with-fixtures ((b test))
                                     (collect a b))))))
 (#((1 1) (2 2) (3 3)) #((1 1) (2 2) (3 3)))

1.1.3Removing fixtures

You can also remove fixtures by using the UNDEFINE-FIXTURE macro. Please notethat just uninterning the fixture name is not enough as the fixtures are savedin an internal hash-map.

If you try to use a fixture that is not defined this will result in a UNDEFINED-FIXTURE condition.

UNDEFINE-FIXTURE

Undefine the given fixture NAME.

If you try to undefine a fixture while it is in use the resulting behavior will be unspecified.

CL-FIXTURES> (list
               (define-simple-fixture fixture () nil :item)
               (collecting (with-fixtures (fixture) (collect fixture)))
               (undefine-fixture fixture)
               (incf-cl:signals-p undefined-fixture
                 (collecting (with-fixtures (fixture) (collect fixture)))))
 (fixture #(:item) fixture T)

UNDEFINED-FIXTURE

This condition will be signaled if a fixture is used that is undefined. This will only happen if this fixture is used during runtime.

1.2Parameterize

'Parameterizing' is like creating anonymous fixtures. However these anonymousfixtures are somewhat more strict than actual fixtures as no inheritance ispossible. The binding is very much like LET but instead of binding a singlevalue all possible values are bound. Once again the Cartesian product of alloptions is calculated.

Such a Cartesian product is not really traditional for parameterized tests. The more traditional behavior is to specify a sequence with different options for each element (this is the way pytest uses it). This behavior is also possible with the WITH-LOCKED-PARAMETERS macro, it is named this way as the options are fixed or locked.

WITH-PARAMETERS

Execute the given body BODY with the Cartesian product of the bindings.

The BINDINGS is of the form (NAME VALUE) where NAME is the name of the variable that will be bound in the body. The VALUE will be evaluated and should return a function or sequence.

If the value returns a function this function should take one argument which is the current continuation. This function should be called with the value that should be yielded. If the value returns a sequence this sequence will be mapped over, so NAME will be every item of this sequence.

This macro is essentially a way to make anonymous fixtures.

CL-FIXTURES> (collecting
               (with-parameters ((a (list 1 2))
                                 (b #(4 5 6))
                                 (c (lambda (cont) (funcall cont :next)
                                                   (funcall cont :item))))
                 (collect a b c)))
#((1 4 :next) (1 4 :item) (1 5 :next) (1 5 :item) (1 6 :next) (1 6 :item)
  (2 4 :next) (2 4 :item) (2 5 :next) (2 5 :item) (2 6 :next) (2 6 :item))

If no parameters are specified BODY will run once. If parameters are specified but one of the sequences has a length of zero or the function does not call the continuation BODY will not be executed.

CL-FIXTURES> (collecting (with-parameters () (collect :one)))
 #(:one)

CL-FIXTURES> (collecting (with-parameters ((a nil)) (collect a)))
 #()

CL-FIXTURES> (collecting (with-parameters ((a nil) (b '(1 2))) (collect a b)))
 #()

The return value of BODY is ignored.

CL-FIXTURES> (with-parameters () :value)
 NIL

WITH-LOCKED-PARAMETERS

Execute the given BODY with the given parameters named by NAMES.

The forms BODY will be executed with the variables in NAMES bound to the values in BINDINGS. This BINDINGS form should be of the form (combination-1 combination-2 ...). Each combination will be evaluated and should return a list of the length of the list specified in the NAMES variable. The first variable in NAMES will be bound to the first value in the combination list, and the second value to the second item and so on.

The return value of the BODY is not preserved.

CL-FIXTURES> (collecting
               (with-locked-parameters (a b)
                   ('(1 2)
                    (list 3 4))
                 (collect a b)))
 #((1 2) (3 4))

If no parameters are specified BODY will be executed once. If there are parameters specified but no bindings BODY will not be executed.

CL-FIXTURES> (collecting (with-locked-parameters () () (collect :one)))
 #(:one)

CL-FIXTURES> (collecting
               (with-locked-parameters (a) ()
                 (declare (ignore a))
                 (collect :one)))
 #()

Please note that the bindings and BODY are evaluated by turn. So first the first binding, the first time BODY the second biding, BODY for the second time and so on.

CL-FIXTURES> (with-output-to-string (str)
                 (with-locked-parameters (a) ((list (format str "Hello"))
                                              (list (format str "Bye")))
                   (declare (ignore a))
                   (format str " Thomas.~%")))
 "Hello Thomas.
Bye Thomas.
"

2Efficiency

Like any other library we have to talk a bit about efficiency. As this libraryfocuses on being used in a unit-testing setup it is not optimized for speed.Quite a few intermediated anonymous functions are used, some of which could beoptimized out. However this library tries to be fast and efficient enough to notbreak your test setup.

A Cartesian product can become quite large quite quickly as the size is equal to (reduce #'* (mapcar #'length sequences)). Therefore this library does not try to first calculate it but it executes the values directly. This has the advantage of not blowing up your memory usage, however it will use some stack depth.

Furthermore the user of this library must take caution when doing expensive calculations within the setup of the fixtures or within the body of a WITH-FIXTURES (or alike) macro. This because of the just mentioned size.

3Installation

Installation is possible by simply loading CL-FIXTURESfrom [Quicklisp](http://www.quicklisp.org/beta/).

  (ql:quickload :cl-fixtures)

This should work for all the supported lisp versions which are:

• sbcl
• ccl
• abcl
• ecl

4Contributing

Contributions are welcome. To do this simply start hacking away. Please writetests and documentation. To generate this README you should alterdocs/README.org and execute make docs. To include docstrings in the READMEuse the doc-of djula tag.

5Author

• Thomas Schaper (thomas@libremail.nl)

6Copyright

Copyright © 2017 Thomas Schaper (thomas@libremail.nl)

7License

Licensed under the MIT License.