the-cost-of-nothing

2017-06-30

A LISP programmer knows the value of everything, but the cost of nothing.

--- Alan Perlis

This library attempts to measure the execution time of Lisp code using only portable mechanisms, i.e. get-internal-run-time. Using some macrology and sampling, the measurements can be accurate to the nanosecond.

Apart from the benchmark utilities, the library features a test suite that reports the runtime of various Common Lisp features like CLOS, function calls, hashtables, garbage collection and many others. This may give valuable performance insights, e.g. for which size hashtables outperform linked lists on a given implementation.

I hope you find it useful. If you have new ideas for benchmarks, or suggestions on how to improve existing ones, feel free to contact me.

Usage

To run all benchmarks, simply execute

(asdf:test-system :the-cost-of-nothing)

To obtain the execution time of an expression in seconds, type

(benchmark EXPRESSION)

To measure only certain parts of an expression, type

(nested-benchmark
  (foo)
  (bar)
  (benchmark SUBEXPRESSION1)
  (let ((a (baz)))
    (benchmark SUBEXPRESSION2)))
Furthermore the library exports the following functions
  • MEASURE-EXECUTION-TIME
  • MEASURE-EXECUTION-TIME-OF-THUNK

Implementation

A statement like

(nested-benchmark
 (loop for key across keys do
   (benchmark (gethash key table))))

is a macro that expands into something like

(measure-execution-time
 (lambda (#:iterations723)
   (loop :repeat #:iterations723
         :do (loop for key across keys
                   do (touch (gethash key table)))))
 :overhead
 (lambda (#:iterations723)
   (loop :repeat #:iterations723
         :do (loop for key across keys
                   do (touch nil)))))

The function TOUCH is explicitly declared notinline, to prevent compiler optimization. The function MEASURE-EXECUTION-TIME invokes its argument functions with higher and higher numbers of iterations, until the difference between them is significant.

Remember

There are lies, damned lies, and benchmarks.

Author
Marco Heisig <marco.heisig@fau.de>
License
GPLv3
Categories
benchmark