burgled-batteries: A Common Lisp / Python Bridge

burgled-batteries provides a shim between Python (specifically, the CPython implementation of Python) and Common Lisp.

1Synopsis

  (asdf:load-system "burgled-batteries")
  (in-package #:burgled-batteries)
  (startup-python)
  
  (run "1+1") ; => 2
  
  (import "feedparser")
  (defpyfun "feedparser.parse" (thing))
  (documentation 'feedparser.parse 'function)
  ; => "Parse a feed from a URL, file, stream, or string"
  (feedparser.parse "http://pinterface.livejournal.com/data/atom")
  ; => #<HASH-TABLE>
  
  (shutdown-python)

2Why burgled-batteries

CLPython is great when it works. However, if you're using a low-resourcecomputer—perhaps an underpowered VPS, or a Linux box salvaged from the 90s—, orneed access to a Python library written in C, or there's a bug and you can't bebothered to narrow it down to a small test case, CLPython can't help you. Twoout of three of those are more your problem than CLPython's, but hey, I'm nothere to judge.

While a number of other Python-by-FFI options exist, burgled-batteries aims for a CLPython-esque level of integration. In other words, deep integration. You shouldn't have to care that the library you're using was written in Python—it should Just Work.

Certainly, b-b is not there yet. It may never be there completely. But we'll try, dagnabbit.

3Basic Type Mapping

Python objects are converted to a Lisp object where possible. Where aconversion is unknown, a pointer to the CPython object is returned (or, ifinside a refcnt barrier, a wrapper around the pointer which will become invalidupon exiting the barrier). In general, this mapping follows the lead ofCLPython.

Python Type	Lisp Type
Boolean	(member T NIL)
Integer, Long	Integer
Float	Double-float
Dict	Hashtable
Unicode	String
List	Adjustable vector
Tuple	List
Complex	Complex
ByteArray	Octet vector
Exception	Condition
<Unknown>	<pointer>

4CPython FFI

Anything dealing with the CPython API can be found in the PYTHON.CFFI package.See the docstring for that package for more information, as well as Python's C API.

5Calling Python from Lisp

IMPORT

Equivalent to "import <foo>" in Python.

RUN

Mimicks the RUN function from CLPython.

DEFPYFUN

Similar to CFFI's DEFCFUN—defines a Lisp function which calls aPython function. Ideally, this will eventually be obviated bysomething which uses Python introspection to figure out anapropriate argument list so you don't have to, but it'll get yougoing for now.

6Calling Lisp from Python

Not yet supported, but see ffi-callbacks.lisp for some experimentation and notesalong those lines.

7Avoid the Hassle of Reference Counts

Because dealing with reference counts is Just No Fun and Not Lispy At All, aswell as Inevitable—at some point there will be an object for which notranslation is known—, b-b provides multiple options to avoid dealing withrefcnts for those untranslatable pointers. See the macroCPYTHON:WITH-UNKNOWN-TRANSLATION-POLICY.

Note that this policy also also affects the EXCEPTION-* slots of PYTHON-CONDITION, and so they may or may not be available for inspection depending on the translation policy in effect and the manner of handling.

For example, under the default policy of :DISCARD, you would see something like:

(defun reveal-effect (c)
  (format t "~A~%" (slot-value c 'exception-type)))

(handler-bind ((python-condition #'reveal-effect))
  (burgled-batteries:run "1/0"))
; prints #.(SB-SYS:INT-SAP #X?????)

(handler-case (burgled-batteries:run "1/0")
  (python-condition (c) (reveal-effect c)))
; prints NIL

If you'd like access to Python types without a known translation, :BARRIER or :FINALIZE are highly recommended over :PASS-THROUGH. They do, however, come with some caveats which you should be aware of.

8Requirements

Burgled-batteries links with C code, and accordingly requires the appropriate headers and library files. On Debian-based systems, you can get these via:

  sudo apt-get install python-dev

9Testing

  (asdf:test-system "burgled-batteries")

Tests have been run under SBCL, Clozure CL, and CLISP.

10To Do

• Output redirection
• Callbacks
• Whole-module import (into own package?)
• Python object <-> CLOS mappings
• Provide facilities for user code to define own Lisp-Python mappings
• Better integrate Quickcheck tests, so LIFT knows about quickcheck failures
• Pathname support (requires FILE* support)

11Prior Art / Other Solutions

Python-on-Lisp

It was a good try back in 2006, but unfortunately hasexperienced significant bitrot. It did manage to provide callbacks andwriting to Lisp streams, however (both of which are still on b-b's TODOlist). Very handy features!

pythononlisp-ex

A fork of Python-on-Lisp which shimmies things betweenLisp and Python using JSON. As you might expect, this falls down as soon asyou hit something which can't be represented in JSON, which covers quite alot of types.

Pyffi

A more streamlined, and less featureful, Python FFI which came afterPoL. Technically, burgled-batteries began life as some patches to Pyffibecause it seemed to be the best place to start. (Almost nothing of Pyffiremains.)

CLPython

A very fine Python compiler written in Common Lisp. It requiresa somewhat beefier machine than CPython. Unfortunately, it doesn't work withPython libraries written in C (e.g., numpy), and so is unable to handle thefull gamut of Python libraries one might wish to borrow.