A BibTeX system in Common Lisp ****************************** CL-BibTeX is a replacement for the BibTeX program, written in Common Lisp. The aim is to enable the user to format bibliographic entries using Common Lisp programs, rather than using the stack language of BibTeX style files. It is released under the terms of version 2.1 of the GNU Lesser General Public License as published by the Free Software Foundation (see file COPYING) or any later version, as clarified by the preamble found in COPYING-preamble.txt. (Note: run-lisp is included from CLOCC and has its own license.) Development now takes place on GitHub at https://github.com/mkoeppe/cl-bibtex A project home page is at http://www.nongnu.org/cl-bibtex/ (The old development pages at http://savannah.nongnu.org/projects/cl-bibtex/ are obsolete.) There is a mailing list `cl-bibtex-users@nongnu.org'. You can subscribe to the list at http://lists.nongnu.org/mailman/listinfo/cl-bibtex-users Reasons ======= A powerful tool is needed for dealing with citations in scientific documents. BibTeX is good for formatting bibliographies, but customizing the format is a pain because it requires writing/changing a program in BAFLL (BibTeX Anonymous Forth-Like Language -- Drew McDermott in comp.lang.lisp), which looks like this: FUNCTION {tie.or.space.connect} { duplicate$ text.length$ #3 < { "~" } { " " } if$ swap$ * * } Other citation-related processing like producing a list of authors and citation indices is usually done with a combination of BibTeX, TeX macros, and ad-hoc scripting with Perl or AWK. The new CL-BibTeX system aims to replace all this. Status ====== CL-BibTeX is a compatible and stable re-implementation of the BibTeX program, which was written by Oren Patashnik in the WEB language. It consists of the following components: * A reader for BibTeX style files (.bst), * a reader and a writer for bibliography databases (.bib), * a reader for .aux files, * an implementation of BibTeX's built-in functions (like for parsing and formatting names, etc.), * an interpreter for the stack language of BibTeX style files. Thus CL-BibTeX allows to both use existing BibTeX style files, and to write new BibTeX styles in Common Lisp. In addition to this, CL-BibTeX contains: * A compiler that transforms BibTeX style files into comprehensible Common Lisp programs (using type analysis). The resulting Common Lisp programs can serve as a starting point for the customization of the style. Future directions of CL-BibTeX's development can be found in the TODO file. Where to start ============== Compile and load all Lisp files, using ASDF or bibtex.system. The function BIBTEX is like the bibtex program. Pass the file name STEM; BIBTEX will read STEM.aux, interpret a BST file, and write STEM.bbl: (bibtex-compiler:bibtex "example") In fact, when the STEM.aux file requests a bibliography style, BIBTEX first looks whether a Lisp function that implements the style has been defined using DEFINE-BIBTEX-STYLE. Otherwise, it tries to find (using kpathsearch) and load a Lisp file named "STYLE.lbst"; it is supposed to define the style using DEFINE-BIBTEX-STYLE. Finally, BIBTEX tries to find "STYLE.bst" and interpret it. A BibTeX style implemented in Lisp is supposed to read all bibliography files in *BIB-FILES* and to write the formatted bibliography to the stream *BBL-OUTPUT*. The package BIBTEX-RUNTIME contains useful routines for reading AUX files, formatting names, etc. An important function is READ-ALL-BIB-FILES-AND-COMPUTE-BIB-ENTRIES. You can invoke the BST-to-Common-Lisp compiler with the function COMPILE-BST-FILE: (bibtex-compiler:compile-bst-file (kpathsea:find-file "amsabbrv.bst") "amsabbrv.lbst") The resulting Lisp file is a readable Common Lisp version of the BST file, which can be run on the AUX file to produce a BBL file, without using the BST interpreter: (compile-file "amsabbrv.lbst") (load "amsabbrv.lbst") ; calls DEFINE-BIBTEX-STYLE (bibtex-compiler:bibtex "example") ; now uses the Lisp style There is also a little shell script "bibtex" for invoking CL-BIBTEX from the shell; it uses the "run-lisp" script from CLOCC. Restrictions of the compiler ============================ The BST->CL compiler works by analyzing the type of the wizard-defined functions (i.e., those defined by a FUNCTION command in the BST file). The compiler tries to find out how many "literals" (values) a function pops from the stack and how many it pushes and determines their types (BOOLEAN, INTEGER, STRING). * If a wizard-defined function tries to pop a function literal that it hasn't pushed, or leaves a function literal on the stack, you lose. (In other words, you can't define higher-order BST functions.) * The two branches of an IF$ function must deliver the same net number of values. (As a special exception, the type of IF$ within a WHILE$ body which occurs in the FORMAT.NAMES function of the standard styles is also supported. This is reported with a warning message.) On the compilation of a language that no-one can read or write ============================================================== The language of BibTeX style files (BAFLL) is expected to be read or written by "wizards" only. (This is how the original BibTeX calls the authors of BibTeX style files.) In fact, when casual users try to customize a BibTeX style file, they often come up with slightly broken styles that appear to work but which fail in corner cases. The reason is that programs written in BAFLL can hardly be read, written, or even debugged. This causes an extra difficulty for the compiler, because broken corner cases tend to misbehave on the stack, rendering the function unanalyzable, hence uncompilable. The compiler now contains a heuristic that tries to fix these situations, but it may fail. In this case, you should try to understand the error messages, fix the BibTeX style file accordingly, and try compiling it again. Implementation dependencies =========================== It does not run on some 0.7.? version of SBCL because it does not like (COPY-READTABLE NIL). SBCL 0.7.9 seems to be fine. I noticed that CMUCL 3.0.8 18c+ does not like (PEEK-CHAR T STREAM NIL #\Space); it won't skip over whitespace when the EOF character is whitespace? (I don't use this any more in CL-BibTeX.) CLISP 2.30: 1. When I invoke COMPILE-BST-FILE, CLISP says "Lisp stack overflow. RESET"; this seems to be caused by the pprint-dispatch function for DEFINE-BIBTEX-STYLE. 2. The format directive ~<...~:> (logical block) does not seem to work properly. (I have used reader conditionalization to make CL-BibTeX work with CLISP.) Have fun! -- Matthias Koeppe <mkoeppe@math.ucdavis.edu>