A library providing an easy-to-use API for working with memory-mapped files.
Trivial-MMAP is a Common Lisp library that aims to provide a high-level, easy-to-use API for working with memory-mapped files.
The library exports the following symbols:
mmap-filea function that maps a file into memory and returns an object containing a pointer to the memory-mapped file along with other meta data;
munmap-filea function that removes a memory-mapped file from memory;
mmap-read-chara helper function that facilitates reading a character from a memory-mapped file;
mmap-read-bytea helper function for reading one byte from a memory-mapped file;
with-mmap-filea helper macro that ensures that a memory-mapped file is safely unmapped (using
munmap-filefunction) after we are done with it.
In additions, the following are the exported symbols of the predicate
and accessor functions for dealing with the memory-mapped file object
returned by the
mmapped-file-pa predicate function that returns true if a variable is a memory-mapped file object (and false otherwise);
mmapped-file-pointeran accessor function that, given a memory-mapped file object, returns a raw pointer address of the file;
mmapped-file-sizean accessor function that, given a memory-mapped file object, returns the size of the file;
mmapped-file-offsetan accessor function that, given a memory-mapped file object, returns the offset into the file.
Trivial-MMAP is based on code snippets in a blog post by Nicolas Martyanoff.
The following code snippet shows how to use Trivial-MMAP's
mmap-read-char to read one character at a time
from a file containing Twitter social graph (780 MB uncompressed).
CL-USER> (time (trivial-mmap:with-mmap-file (mmapped-file "/home/zodmaner/twitter_rv_15066953.net") (loop while (< (trivial-mmap:mmapped-file-offset mmapped-file) (trivial-mmap:mmapped-file-size mmapped-file)) do (assert (characterp (trivial-mmap:mmap-read-char mmapped-file)))))) Evaluation took: 3.400 seconds of real time 3.400000 seconds of total run time (3.360000 user, 0.040000 system) 100.00% CPU 8,479,536,815 processor cycles 65,952 bytes consed
The little toy benchmark above was run on my laptop (an i5 ThinkPad
X230 (Ivy Bridge) with 8 GB of RAM and an SSD). For comparison,
reading the same file using
around 14 seconds of real time on the same machine.
Of course, you could avoid the
with-mmap-file macro if you want to.
The following code snippet is equivalent to the above, but only use
the functions provided by Trivial-MMAP:
(time (let ((mmapped-file (trivial-mmap:mmap-file "/home/zodmaner/twitter_rv_15066953.net"))) (unwind-protect (loop while (< (trivial-mmap:mmapped-file-offset mmapped-file) (trivial-mmap:mmapped-file-size mmapped-file)) do (assert (characterp (trivial-mmap:mmap-read-char mmapped-file)))) (trivial-mmap:munmap-file mmapped-file))))
Although from my experiences the version that uses the
with-mmap-file macro is usually around 10% to 15% faster compared to
the version that did not use the macro.
Of course, this library is far from being completed. Here is the list of some of the current limitations/missing features:
- Currently the library only supports reading data from memory-mapped files.
- The library only supports Linux. It may work on MacOS, but definitely not on Windows.
As always, patches and bug reports are more then welcome!
Author & Maintainer
Smith Dhumbumroong (email@example.com)