cl-naive-store

2021-06-30

This is a persisted, in memory (lazy loading) document store for Common Lisp.

Upstream URL

gitlab.com/Harag/cl-naive-store

Author

Phil Marneweck, Pascal J. Bourguignon <pjb@informatimago.com>

License

MIT

README.org

*WARNING*

I have tested this version with live systems but am still rolling it out to all my live systems so there might be some bug churn over the next couple of weeks as this version is put through its paces.

1cl-naive-store

cl-naive-store is a database (document store to more precise) which is persisted, in memory (lazy loading), indexed and written completely in Common Lisp.

The naive bit comes from the fact that data is persisted as plists in files to make them human and machine readable, also there is no rocket science code.

The store was designed to be customisable, just about anything can be customised, have a look at the implementation api to get an idea of what is possible.

1.1License

MIT the doc is here.

1.2News

What was supposed to take weeks ended up taking months, but we are finally there. The 2021.5.18 version adds sharding to cl-naive-store and as a consequence of that the use of parallel processing internally and thread safety over all (fingers crossed!).

There is more refinement that we would like to do in the future for the parallel processing but what we have now is fast enough and simple enough for a good start.

All the documentation from the wiki has been placed into a docs/ folder as org-mode files.

1.3Acknowledgements

The latest version was financially supported by Managing Transformation Solutions (Pty) Ltd, thank you for your continued support of open source software!

I must also give a big shout out to Pascal J. Bourguignon for code reviews, threading code, test code and a whole lot more. His guidance was invaluable, even if I did not always listen to him. /Pascal does not endorse this software in any way, shape or form, he assisted me when and where I asked for specific help./

1.4Features

1.4.1Persisted

Data can be written to file per document update or as a batch update.

1.4.2In memory

Data is loaded into memory for querying and lookups, that makes them fast.

1.4.3Lazy Loading

Data is only loaded when needed and not before. If you use the store correctly it means that you will only have the data that users requested up to that point in memory.

1.4.4Indexed

Documents and key values can be indexed, and a user can specify their own additional indexes as well. Queries and lookups can both be done using indexes which speeds up the retrieval of data considerably.

1.4.5Sharding

Sharding is the breaking down of files into smaller files, in the case of naive-store that means that instead of one file per collection there could be many.

Sharding is done based on the actual data in collections. The user specifies which elements of a document it wants to use for sharding on a collection. If none is specified no sharding is done.

1.4.6Layered Design

cl-naive-store can do a lot but you as the user decides how much of the store's functionality you want to use for your own project.

Functionality was broken down into these packages:

  • cl-naive-store.naive-core
  • cl-naive-store.document-types
  • cl-naive-store.document-type-defs
  • cl-naive-store.naive-documents
  • cl-naive-store.naive-indexed
  • cl-naive-store.naive-merkle
  • cl-naive-store.test

The following .asd files can be used to load different functionality:

  • cl-naive-store.naive-core.asd loads the most basic functionality forcl-naive-store. Use this if you don't any of the other extensions.
  • cl-naive-store.naive-merkle.asd loads naive-documents and theexperimental merkle functionality.
  • cl-naive-store.naive-indexed.asd loads naive-core and indexfunctionality.
  • cl-naive-store.document-types.asd loads naive-core and document-typefunctionality.
  • cl-naive-store.document-defs.asd loads naive-core, document-typesand type definition functionality.
  • cl-naive-store.documents.asd loads naive-core, naive-indexed,documents-types, document-type-defs and document functionality.
  • cl-naive-store.asd loads the whole shebang.
  • cl-naive-store.test.asd loads tests

1.5Documentation

Documentation can be found in the docs folder in the repository.

1.6Examples

Examples are in the examples folder in the git repository. If those are to simplistic for you have a look at the code in the tests.

1.7Dependencies

  • cl-fad
  • iron-clad
  • cl-murmurhash
  • split-sequence
  • uuid
  • local-time
  • cl-getx
  • bordeaux-threads
  • lparallel
  • cl-cpus

1.8Supported CL Implementations

All Tests pass on SBCL an CCL

1.9Development Roadmap

Have a look at the issues in gitlab, future development is tagged accordingly.

1.10Tests

Go to the tests folder

cd cl-naive-store/tests/

Run make with any of the following

  • test
  • test-load-systems
  • test-run-tests
  • run-tests-ccl
  • run-tests-sbcl

For example:

make test-run-tests

You should see the following at the end.

SUCCESS COUNT:       46
FAILURE COUNT:        0
TOTAL TESTS:         46
Completed Test CL-NAIVE-STORE-TESTS:TEST-ALL

Dependencies (11)

  • alexandria
  • bordeaux-threads
  • cl-cpus
  • cl-fad
  • cl-getx
  • cl-murmurhash
  • ironclad
  • local-time
  • lparallel
  • split-sequence
  • uuid

Dependents (0)

    • GitHub
    • Quicklisp
    • Sponsor