Caveman2 - Lightweight web application framework

Usage

(defparameter *web* (make-instance '<app>))

@route GET "/"
(defun index ()
  (render #P"index.tmpl"))

@route GET "/hello"
(defun say-hello (&key (|name| "Guest"))
  (format nil "Hello, ~A" |name|))

About Caveman2

What's different from Caveman "1"?

Everything. Caveman2 was written from scratch.

These are noteworthy points.

• Is based on ningle
• Has database integration
• Uses new, separate configuration system (Envy)
• Has new routing macro

The reason I wrote it from scratch:

One of the most frequently asked questions was "Which should I use: ningle or Caveman? What are the differences?" I think these were asked so frequently because Caveman and ningle were too similar. Both of them are called "micro", and had no database support.

With Caveman2, Caveman is no longer a "micro" web application framework. It supports CL-DBI, and has database connection management by default. Caveman has started growing up.

Design Goal

Caveman is intended to be a collection of common parts of web applications. With Caveman2, I use three rules to make decisions:

• Be extensible.
• Be practical.
• Don't force anything.

Quickstart

You came here because you're interested in living like a caveman, right? This isn't Disneyland, but we can start here. Let's get into a cave!

Installation

Caveman2 is now available on Quicklisp.

(ql:quickload :caveman2)

Generating a project skeleton

(caveman2:make-project #P"/path/to/myapp/"
                       :author "<Your full name>")
;-> writing /path/to/myapp/.gitignore
;   writing /path/to/myapp/README.markdown
;   writing /path/to/myapp/app.lisp
;   writing /path/to/myapp/db/schema.sql
;   writing /path/to/myapp/shlyfile.lisp
;   writing /path/to/myapp/myapp-test.asd
;   writing /path/to/myapp/myapp.asd
;   writing /path/to/myapp/src/config.lisp
;   writing /path/to/myapp/src/db.lisp
;   writing /path/to/myapp/src/main.lisp
;   writing /path/to/myapp/src/view.lisp
;   writing /path/to/myapp/src/web.lisp
;   writing /path/to/myapp/static/css/main.css
;   writing /path/to/myapp/t/myapp.lisp
;   writing /path/to/myapp/templates/_errors/404.html
;   writing /path/to/myapp/templates/index.tmpl
;   writing /path/to/myapp/templates/layout/default.tmpl

Start a server

This is an example that assumes that the name of your application is "myapp". Before starting the server, you must first load your app.

(ql:quickload :myapp)

Your application has functions named start and stop to start/stop your web application.

(myapp:start :port 8080)

As Caveman is based on Clack/Lack, you can choose which server to run on -- Hunchentoot, Woo or Wookie, etc.

(myapp:start :server :hunchentoot :port 8080)
(myapp:start :server :fcgi :port 8080)

I recommend you use Hunchentoot on a local machine, and use Woo in a production environment.

You can also start your application by using clackup command.

$ ros install clack
$ which clackup
/Users/nitro_idiot/.roswell/bin/clackup

$ APP_ENV=development clackup --server :fcgi --port 8080 app.lisp

Routing

Caveman2 provides 2 ways to define a route -- @route and defroute. You can use either.

@route is an annotation macro, defined by using cl-annot. It takes a method, a URL-string, and a function.

@route GET "/"
(defun index ()
  ...)

;; A route with no name.
@route GET "/welcome"
(lambda (&key (|name| "Guest"))
  (format nil "Welcome, ~A" |name|))

This is similar to Caveman1's @url except for its argument list. You don't have to specify an argument when it is not required.

defroute is just a macro. It provides the same functionality as @route.

(defroute index "/" ()
  ...)

;; A route with no name.
(defroute "/welcome" (&key (|name| "Guest"))
  (format nil "Welcome, ~A" |name|))

Since Caveman bases on ningle, Caveman also has the Sinatra-like routing system.

;; GET request (default)
@route GET "/" (lambda () ...)
(defroute ("/" :method :GET) () ...)

;; POST request
@route POST "/" (lambda () ...)
(defroute ("/" :method :POST) () ...)

;; PUT request
@route PUT "/" (lambda () ...)
(defroute ("/" :method :PUT) () ...)

;; DELETE request
@route DELETE "/" (lambda () ...)
(defroute ("/" :method :DELETE) () ...)

;; OPTIONS request
@route OPTIONS "/" (lambda () ...)
(defroute ("/" :method :OPTIONS) () ...)

;; For all methods
@route ANY "/" (lambda () ...)
(defroute ("/" :method :ANY) () ...)

Route patterns may contain "keywords" to put the value into the argument.

(defroute "/hello/:name" (&key name)
  (format nil "Hello, ~A" name))

The above controller will be invoked when you access "/hello/Eitaro" or "/hello/Tomohiro", and name will be "Eitaro" or "Tomohiro", as appropriate.

(&key name) is almost same as a lambda list of Common Lisp, except it always allows other keys.

(defroute "/hello/:name" (&rest params &key name)
  ;; ...
  )

Route patterns may also contain "wildcard" parameters. They are accessible by using splat.

(defroute "/say/*/to/*" (&key splat)
  ; matches /say/hello/to/world
  (format nil "~A" splat))
;=> (hello world)

(defroute "/download/*.*" (&key splat)
  ; matches /download/path/to/file.xml
  (format nil "~A" splat)) 
;=> (path/to/file xml)

If you'd like to write use a regular expression in a URL rule, :regexp t should work.

(defroute ("/hello/([\\w]+)" :regexp t) (&key captures)
  (format nil "Hello, ~A!" (first captures)))

Normally, routes are tested for a match in the order they are defined, and only the first route matched is invoked, with the following routes being ignored. However, a route can continue testing for matches in the list, by including next-route.

(defroute "/guess/:who" (&key who)
  (if (string= who "Eitaro")
      "You got me!"
      (next-route)))

(defroute "/guess/*" ()
  "You missed!")

You can return following formats as the result of defroute.

• String
• Pathname
• Clack's response list (containing Status, Headers and Body)

Redirection

Redirect to another route with(redirect "url"). A second optional argument is the status code, 302 by default.

Reverse URLs

When you defined routes with names, you can find the URL from a name with (url-for route-name &rest params).

The function will throw an error if no route is found.

More helper functions

See also:

• add-query-parameters base-url params

Structured query/post parameters

Parameter keys containing square brackets ("[" & "]") will be parsed as structured parameters. You can access the parsed parameters as _parsed in routers.

<form action="/edit">
  <input type="name" name="person[name]" />
  <input type="name" name="person[email]" />
  <input type="name" name="person[birth][year]" />
  <input type="name" name="person[birth][month]" />
  <input type="name" name="person[birth][day]" />
</form>

(defroute "/edit" (&key _parsed)
  (format nil "~S" (cdr (assoc "person" _parsed :test #'string=))))
;=> "((\"name\" . \"Eitaro\") (\"email\" . \"e.arrows@gmail.com\") (\"birth\" . ((\"year\" . 2000) (\"month\" . 1) (\"day\" . 1))))"

;; With assoc-utils
(ql:quickload :assoc-utils)
(import 'assoc-utils:aget)
(defroute "/edit" (&key _parsed)
  (format nil "~S" (aget _parsed "person")))

Blank keys mean they have multiple values.

<form action="/add">
  <input type="text" name="items[][name]" />
  <input type="text" name="items[][price]" />

  <input type="text" name="items[][name]" />
  <input type="text" name="items[][price]" />

  <input type="submit" value="Add" />
</form>

(defroute "/add" (&key _parsed)
  (format nil "~S" (assoc "items" _parsed :test #'string=)))
;=> "(((\"name\" . \"WiiU\") (\"price\" . \"30000\")) ((\"name\" . \"PS4\") (\"price\" . \"69000\")))"

Templates

Caveman uses Djula as its default templating engine.

{% extends "layouts/default.html" %}
{% block title %}Users | MyApp{% endblock %}
{% block content %}
<div id="main">
  <ul>
  {% for user in users %}
    <li><a href="{{ user.url }}">{{ user.name }}</a></li>
  {% endfor %}
  </ul>
</div>
{% endblock %}

(import 'myapp.view:render)

(render #P"users.html"
        '(:users ((:url "/id/1"
                   :name "nitro_idiot")
                  (:url "/id/2"
                   :name "meymao"))
          :has-next-page T))

If you want to get something from a database or execute a function using Djula you must explicity call list when passing the arguments to render so that the code executes.

(import 'myapp.view:render)

(render #P"users.html"
        (list :users (get-users-from-db)))

JSON API

This is an example of a JSON API.

(defroute "/user.json" (&key |id|)
  (let ((person (find-person-from-db |id|)))
    ;; person => (:|name| "Eitaro Fukamachi" :|email| "e.arrows@gmail.com")
    (render-json person)))

;=> {"name":"Eitaro Fukamachi","email":"e.arrows@gmail.com"}

render-json is a part of a skeleton project. You can find its code in "src/view.lisp".

Static file

Images, CSS, JS, favicon.ico and robot.txt in "static/" directory will be served by default.

/images/logo.png => {PROJECT_ROOT}/static/images/logo.png
/css/main.css    => {PROJECT_ROOT}/static/css/main.css
/js/app/index.js => {PROJECT_ROOT}/static/js/app/index.js
/robot.txt       => {PROJECT_ROOT}/static/robot.txt
/favicon.ico     => {PROJECT_ROOT}/static/favicon.ico

You can change these rules by rewriting "PROJECT_ROOT/app.lisp". See Clack.Middleware.Static for detail.

Configuration

Caveman adopts Envy as a configuration switcher. This allows definition of multiple configurations and switching between them according to an environment variable.

This is a typical example:

(defpackage :myapp.config
  (:use :cl
        :envy))
(in-package :myapp.config)

(setf (config-env-var) "APP_ENV")

(defconfig :common
  `(:application-root ,(asdf:component-pathname (asdf:find-system :myapp))))

(defconfig |development|
  `(:debug T
    :databases
    ((:maindb :sqlite3 :database-name ,(merge-pathnames #P"test.db"
                                                        *application-root*)))))

(defconfig |production|
  '(:databases
    ((:maindb :mysql :database-name "myapp" :username "whoami" :password "1234")
     (:workerdb :mysql :database-name "jobs" :username "whoami" :password "1234"))))

(defconfig |staging|
  `(:debug T
    ,@|production|))

Every configuration is a property list. You can choose the configuration which to use by setting APP_ENV.

To get a value from the current configuration, call myapp.config:config with the key you want.

(import 'myapp.config:config)

(setf (osicat:environment-variable "APP_ENV") "development")
(config :debug)
;=> T

Database

When you add :databases to the configuration, Caveman enables database support. :databases is an association list of database settings.

(defconfig |production|
  '(:databases
    ((:maindb :mysql :database-name "myapp" :username "whoami" :password "1234")
     (:workerdb :mysql :database-name "jobs" :username "whoami" :password "1234"))))

db in a package myapp.db is a function for connecting to each databases configured the above. Here is an example.

(use-package '(:myapp.db :sxql :datafly))

(defun search-adults ()
  (with-connection (db)
    (retrieve-all
      (select :*
        (from :person)
        (where (:>= :age 20))))))

The connection is alive during the Lisp session, and will be reused in every HTTP request.

retrieve-all and the query language came from datafly and SxQL. See those sets of documentation for more information.

Set HTTP headers or HTTP status

There are several special variables available during a HTTP request. *request* and *response* represent a request and a response. If you are familiar with Clack, these are instances of subclasses of Clack.Request and Clack.Response.

(use-package :caveman2)

;; Get a value of Referer header.
(http-referer *request*)

;; Set Content-Type header.
(setf (getf (response-headers *response*) :content-type) "application/json")

;; Set HTTP status.
(setf (status *response*) 304)

If you would like to set Content-Type "application/json" for all "*.json" requests, next-route can be used.

(defroute "/*.json" ()
  (setf (getf (response-headers *response*) :content-type) "application/json")
  (next-route))

(defroute "/user.json" () ...)
(defroute "/search.json" () ...)
(defroute ("/new.json" :method :POST) () ...)

Using session

Session data is for memorizing user-specific data. *session* is a hash table that stores session data.

This example increments :counter in the session, and displays it for each visitor.

(defroute "/counter" ()
  (format nil "You came here ~A times."
          (incf (gethash :counter *session* 0))))

Caveman2 stores session data in-memory by default. To change this, specify :store to :session in "PROJECT_ROOT/app.lisp".

This example uses RDBMS to store session data.

      '(:backtrace
        :output (getf (config) :error-log))
      nil)
- :session
+ (:session
+  :store (make-dbi-store :connector (lambda ()
+                                      (apply #'dbi:connect
+                                             (myapp.db:connection-settings)))))
  (if (productionp)
      nil
      (lambda (app)

NOTE: Don't forget to add :lack-session-store-dbi as :depends-on of your app. It is not a part of Clack/Lack.

See the source code of Lack.Session.Store.DBi for more information.

• Lack.Session.Store.Dbi

Throw an HTTP status code

(import 'caveman2:throw-code)

(defroute ("/auth" :method :POST) (&key |name| |password|)
  (unless (authorize |name| |password|)
    (throw-code 403)))

Specify error pages

To specify error pages for 404, 500 or such, define a method on-exception of your app.

(defmethod on-exception ((app <web>) (code (eql 404)))
  (declare (ignore app code))
  (merge-pathnames #P"_errors/404.html"
                   *template-directory*))

Hot Deployment

Though Caveman doesn't have a feature for hot deployment, Server::Starter -- a Perl module -- makes it easy.

$ APP_ENV=production start_server --port 8080 -- clackup --server :fcgi app.lisp

NOTE: Server::Starter requires the server to support binding on a specific fd, which means only :fcgi and :woo are the ones work with start_server command.

To restart the server, send HUP signal (kill -HUP <pid>) to the start_server process.

Error Log

Caveman outputs error backtraces to a file which is specified at :error-log in your configuration.

(defconfig |default|
  `(:error-log #P"/var/log/apps/myapp_error.log"
    :databases
    ((:maindb :sqlite3 :database-name ,(merge-pathnames #P"myapp.db"
                                                        *application-root*)))))

Use another templating library

CL-WHO

(import 'cl-who:with-html-output-to-string)

(defroute "/" ()
  (with-html-output-to-string (output nil :prologue t)
    (:html
      (:head (:title "Welcome to Caveman!"))
      (:body "Blah blah blah."))))
;=> "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">
;   <html><head><title>Welcome to Caveman!</title></head><body>Blah blah blah.</body></html>"

• CL-WHO Website

CL-Markup

(import 'cl-markup:xhtml)

(defroute "/" ()
  (xhtml
    (:head (:title "Welcome to Caveman!"))
    (:body "Blah blah blah.")))
;=> "<?xml version=\"1.0\" encoding=\"UTF-8\"?><!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\"><html><head><title>Welcome to Caveman!</title></head><body>Blah blah blah.</body></html>"

• CL-Markup repository

cl-closure-template

{namespace myapp.view}

{template renderIndex}
<!DOCTYPE html>
<html>
<head>
  <title>"Welcome to Caveman!</title>
</head>
<body>
  Blah blah blah.
</body>
</html>
{/template}

(import 'myapp.config:*template-directory*)

(closure-template:compile-cl-templates (merge-pathnames #P"index.tmpl"
                                                        *template-directory*))

(defroute "/" ()
  (myapp.view:render-index))

• cl-closure-template
• Closure Templates Documentation

See Also

• Clack - Web application environment.
• Lack - The core of Clack.
• ningle - Super micro web application framework that Caveman is based on.
• Djula - HTML Templating engine.
• CL-DBI - Database-independent interface library.
• SxQL - SQL builder library.
• Envy - Configuration switcher.
• Roswell - Common Lisp implementation manager.

Author

• Eitaro Fukamachi (e.arrows@gmail.com)

License

Licensed under the LLGPL License.