http2

#+TITLE: HTTP/2 in Common Lisp

[[https://github.com/zellerin/http2/actions/workflows/test.yml/badge.svg]]

This is an implementation of HTTP/2 protocol as described in
RFC9113 (and RFC7540 before) and RFC7541 (HPACK).

The core part of the library implements writing http2 frames, reading them, and
reacting on read frames. What to do with the frames is managed by application
specific classes derived from ~CONNECTION~ and ~HTTP2-STREAM~. This part should be
relatively fixed.

In addition, helper functions to create server and to make client calls is
provided.  Based on these helpers, demo client and demo server is also provided,
see [[./client/client.lisp][client.lisp]] or [[./server/server.lisp][server.lisp]]. The best way to make servers and clients
includes decisions on how to handle asynchronicity, window management, etc, and may
be still subject to change.

** Status and quick start
In quicklisp. Tested primarily on sbcl, occasionally on ecl.

Almost all parts of the listed standards implemented, see below for exceptions.

The library definitely allows to implement a client or server, and a simple ones
are attached to it as examples.

Currently, all changes happen in v2-main branch that will eventually become version 2 of this. The changes include
switch from frames over Lisp stream to frames as an array of octets; this enables - among other - poll based single thread 
server implementation (as can be seen in the ../tls-server-pocs).

*** Client

The bundled client requests remote page and returns values of body (converted to
string), result status, and response headers.

Improvements to allow for cookie management, ... are out of scope of this
library, as they would probably be better taken over from some existing HTTP/1.x
library or incorporated there.

Improvements to query several resources on one server, and to receive pushes are
considered for future, as they are http2 specific.

#+begin_src lisp
  (ql:quickload 'http2/client)
  (http2/client:retrieve-url "https://example.com")
#+end_src

#+RESULTS:
#+begin_src text
<!doctype html>
<html>
<head>
    <title>Example Domain</title>

    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" conten...[sly-elided string of length 1256]"
"200"
(("content-length" . "1256") ("x-cache" . "HIT") ("vary" . "Accept-Encoding")
 ("server" . "ECS (bsa/EB24)")
 ("last-modified" . "Thu, 17 Oct 2019 07:18:26 GMT")
 ("expires" . "Thu, 16 Jun 2022 09:35:21 GMT")
 ("etag" . "\"3147526947+ident\"") ("date" . "Thu, 09 Jun 2022 09:35:21 GMT")
 ("content-type" . "text/html; charset=UTF-8")
 ("cache-control" . "max-age=604800") ("age" . "239205"))
#+end_src

*** Server
[[file:server/server.lisp][Example https server]] (that indicates h2 support with ALPN) is provided ; TLS key
and certificate is needed ([[./scaffolding/run-server.lisp][test script]] generates a pair when missing). It
handles connections one by thread by default; this is trivial to improve in
several ways (workers, polling...) with customizing ~*dispatch-fn*~ but for now
out of scope of this project.

#+begin_src lisp
  (ql:quickload 'http2/server/example)
  (http2/server-example::run-demo-server :port 1234)
#+end_src

Then point your browser (not eww nor drakma, of course - must support http2) to
https://localhost:1234 or
#+begin_src sh
  curl -k http://localhost:1234/
#+end_src

Then define handlers for your paths or prefixes, see the example server for a
starting point and API documentation for more details.

** Missing pieces
- No handling of priorities is implemented. This is OK, as these are only
  suggestions, and they are more or less dropped in RFC9113 anyway.
- Push promises are not implemented in the client. This is OK, they are disabled
  by default (settings)
- Option to encode headers to /Literal Header Field Never Indexed/ format is not
  implemented. This is simple to do, but interface would need to be though out.
- The algorithm to split data being sent out to windows and frames can be
  confused sometimes
- The MAX-HEADER-LIST-SIZE limit is not enforced.
- Some checks on behaviour of the peer are not enforced

** Implementation details

See [[file:HACKING.org][HACKING.org]] for the details.

** Dependencies

The core library used trivial-gray-streams to implement streams over data frames.

Client and server require usocket and cl+ssl to talk over TLS, and bordeaux-threads for concurrency.

Client uses puri to manipulate URLs.

Server also uses cffi directly to [[file:tls/cl+ssl.lisp][check and confirm alpn]]. The sample server uses
parenscript and cl-who to create html; this is an arbitrary choice and your
server can use anything else.

Additionally, fiasco is used for testing (and Javascript for browser-side testing).

** Speed & scaling
The code was not written with speed as primary concern. Measurements depend on
many tunable factor both on the client and the server.

Having said that, rough test shows that the server on trivial page, single
requestor, 10k requests, over TLS) seems to be about 2-3x faster than
Hunchentoot and a bit slower than [[https://github.com/fukamachi/woo][Woo]]. However, when multiple requests are sent
in parallel from one client, it is much faster than Woo (even with the
Hunchentoot-style request handling).

What is currently relatively slow is conversion between (utf-8) text and binary data.

Anyway, run or modify the [[file:scaffolding/speed-test.lisp][speed test code]] for your results:
: sbcl --script scaffolding/speed-test.lisp

The server should handle any number of requests on one connection (until stream
ids are exhausted, but that is a client problem).

** License
Licensed by MIT license.

Some comments are taken over from the RFCs above and copyrighted by RFC
contributors. I read the copyright licenses for RFC that this is allowed.

** Related software
There is an Akamai code on https://github.com/akamai/cl-http2-protocol that
supported bigger parts of the drafted HTTP/2 protocol in 2014; apparently hard
if not impossible to run now. It used NPN instead of ALPN.

#  LocalWords:  HPACK


alexandria

anaphora

flexi-streams

gzip-stream

trivial-gray-streams

http2/client

http2/server

http2/server/example

http2/test

http2/tls

http2/all

cl+ssl

puri

cl-who

parenscript

bordeaux-threads

fiasco

Glue to wrap HTTP/2 client or server with TLS

http2

Upstream URL

Author

License

http2

Version:

License:

Authors:

Dependencies:

http2/all

Dependencies:

http2/client

Version:

License:

Authors:

Dependencies:

http2/server

Version:

License:

Authors:

Dependencies:

http2/server/example

Version:

License:

Authors:

Dependencies:

http2/test

License:

Dependencies:

http2/tls

Version:

License:

Authors:

Dependencies:

Dependencies (11)

Dependents (0)