An experiment to improve multithreading performance of Hunchentoot without any additional dependencies.

hunchentoot-recycling-taskmaster

* Abstract

hunchentoot-recycling-taskmaster is a taskmaster implementation for Hunchentoot,
aiming to improve connection establishment efficiency through thread-pooling
and flexible thread count adjustment.

* Performance tl;dr

[[file:pics/benchmark-result-2025-12-26/sleep_1ms/keep-alive/100_connections_requests_sec__sleep_1ms__keep-alive__higher_is_better.svg]]

[[file:pics/benchmark-result-2025-12-26/sleep_1ms/no-keep-alive/100_connections_requests_sec__sleep_1ms__no_keep-alive__higher_is_better.svg]]

On this benchmark, my HTTP handler *always responds after 1ms* to simulate some workloads.

- Hunchentoot is an all-rounder. It works well on Keep-alive connections. I think it is good for typical use-cases.
- If your workload does not utilize Keep-alive, hunchentoot-recycling-taskmaster may be useful.
- Woo is very difficult to use. Woo seems fast only when your handlers work with very low latency. See [[#about-woo][About Woo]] below.

In detail, See [[#benchmark][Benchmark]] below.
  
* How to use

** Caution about Lispworks

On Lispworks, hunchentoot-recycling-taskmaster does not work because
Hunchentoot on that does not handle a listen socket directly.

** Installation

*** Loading

Currently hunchentoot-recycling-taskmaster is only in [[https://ultralisp.org/][Ultralisp]], not in [[https://www.quicklisp.org/beta/][Quicklisp]] or [[https://github.com/ocicl/ocicl][ocicl]].

When you use Ultralisp, =ql:quickload= it :

#+begin_src common-lisp
  (ql:quickload "hunchentoot-recycling-taskmaster")
#+end_src

Or download it by yourself.

#+begin_src sh
  cd ~/quicklisp/local-projects
  git clone https://github.com/y2q-actionman/hunchentoot-recycling-taskmaster.git
#+end_src

#+begin_src common-lisp
  (ql:register-local-projects)            ; Do if required
  (ql:quickload "hunchentoot-recycling-taskmaster")
#+end_src

*** Running tests

#+begin_src common-lisp
  (ql:quickload "hunchentoot-recycling-taskmaster-test")
  (asdf:test-system '#:hunchentoot-recycling-taskmaster)
#+end_src

** Starting/stopping server

You can use hunchentoot-recycling-taskmaster just by changing
=hunchentoot:acceptor= to =hunchentoot-recycling-taskmaster:parallel-acceptor=, or
=hunchentoot:easy-acceptor= to =hunchentoot-recycling-taskmaster:parallel-easy-acceptor=.

#+begin_src common-lisp
  (defparameter *test-server*
    (make-instance 'hunchentoot-recycling-taskmaster:parallel-easy-acceptor
		   :port 4242))
  (hunchentoot:start *test-server*)
#+end_src

#+begin_src bash
  curl "http://127.0.0.1:4242/yo"
  # => "Hey!"
#+end_src

To stop it, =hunchentoot:stop= can be used.

#+begin_src common-lisp
  (hunchentoot:stop *test-server*)
#+end_src

See [[file:demo.lisp][demo.lisp]] for the sample codes above.

** API

These symbols are exported from =hunchentoot-recycling-taskmaster= package.
Please see their docstring.

- [Class] =parallel-acceptor=
- [Class] =parallel-easy-acceptor=
- [Class] =parallel-ssl-acceptor=
- [Class] =parallel-easy-ssl-acceptor=
- [Class] =recycling-taskmaster=
- [Variable] =*default-standby-thread-count*=
  
- [Function] =abandon-acceptor=
- [Condition] =recycling-taskmaster-corrupted-error=

* How it works

This section shows how hunchentoot-recycling-taskmaster works
comparing other implementations.

** Hunchentoot; make one thread per connection.

[[file:pics/architecture/hunchentoot-architecture.dot.png]]

Hunchentoot uses one thread per one connection.  It means when a
client use one keep-alive connection for multiple requests,
Hunchentoot dedicate one thread to that connection.  There is some
delay for new connections, but it works well on keep-alive
connections.

** quux-hunchentoot and cl-tbnl-gserver-tmgr; Thread pooling

[[file:pics/architecture/hunchentoot-thread-pooling.dot.png]]

These implementations utilize a thread pool around Hunchentoot.
Instead of making a new thread for a new connection, they reuse
threads kept in its thread pool, reducing latency for new connections.

However, their benchmarks don't show significant differences from the
original Hunchentoot. I suspect this is for two reasons:

1. HTTP benchmarking tool, such as =wrk=, utilizes keep-alive
   connections.  Effects of thread-pooling are limited at the
   beginning of benchmarking.
2. Their thread-pool's size may be fixed. They cannot
   increase threads like Hunchentoot even on high workloads.

** hunchentoot-recycling-taskmaster

[[file:pics/architecture/hunchentoot-recycling-taskmaster-architecture.dot.png]]

hunchentoot-recycling-taskmaster tries to take both, thread-pooling
and changing the number of threads dynamically. In
hunchentoot-recycling-taskmaster, threads don't only work on connected
sockets but also some management tasks, such as accepting a new
connection from the listen socket, creating a new thread, or
terminating themselves.

For this management, all threads share the listen socket for
synchronizing acceptance and track how many threads on it.  Since
using the listen socket, hunchentoot-recycling-taskmaster successfully
implements these mechanisms without adding new dependencies to
hunchentoot.

* Benchmark

** Running benchmarks
To run benchmark by yourself, do below:

#+begin_src common-lisp
  (ql:quickload "hunchentoot-recycling-taskmaster-benchmark")
  (asdf:test-system '#:hunchentoot-recycling-taskmaster-benchmark)
#+end_src

** My Environments

*** My machine

| cl function name            |                                     |
|-----------------------------+-------------------------------------|
| LISP-IMPLEMENTATION-TYPE    | SBCL                                |
| LISP-IMPLEMENTATION-VERSION | 2.2.9.debian                        |
| MACHINE-TYPE                | X86-64                              |
| MACHINE-VERSION             | 13th Gen Intel(R) Core(TM) i7-1360P |
| SOFTWARE-TYPE               | Linux                               |
| SOFTWARE-VERSION            | 5.15.153.1-microsoft-standard-WSL2  |

*** Server library versions and parameters

| name                             | parameters             | version | Git commit                               |
|----------------------------------+------------------------+---------+------------------------------------------|
| hunchentoot-recycling-taskmaster | standby-thread-count 8 |   0.0.1 | 82913edaf3f65afb189f0d72ddeb6339bf0499ae |
| hunchentoot                      |                        |   1.3.1 | d1617e9d4eab6cb801c56cf36d9b0aab134fb7e6 |
| quux-hunchentoot                 |                        |   1.0.2 | quux-hunchentoot-20211230-git            |
| cl-tbnl-gserver-tmgr             | max-thread-count 8     |   0.1.1 | 1ae71c9324e876761cd1ee51768a34f0793e6879 |
| wookie                           |                        |  0.3.15 | 1f74b6c24b463c1e6fff35377e477934f72bac20 |
| woo                              | worker-num 8           |  0.12.0 | 7f5219c55d49190f5ae17b123a8729b31c5d706e |

*** benchmarking tool

I used =wrk= like below

#+begin_src bash
  # keep-alive
  wrk -t 4 -c 100 -d 10 http://localhost:4242/yo
  # simulating no keep-alive
  wrk "-H Connection: close" -t 4 -c 100 -d 10 http://localhost:4242/yo
#+end_src

** Results

*** keep-alive, sleep 1ms

[[file:pics/benchmark-result-2025-12-26/sleep_1ms/keep-alive/100_connections_requests_sec__sleep_1ms__keep-alive__higher_is_better.svg]]
[[file:pics/benchmark-result-2025-12-26/sleep_1ms/keep-alive/100_connections_latency(us)__sleep_1ms__keep-alive__lower_is_better.svg]]

On this benchmark, my HTTP handler *always responds after 1ms* to simulate some workloads.

Hunchentoot was quite fast.  As mentioned above, Hunchentoot assigns
one thread per connection and keeps using that thread until the
connection is closed.  Therefore, if a connection is kept alive,
there's no delay from thread creation, and it doesn't become
Hunchentoot slow.  =wrk= uses keep-alive by default, so this result
benefited from this.

cl-tbnl-gserver-tmgr was not very fast. This is presumably because it
only assigns 8 threads as workers.

Woo was not fast even set to 8 threads. See [[#about-woo][About Woo]] below.

*** no keep-alive, sleep 1ms

[[file:pics/benchmark-result-2025-12-26/sleep_1ms/no-keep-alive/100_connections_requests_sec__sleep_1ms__no_keep-alive__higher_is_better.svg]]
[[file:pics/benchmark-result-2025-12-26/sleep_1ms/no-keep-alive/100_connections_latency(us)__sleep_1ms__no_keep-alive__lower_is_better.svg]]

On this benchmark, HTTP handler still causes 1ms latency, and I
simulated "no keep-alive" by adding =wrk= option ="-H Connection: close"= .

Hunchentoot became slow. In this test, Hunchentoot works like "one
thread per one request" so latencies caused by creating a new thread
affected the result.

hunchentoot-recycling-taskmaster is designed to work well in this
situation.

*** keep-alive, sleep 0ms

[[file:pics/benchmark-result-2025-12-26/sleep_0ms/keep-alive/100_connections_requests_sec__sleep_0ms__keep-alive__higher_is_better.svg]]
[[file:pics/benchmark-result-2025-12-26/sleep_0ms/keep-alive/100_connections_latency(us)__sleep_0ms__keep-alive__lower_is_better.svg]]

On this benchmark, I set HTTP handler's latency to 0ms. (However some
latencies caused by small computations still exist. See [[#the-delay-that-i-overlooked][the section about this]] below.)

hunchentoot-recycling-taskmaster became fast. I think this is because
threads are accepting connections in parallel.

*** no keep-alive, sleep 0ms

[[file:pics/benchmark-result-2025-12-26/sleep_0ms/no-keep-alive/100_connections_requests_sec__sleep_0ms__no_keep-alive__higher_is_better.svg]]
[[file:pics/benchmark-result-2025-12-26/sleep_0ms/no-keep-alive/100_connections_latency(us)__sleep_0ms__no_keep-alive__lower_is_better.svg]]

On this benchmark, I set HTTP handler's latency to 0ms like above, and set
=wrk= option ="-H Connection:close"= .

Here, Woo finally takes first place, with
hunchentoot-recycling-taskmaster coming in a close second.

*** Other results

- See [[pics/benchmark-result-2025-12-26/benchmark-result-2025-12-26.org][benchmark-result-2025-12-26.org]] for graphs with "400 connections" parameters.
- See [[benchmark/benchmark-result-2025-12-26][this directory]] for raw data.

** About Woo

*** How to sleep?

Woo becomes significantly slow if the handler is even slightly delayed.
With the following setup for the "1ms sleep" benchmark, I observed poor results:"

#+begin_src lisp
  (defparameter *handler-sleep-seconds* 0)

  (defun handler-small-sleep ()
    (sleep *handler-sleep-seconds*))

  (woo:run
   (lambda (env)
     (declare (ignore env))
     (handler-small-sleep)
     '(200 (:content-type "text/plain") ("Hello, World")))
   :worker-num 8)
#+end_src

This is because Woo is an async server that handles multiple
connections simultaneously in a thread.  If a delay occurs in the
processing of one connection, all other connections in the same thread
will be delayed.  Given its async architecture, =sleep= like above is
obviously discouraged.

In cases like this, you generally don't =sleep= inside the async
server's event loop.  You run time-consuming processing outside the
event loop, and when it's finished, you notify the event loop of the
content to be sent and received, or set up a callback to be called.
[[benchmark/wookie.lisp][My code for benchmarking Wookie]] does it.

But for some reason, Woo doesn't seem to have such a mechanism.  I
couldn't find it.  [[https://github.com/quickdocs/quickdocs-api/tree/master][quickdocs-api]], which is said to use Woo, do not
seem to take such considerations into account.  [[https://github.com/TechEmpower/FrameworkBenchmarks/pull/8313][This code]] is also.
# (It may be a one reason of not good benchmark linked by [[https://www.reddit.com/r/Common_Lisp/comments/1506kqb/comment/js220a6/][this Reddit thread]].)
# another one says "Woo has no async foundation", too: https://news.ycombinator.com/item?id=29019217
# https://lisp-journey.gitlab.io/blog/why-turtl-switched-from-lisp-to-js/
# https://killtheradio.net/technology/cl-async-non-blocking-asynchronous-programming-for-common-lisp/
# https://www.reddit.com/r/lisp/comments/q4s32u/asynchronous_web_programming_in_cl/

Some people on [[https://news.ycombinator.com/item?id=29019217][here]] or [[https://www.reddit.com/r/Common_Lisp/comments/1hmxv4k/comment/m4feb4b/][here]] have said that "offloading is possible
with lparallel", but I have yet to find any code that actually does
this. The following naive code, which creates a thread in the handler,
will result in an error.

#+begin_src common-lisp
  ;; Making a thread like below don't works because `woo.ev:*evloop*' is NIL.
  (defparameter *woo-callback-threads-app*
    (lambda (_env)
      (declare (ignore _env))
      (lambda (callback)
	(bt:make-thread (lambda ()
			  (funcall callback '(200 (:content-type "text/plain") ("Hello, World"))))))))

  (clack:clackup *woo-callback-threads-app* :server :woo)
#+end_src

#+begin_src common-lisp
  ;; Binding like that does not works also.
  (let ((evloop woo.ev:*evloop*))
    (bt:make-thread (lambda (&aux (woo.ev:*evloop* evloop))
		      (funcall callback '(200 (:content-type "text/plain") ("Hello, World"))))))
#+end_src

This is because the =woo.ev:*evloop*= variable, which is the actual
event loop, cannot be referenced in a newly created thread with
=(make-thread)= and become =NIL=.

# By the way, using lparallel like above will be "one-thread-per-request" model.
# I think that will poorly perform than "one-thread-per-connection" of Hunchentoot.

This problem can be solved by directly handling =libev=, which Woo
depends on, directly handling the event loop.  Gemini-CLI wrote [[benchmark/about_woo/woo-timer-cb--written-by-gemini--working.lisp][this
code]] to do so.  It certainly works, and the benchmarks aren't
bad. However, I don't know how this code works.  (Please don't ask
me.)

So, to benchmark "sleep 1ms" in Woo, I have no choice but to use the
bad code above.  This is a huge disadvantage to Woo, but it's
unavoidable due to the lack of async support in it. This is also the
way quickdocs-api is written.

*** The delay that I overlooked

Some people might look at the above [[#keep-alive-sleep-0ms][keep-alive sleep 0ms result]] and
wonder "Why is Woo not as fast as people say?"  I thought the same
thing, investigated and found that the cause was that the handler
definition for this benchmark calls =(handler-small-sleep)=
defined above. Let's see it again:

#+begin_src common-lisp
  (defparameter *handler-sleep-seconds* 0)

  (defun handler-small-sleep ()
    (sleep *handler-sleep-seconds*))
#+end_src

This code does the following:
1. Eval the special variable (which becomes 0).
2. Call =cl:sleep= with 0 (which should return immediately).

These two processes should not take much time, but unfortunately they
may cause performance problems in Woo.  Changing =handler-small-sleep=
as follows made it faster, as shown in the following graph.

#+begin_src common-lisp
  (defun handler-small-sleep ()
    )
#+end_src

[[file:pics/benchmark-result-2025-12-26/sleep_0ms_no_special_vars/keep-alive/100_connections_requests_sec__sleep_commented_out__keep-alive__higher_is_better.svg]]
[[file:pics/benchmark-result-2025-12-26/sleep_0ms_no_special_vars/keep-alive/100_connections_latency(us)__sleep_commented_out__keep-alive__lower_is_better.svg]]

I tried some tests after a day, resulted here (with =(defparameter *handler-sleep-seconds* 0)=) :

|                                                                          | requests/sec | latency |
|--------------------------------------------------------------------------+--------------+---------|
| =(sleep *handler-sleep-seconds*)=                                        |     44770.99 | 2.27ms  |
| =(sleep 0)=                                                              |     49158.22 | 2.06ms  |
| =*handler-sleep-seconds*=                                                |    215447.83 | 0.95ms  |
| do nothing                                                               |    247789.97 | 1.00ms  |
| =(when (plusp *handler-sleep-seconds*) (sleep *handler-sleep-seconds*))= |    240050.33 | 1.26ms  |
 

So, to get the most out of Woo's performance, I should not call
=sleep= even with 0. 

In other words, when using Woo I will be troubled by overlooking any
little delays. I must examine handlers should work with very low
latency.  If Woo's situation doesn't change, it could be argued that
Woo is very fast only when your handlers perform minimal computation.
I think Woo is good for serving static pages
# or microbenchmarking.
# (but why did such a thing by Lisp? Use others, like nginx.)

* TODO TODO list

** register to systems

- [ ] Quicklisp
- [X] Ultralisp
- [ ] ocicl

** Support Lispworks

Because Hunchentoot on Lispworks does not hold a listen socket on its
structure, hunchentoot-recycling-taskmaster cannot utilize it.

** Benchmark other servers

- [ ] conserv :: Causing =cl:type-error= when I used its sample code.
- [ ] house :: Very fragile. See [[benchmark/about_house/about_house.org][my memo]] .
- [ ] teepeedee2 :: cannot be loaded by my machine because of [[benchmark/loading.org]["heap exhausted" error]].

** Ideas

- [ ] Using atomic variables -- its impact is small on SBCL, large on Allegro CL. See [[/atomic-op-taskmaster]] in this repository.
- [ ] make the number of =standby-thread-count= to variadic.

* License

BSD 2-Clause, same as Hunchentoot. See [[file:LICENSE][LICENSE]].


An experiment to improve multithreading performance of Hunchentoot without any additional dependencies. -- supplimental atomic-op experiment.

hunchentoot-recycling-taskmaster

Upstream URL

Author

License

hunchentoot-atomic-op-taskmaster

Version:

License:

Authors:

Dependencies:

hunchentoot-atomic-op-taskmaster-test

Version:

License:

Authors:

Dependencies:

hunchentoot-recycling-taskmaster

Version:

License:

Authors:

Dependencies:

hunchentoot-recycling-taskmaster-test

Version:

License:

Authors:

Dependencies:

Dependencies (2)

Dependents (0)