## simple-neural-network

2022-11-07

Simple neural network

# 1Description

**simple-neural-network** is a Common Lisp library for creating, training and
using basic neural networks. The networks created by this library are
feedforward neural networks trained using backpropagation. The activation
function used by the neurons is `A(x) = 1.7159 * tanh(0.66667 * x)`

.

**simple-neural-network** depends on the **cl-store** and **lparallel** libraries.

# 2License

**simple-neural-network** is released under the GPL-3 license. See the LICENSE
file for details.

# 3API

The functions are in the *simple-neural-network* package. You can use the
shorter *snn* nickname if you prefer.

The library works with double floats. Your inputs and targets must therefore be
vectors of `double-float`

numbers. For better results, they should also be
normalized to contain values between -1 and 1. The `find-normalization`

helper
function can be used to generate normalization and denormalization functions
from sample inputs, but it might not be adapted to every use case.

If `lparallel:*kernel*`

is set or bound, some computations will be done in
parallel. This is only useful if the network is big enough, because the
overhead of task management can instead slow things down for small networks.

`(create-neural-network input-size output-size &rest hidden-layers-sizes)`

Create a neural network having *input-size* inputs, *output-size* outputs, and
optionally some intermediary layers whose sizes are specified by
*hidden-layers-sizes*. The neural network is initialized with random weights
and biases.

```
(train neural-network inputs targets learning-rate
&key batch-size momentum-coefficient)
```

Train the *neural-network* with the given *learning-rate* and
*momentum-coefficient* using some *inputs* and *targets*. The weights are
updated every *batch-size* inputs.

`(predict neural-network input &optional output)`

Return the output computed by the *neural-network* for a given *input*. If
*output* is not `nil`

, the output is written in it, otherwise a new vector is
allocated.

`(store neural-network place)`

Store the *neural-network* to *place*, which must be a stream or
a pathname-designator.

`(restore place)`

Restore the neural network stored in *place*, which must be a stream or
a pathname-designator.

`(copy neural-network)`

Return a copy of the *neural-network*.

`(index-of-max-value values)`

Return the index of the greatest value in *values*.

`(same-category-p output target)`

Return `t`

if calls to `index-of-max-value`

on *output* and *target* return the
same value, and `nil`

otherwise. This function is only useful when the neural
network was trained to classify the inputs in several categories (when targets
contain a 1 for the correct category and and -1 for all the other categories).

`(accuracy neural-network inputs targets &key test)`

Return the rate of good guesses computed by the *neural-network* when testing
it with some *inputs* and *targets*. *test* must be a function taking an output
and a target returning `t`

if the output is considered to be close enough to
the target, and `nil`

otherwise. `same-category-p`

is used by default.

`(mean-absolute-error neural-network inputs targets)`

Return the mean absolute error on the outputs computed by the *neural-network*
when testing it with some *inputs* and *targets*.

`(find-normalization inputs)`

Return four values. The first is a normalization function taking an input and returning a normalized input. Applying this normalization function to the inputs gives a data set in which each variable has mean 0 and standard deviation 1. The second is a denormalization function that can compute the original input from the normalized one. The third is the code of the normalization function. The fourth is the code of the denormalization function.

```
(find-learning-rate neural-network inputs targets
&key batch-size momentum-coefficient epochs
iterations minimum maximum)
```

Return the best learing rate found in *iterations* steps of dichotomic search
(between *minimum* and *maximum*). In each step, the *neural-network* is
trained *epochs* times using some *inputs*, *targets*, *batch-size* and
*momentum-coefficient*.

# 4Examples

Neural network for the XOR function:

```
(asdf:load-system "simple-neural-network")
(defun normalize (input)
(map 'vector (lambda (x) (if (= x 1) 1.0d0 -1.0d0)) input))
(defun denormalize (output)
(if (plusp (aref output 0)) 1 0))
(defvar inputs (mapcar #'normalize '(#(0 0) #(0 1) #(1 0) #(1 1))))
(defvar targets (mapcar #'normalize '(#(0) #(1) #(1) #(0))))
(defvar nn (snn:create-neural-network 2 1 4))
(dotimes (i 1000)
(snn:train nn inputs targets 0.1))
(denormalize (snn:predict nn (normalize #(0 0))))
-> 0
(denormalize (snn:predict nn (normalize #(1 0))))
-> 1
(denormalize (snn:predict nn (normalize #(0 1))))
-> 1
(denormalize (snn:predict nn (normalize #(1 1))))
-> 0
```

Neural network for the MNIST dataset, using parallelism (2 threads):

```
;; Note: the mnist-load function used below is defined in "tests/tests.lisp".
(setf lparallel:*kernel* (lparallel:make-kernel 2))
(defvar nn (snn:create-neural-network 784 10 128))
(multiple-value-bind (inputs targets) (mnist-load :train)
(dotimes (i 3)
(snn:train nn inputs targets 0.003d0)))
(multiple-value-bind (inputs targets) (mnist-load :test)
(snn:accuracy nn inputs targets))
-> 1911/2000
```

# 5Tests

The tests require the **fiveam** and **chipz** libraries. They can be run with:

`(asdf:test-system "simple-neural-network")`