Alternative I/O mechanism to a stream or vector

fast-io

# fast-io

**Now with
[static-vectors](https://github.com/sionescu/static-vectors)
support!**

```lisp
(deftype octet '(unsigned-byte 8))
(deftype octet-vector '(simple-array octet (*)))
```

Fast-io is about improving performance to octet-vectors and octet
streams (though primarily the former, while wrapping the latter).
Imagine we're creating messages for the network. If we try and fill an
octet-vector with 50 bytes, 50000 times, here are the results (SBCL
1.0.57):

<table>
<tr>
 <th></th>
 <th align=right><tt>vector-push-extend</tt>:</th>
 <th align=right><tt>flexi-streams</tt>:</th>
 <th align=right><tt>fast-io</tt>:</th>
</tr>
<tr>
 <td align=right>Time:</td>
 <td align=right>0.767s</td>
 <td align=right>2.545s</td>
 <td align=right>0.090s</td>
</tr>
<tr>
 <td align=right>Bytes consed:</td>
 <td align=right>104,778,352</td>
 <td align=right>274,452,768</td> 
 <td align=right>18,373,904</td>
</tr>
</table>

(See `t/benchmarks.lisp` for the exact code used.)

It *should* be surprising that it takes a nontrivial effort to achieve
relatively decent performance to octet-vectors, but probably isn't.
However, fast-io provides a relatively straightforward interface for
reading and writing either a stream or a vector:

```lisp
;;; Write a byte or sequence, optionally to a stream:

(with-fast-output (buffer [STREAM | :vector | :static])
 (fast-write-byte BYTE buffer))

(with-fast-output (buffer [STREAM | :vector | :static])
 (fast-write-sequence OCTET-VECTOR buffer [START [END]]))

;;; Read from a vector or stream:

(with-fast-input (buffer VECTOR [STREAM])
 (fast-read-byte buffer))

(with-fast-input (buffer VECTOR [STREAM])
 (let ((vec (make-octet-vector N)))
 (fast-read-sequence vec buffer [START [END]])))
```

## Multi-byte and Endianness

Fast-io provides a host of read and write functions for big- and little-endian reads. See the [Dictionary](#reading-and-writing) below.

## Static Vectors

You may now specify `:static` instead of a stream to
`WITH-OUTPUT-BUFFER`. This returns an octet-vector created with
[static-vectors](https://github.com/sionescu/static-vectors),
which means that passing the buffered data directly to a foreign
function is now that much more efficient:

```lisp
(let ((data (with-fast-output (buffer :static)
 (buffer-some-data buffer))))
 (foreign-send (static-vectors:static-vector-pointer data))
 (static-vectors:free-static-vector data))
```

Note that the restriction for manually freeing the result remains.
This avoids multiple inefficient (i.e., byte-by-byte) copies to
foreign memory.

## Streams

Obviously, the above API isn't built around Lisp streams, or even
gray-streams. However, fast-io provides a small wrapper using
`trivial-gray-streams`, and supports `{WRITE,READ}-SEQUENCE`:

```lisp
(let ((stream (make-instance 'fast-io:fast-output-stream)))
 (write-sequence (fast-io:octets-from '(1 2 3 4)) stream))
```

Both `fast-input-stream` and `fast-output-stream` support backing a
stream, much like using the plain fast-io buffers. However, using the
gray-streams interface is a 3-4x as slow as using the buffers alone.
Simple benchmarks show the gray-streams interface writing 1M 50-byte
vectors in about 1.7s, whereas simply using buffers is about 0.8s.
Consing remains similar between the two.

## Dictionary

### Octets

Most functions operate on or require octet-vectors, i.e.,

```lisp
(deftype octet () '(unsigned-byte 8))
(deftype octet-vector '(simple-array octet (*)))
```

Which is exactly what is defined and exported from `fast-io`. Also:

* `make-octet-vector LEN` Make an octet-vector of length `LEN`.
* `octets-from SEQUENCE` Make an octet-vector from the contents of `SEQUENCE`.

### Buffers

* `make-input-buffer &key VECTOR STREAM POS` Create an input buffer for use with input functions. `:vector` specifies the vector to be read from. `:stream` specifies the stream to read from. `:pos` specifies the offset to start reading into `VECTOR` If both `:vector` and `:stream` is provided, the input buffer reads from the vector first, followed by the stream.
* `make-output-buffer &key OUTPUT` Create an output buffer for use with output functions. `:output` specifies an output stream. If `:output :static` is specified, and static-vectors is supported, output will be to a static-vector.
* `finish-output-buffer BUFFER` Finish the output and return the complete octet-vector.
* `buffer-position BUFFER` Return the current read/write position for `BUFFER`.

* `with-fast-input (BUFFER VECTOR &optional STREAM (OFFSET 0)) &body body` Create an input buffer called `BUFFER`, optionally reading from `VECTOR`, followed by reading from `STREAM`. If `OFFSET` is specified, start reading from this position in `VECTOR`.
* `with-fast-output (BUFFER &optional OUTPUT) &body BODY` Create an output buffer named `BUFFER`, optionally writing to the stream `OUTPUT`. This will automatically `FINISH-OUTPUT-BUFFER` on `BUFFER`. Thus the `with-fast-output` form evaluates to the completed octet-vector.

### Reading and Writing

* `fast-read-byte INPUT-BUFFER &optional (EOF-ERROR-P t) EOF-VALUE` Read a byte from `INPUT-BUFFER`. If `EOF-ERROR-P` is `t`, reading past the end-of-file will signal `CL:END-OF-FILE`. Otherwise, it will return `EOF-VALUE` instead.
* `fast-write-byte BYTE OUTPUT-BUFFER` Write a byte to `OUTPUT-BUFFER`.
* `fast-read-sequence SEQUENCE INPUT-BUFFER &optional (START 0) END` Read from `INPUT-BUFFER` into `SEQUENCE`. Values will be written starting at position `START` and, if `END` is specified, ending at `END`. Otherwise values will be written until the length of the sequence, or until the input is exhausted.
* `fast-write-sequence SEQUENCE OUTPUT-BUFFER &optional (START 0) END` Write `SEQUENCE` to `OUTPUT-BUFFER`, starting at position `START` in `SEQUENCE`. If `END` is specified, values will be written until `END`; otherwise, values will be written for the length of the sequence.

For multi-byte reads and writes requiring endianness, fast-io provides functions in the following forms:

* `write[u]{8,16,32,64,128}{-be,-le}`: E.g., `(write32-be VALUE BUFFER)` will write the specified 32-bit value to the specified buffer with a *big-endian* layout. Likewise, `(writeu16-le VALUE BUFFER)` will write an *unsigned* 16-bit value in *little-endian* layout.
* `read[u]{8,16,32,64,128}{-be,-le}`: Similarly, `(read64-le BUFFER)` will read a 64-bit value from the buffer with little-endian layout.

fast-io

Upstream URL

Author

License

fast-io

Version:

License:

Authors:

Dependencies:

fast-io-test

Dependencies:

Dependencies (4)

Dependents (26)