bp
2023-06-18
Bitcoin Protocol components in Common Lisp
BP - Bitcoin Protocol
This is a Common Lisp implementation of the various components of the Bitcoin Protocol. The serialization and deserialization utils may be used for reading the block data both from peers and from local database on disk. EC-based cryptographic operations are implemented as FFI bindings to the secp256k1 using cffi, while hash-functions are taken from ironclad. Low-level networking is implemented using usocket, HTTP client code uses aserve and JSON handling is done with jsown.
THIS BITCOIN CONSENSUS RULES IMPLEMENTATION IS NOT, AND WILL PROBABLY NEVER BE FULLY COMPLIANT WITH BITCOIN CORE IMPLEMENTATION. DO NOT RELY ON IT FOR VALIDATING YOUR MAINNET TRANSACTIONS, AS IT MAY EASILY PUT YOU OUT OF SYNC WITH THE NETWORK IN A LOT OF CORNER CASES.
Table of Contents
Installation
Elliptic curve cryptography utilities (transaction signing and
verification) use a secp256k1 library, so it must be installed
before building the bp system (either manually, or using the system
package manager if available):
# Ubuntu $ apt install libsecp256k1 libsecp256k1-dev # Arch Linux $ pacman -Syu libsecp256k1 # macOS $ brew tap cuber/homebrew-libsecp256k1 $ brew install libsecp256k1
Once secp256k1 is ready, bp can be installed via quicklisp tool:
CL-USER> (ql:quickload "bp")
Alternatively, bp system can be loaded from sources, assuming the
following Common Lisp packages are available locally:
In order to load bp from sources, evaluate the following form (this
assumes that ASDF is able to find the system definition; more on that
here):
CL-USER> (asdf:load-system "bp")
Core
Currently this library only provides utilities for stateless interaction with Bitcoin from REPL. Storage, wallet and full node capabilities are somewhere in a distant future.
Note that at this point only the symbols exported from the package
bp/core/all (nicknamed bp) can be considered an API - changes to
these functions and classes will be kept to a minimum. Everything else
will likely be changing a lot.
Chain interface
Functions bp:get-block-hash, bp:get-block and bp:get-transaction
allow to pull chain data from any external supplier specified with the
bp:with-chain-supplier macro:
CL-USER> (bp:with-chain-supplier (bprpc:node-rpc-connection :url "http://localhost:8332" :username "btcuser" :password "btcpassword") (bp:get-transaction "0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098")) #<BP/CORE/TRANSACTION:TX 0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098>
Non-nil keyword argument :encoded can be used with bp:get-block
and bp:get-transaction to return serialized transaction hex-encoded
in a string:
CL-USER> (bp:get-transaction "14...c3" :encoded t) "010000000...ae00000000"
Under the hood, these operations call corresponding generic functions
bp:chain-get-{block-hash,block,transaction} which take the supplier
object as an explicit first argument.
Model
Bitcoin data entities are represented by the following structures:
bp:block-header,bp:cblock,bp:tx,bp:txin,bp:txout,bp:script.
Functions named bp:block-* (both for bp:block-header and
bp:cblock), bp:tx-*, bp:txin-* and bp:txout-* provide access
to the components of the corresponding entities.
Serialization
Functions bp:parse and bp:serialize can be used to read and write
any Bitcoin entity from and to any octet stream respectively:
CL-USER> (ironclad:with-octet-input-stream (stream #(1 0 ... 0 0)) (bp:parse 'bp:tx in-stream)) #<BP/CORE/TRANSACTION:TX 17e590f116d3deeb9b121bbb1c37b7916e6b7859461a3af7edf74e2348a9b347> CL-USER> (ironclad:with-octet-output-stream (stream) (bp:parse 'tx out-stream)) #(1 0 ... 0 0)
Note that while bp:serialize function take an entity as its first
argument, bp:parse takes the symbol naming the class of the entity,
behaving as class method.
Functions bp:decode and bp:encode wrap above functions to decode
and encode Bitcoin entities from and to hex-encoded strings:
CL-USER> (bp:decode 'bp:tx "0100000002f8615378...e097a988ac00000000") #<BP/CORE/TRANSACTION:TX 17e590f116d3deeb9b121bbb1c37b7916e6b7859461a3af7edf74e2348a9b347> CL-USER> (bp:encode *) "0100000002f8615378...e097a988ac00000000"
Validation
Functions bp:validate and bp:validp take an entity as well as the
optional context parameters, and validate it according to an
approximation of Bitcoin consensus rules.
Both functions return t if the entity is valid, but the
bp:validate function signals an error otherwise, while the
bp:validp function simply returns nil.
Both functions assume the chain supplier context (i.e. they are called
within the body of bp:with-chain-supplier).
Dynamic variable bp:*trace-script-execution* can be used to enable
printing the steps of script execution (chain supplier macro omitted):
CL-USER> (setf bp:*trace-script-execution* t) T CL-USER> (bp:validate (bp:get-transaction "17e590f116d3deeb9b121bbb1c37b7916e6b7859461a3af7edf74e2348a9b347")) op: OP_PUSH22 payload: #(0 14 a4 b4 ca 48 de b 3f ff c1 54 4 a1 ac dc 8d ba ae 22 69 55) commands: <> stack: () op: OP_HASH160 payload: - commands: <OP_PUSH20 OP_EQUAL> stack: (#(0 14 a4 b4 ca 48 de b 3f ff c1 54 4 a1 ac dc 8d ba ae 22 69 55)) op: OP_PUSH20 payload: #(29 28 f4 3a f1 8d 2d 60 e8 a8 43 54 d 80 86 b3 5 34 13 39) commands: <OP_EQUAL> stack: (#(29 28 f4 3a f1 8d 2d 60 e8 a8 43 54 d 80 86 b3 5 34 13 39)) op: OP_EQUAL payload: - commands: <> stack: (#(29 28 f4 3a f1 8d 2d 60 e8 a8 43 54 d 80 86 b3 5 34 13 39) #(29 28 f4 3a f1 8d 2d 60 e8 a8 43 54 d 80 86 b3 5 34 13 39)) op: OP_FALSE payload: - commands: <OP_PUSH20> stack: () op: OP_PUSH20 payload: #(a4 b4 ca 48 de b 3f ff c1 54 4 a1 ac dc 8d ba ae 22 69 55) commands: <> stack: (#()) T
Validating certain entities requires additional information (block
height, transactions index, block/transaction itself, etc), which can
be packed into an instance of bp:validation-context class. For
example, validating a coinbase transaction will fail, because the only
transaction input it contains will have its previous-tx-id set to 0,
which is invalid for regular transactions. For example, to be
considered valid, a coinbase transaction must be the first transaction
of its block, while the block itself is required for amount
verification (to calculate the collected fees) and block height may be
needed to perform the BIP-0034 check, so such a
transaction can be validated using the following form:
CL-USER> (let* ((block (bp:get-block "00000000000000d0dfd4c9d588d325dce4f32c1b31b7c0064cba7025a9b9adcc")) (context (make-instance 'bp:validation-context :tx-index 0 :height 227836 :block block)) (bp:validate (bp:get-transaction "0f3601a5da2f516fa9d3f80c9bf6e530f1afb0c90da73e8f8ad0630c5483afe5") :context context))) T
Network
BP provides simple utilities for interacting with Bitcoin network - a subset of network messages and functions for establishing connections with other network nodes as well as requesting blocks and transactions.
In order to demontrate interaction with Bitcoin network, we can start
a regtest Bitcoin node:
# Start Bitcoin daemon: $ bitcoind --daemon --regtest --datadir=$HOME/.bitcoin # Generate a few blocks: $ bitcoin-cli --regtest generatetoaddress 5 $(bitcoin-cli --regtest getnewaddress) # Enable net logging: $ bitcoin-cli --regtest logging "[\"net\"]" # Tail log file to see the incoming messages: $ tail -f ~/.bitcoin/regtest/debug.log
Executing the following forms from Lisp REPL will perform a handshake with Bitcoin node:
CL-USER> (defvar *node* (make-instance 'bpnet:simple-node :network :regtest)) ... CL-USER> (bpnet:connect-peer *node* :host "127.0.0.1" :port 18444) ...
bpnet:simple-node is a very simple network node implementation that
maintains a single peer connection and provides bpnet:send-message
and bpnet:receive-message functions for sending and receiving
messages, respectively.
Alternatively, bpnet:simple-node can be asked to discover a peer
using a hardcoded DNS seed, but this is currently only supported on
mainnet. The following form will select a random peer and shake hands
with it:
CL-USER> (setf *node* (make-instance 'bpnet:simple-node :peer :discover)) ...
Objects of bpnet:simple-node partially implement chain supplier
interface - bp:chain-get-block-hash is currently not supported,
bp:chain-get-transaction only returns transactions that are
currently in the mempool or in relay set (this is an intentional
limitation of the Bitcoin gossip protocol to prevent
clients from assuming all nodes keep full transaction indexes).
bp:chain-get-block works as expected. In the example below
<block-hash> must be a hash of one of the blocks generated by the
generatetoaddress command above:
CL-USER> (bp:chain-get-block *node* <block-hash>) ...
RPC
bprpc package provides that bprpc:node-rpc-connection class which
is is an RPC client to the bitcoind RPC server. It was mentioned
above as one of the implementations of the chain supplier interface,
but it also supports the following RPC operations that correspond to
the bitcoind RPC methods (and bitcoin-cli commands) with the same
name:
bprpc:getblockhash;bprpc:getblock;bprpc:getrawtransaction;bprpc:getchaintxstats.
Note that results of RPC operations are jsown JSON structures, so
specific parts of these structures have to be extracted manually:
cl-user> (let* ((node-connection (make-instance 'bprpc:node-rpc-connection :url <url>)) (chain-stats (bprpc:getchaintxstats node-connection)) (chain-blocks (jsown:val chain-stats "window_final_block_height")) (chain-txs (jsown:val chain-stats "txcount"))) (format t "Blocks: ~a, transactions: ~a~%" chain-blocks chain-txs))
Examples
API changes
See CHANGELOG.md.
License
Copyright (c) 2019-2021 Seibart Nedor <rodentrabies@protonmail.com>
Licensed under MIT License. See LICENSE.