chipi


*** Chipi - A (House) Automation for Common Lisp

!!! I'm running the core framework in production 24/7 for many months. But still considered beta. !!!

****  Motivation

I use home automation in my home. Currently utilizing [[https://www.openhab.org/][openHAB]]. While openHAB has many features and supports many devices and so on, it is also a heavy beast.

So my aim was to eventually replace openHAB. There are a few things still missing to do this. The current state of the Chipi project lacks the following for me to be able to replace openHAB:

- support for KNX bus (finished)
- a REST interface (in progress, implemented, documentation missing)
- a UI to interact with the system

See plugin section below.

**** What does Chipi currently do

***** Chipi consists of the following components:

- item: items are a primitive to represent the temperatur of a sensor, the state of light in your house, the state of your window being open or closed, etc.

- itemgroup: items can be organized in groups

- binding: bindings are a means to update an items value or distribute a value change to somewhere

- persistence: persistences allow to persist and load the state of items. Currently two types of persistences are supported:
    - simple-persistence that just stores the current state to file and can retrieve it
    - historic persistence with a support for influxdb to store each changed item value in influxdb and also retrieve values for a specific time range to calculate averages, peaks or whatever.

- rules: rules are kind of scripts that are run on certain 'triggers'. Triggers may be a change of an item value, or a cron timing.

**** How does it work

Chipi is largely based on [[https://github.com/mdbergmann/cl-gserver][Sento]], a Common Lisp Actor Framework where features like thread-safety, queuing and an event-bus are very handy for Chipi.

How it works is to define and combine all the mentioned primitives in a file. This file is a Common Lisp source file. The definition of the primitives (item, binding, ...) is done using a defined DSL (domain specific language) which largely consists of a few macros. But it is also possible to define your own macros, functions or variables.

A full production example that defines items, bindings, persistences and rules can be seen [[https://github.com/mdbergmann/cl-etaconnector/blob/master/eta-hab.lisp][here]]. This script, very much based on Common Lisp, is a full example configuration of Chipi. At the end of the script evaluation of all items, persistences and rules will start their work.

This particular example uses Common Lisp libraries like [[https://github.com/bendudson/py4cl][py4cl]] (to talk to ina219 analog 2 digital converter for a pressure sensor in a cistern plugged to the GPIOs of a Raspberry PI 4), [[https://github.com/edicl/drakma][Drakma]] to query data of a [[https://www.shelly-support.eu/][Shelly]] power switch or [[https://github.com/snmsts/cserial-port][cserial]] to communicate with some other device to retrieve temperatur sensor data.

**** But let's go slowly

The top of the Chipi standalone script should include this (=eval-when=) in order to make sure that dependencies are loaded properly. But this is not mandatory if you find another way of loading dependencies.

#+begin_src lisp
(eval-when (:compile-toplevel :load-toplevel :execute)
  (ql:quickload :chipi))  ; at least this if you don't have more
#+end_src

Next thing is that you might want to define a separate package using =defpackage=, it allows to inherit symbols using =:use= or =:import-from= that you want to use in the script.

***** Defining the global environment
The first thing Chipi specific that should be defined is the 'config'. This is done by =(defconfig)=. This expression should be before items, persistences and rules because it defines and starts the underlying actor system and sets up some other necessary structures.

***** Defining persistences
Persistences are used to persist the value of an item and to retrieve it back. The influx persistence also allows to retrieve values in a certain time range.

****** Simple persistence
To define a 'simple' persistence you do:

#+begin_src lisp
  (defpersistence :simple
    (lambda (id)
      (make-simple-persistence id :storage-root-path #P"some-path")))
#+end_src

The =:simple= here is the persistence id passed along to the lambda factory function.

The only information =simple-persistence= requires is where to store the item values. Each item is stored in its own file and just stores the last value. The value plus a timestamp (=GET-UNIVERSAL-TIME=) is stored in JSON format like this:

#+begin_src
{"value":66.19999694824219,"timestamp":3907315980}
#+end_src

Under normal circumstances you don't need to deal with simple persistences directly. To retrieve an item value you'd call =ITEM:GET-VALUE=. Though it is possible to call =PERSP:FETCH= and retrieve the current value from the persistence directly.

****** Influx persistence
A persistence that allows retrieving item values of a time range, i.e. to build daily or weekly average values, is influx. Currently only influx v2 is supported. Define such persistence like this:

#+begin_src lisp
  (defpersistence :influx
      (lambda (id)
        (make-influx-persistence
         id
         :base-url "http://your-host:8086"
         :token "your-token"
         :org "your-org"
         :bucket "your-bucket")))
#+end_src

To retrieve a range of values you can call =PERSP:FETCH= with an optional range instance. Currently only =RELATIVE-RANGE= exists.

***** Bindings
Bindings are a means to interact with sources or targets, meaning they allow interactivity with the item value.
Bindings are defined as part of an item definition and *not* on toplevel. A basic binding definition looke like this:

#+begin_src lisp
  (binding :initial-delay 5
           :delay 60
           :pull (lambda () 0) ;;pull value from somewhere
           :transform (lambda (value) (1+ value))
           :push (lambda (value)) ;; push to somewhere else
           :call-push-p t)
#+end_src

This binding uses the =pull= function to retrieve a value, which is passed on to the item value. When to =pull= is determined by =:initial-delay= and =:delay= in seconds where the former is an 'initial delay' and the latter a repetetive delay. =:call-push-p= actually defines whether the =push= function is called when the value was updated. The =push= function can be used to push the value elsewhere if required. Both =pull= and =push= are optional. Though one of the two should be used, otherwise the binding doesn't make much sense.
What is =transform=? It is optional but can be used to transform the value retrieved with =pull=. =transform= should return a transformed value.

Thinking further, I'd like to have bindings that are specific to pulling from http, serial, or whatever, and allow to be specified in that way. The =pull=, =push= functions are very generic but may require repetition and are not enough specialized.

See next how to define and attach bindings on items.

(Also see below for bindings available as plugins.)

***** Defining item groups
#+begin_src lisp
  (defitemgroup 'group1 "Group1")
#+end_src

This defines a group ='group1= with label "Group1". See below for how to add items to groups.

***** Defining items
The simplest form to define an item is:

#+begin_src lisp
  (defitem 'myitem "My Item" 'integer)
#+end_src

This defines a plain item that can hold a value. You could manually use =SET-VALUE= function to give it a value or =GET-VALUE= to retrieve its value. In some cases this is useful in 'rules'. See later.

The three parameters define an id of the item (for easier lookup), a label and a type hint. The type hint is not necessary (can be specified as =NIL=) unless you want to use influx db where under the hoods it is necessary to bring the value in the right format based on what type the value is in. Checkout [["https://github.com/mdbergmann/chipi/blob/main/src/persistence-influx.lisp"][influx persistence]] for which types are supported. However, even if not required it might be a good idea to define the type for clarity.

Usually you'd want to at least define an initial value. You can do so by:

#+begin_src lisp
  (defitem 'myitem "My Item" 'integer
    :initial-value 0)
#+end_src

Items can be defined for group membership. It can be done like so:

#+begin_src lisp
  (defitem 'myitem "My Item" 'integer
    :initial-value 0
    :group 'group1)
#+end_src

****** Define and attach bindings

In many cases you want to retrieve the item value from somewhere and maybe also want to push it somewhere else once it was set. This is where bindings come in. There can be more =binding= definitions on an item but this only really makes sense if you plan to =push= to more places. An item definition with binding looks like this:

#+begin_src lisp
  (defitem 'myitem "My Item" 'integer
    :initial-value 0
    (binding :initial-delay 0.1
             :delay 30
             :pull (lambda () (do-some-http-get))
             :push (lambda (value) (do-some-http-post))
             :call-push-p t))
#+end_src

****** Attaching persistences on the item definitions

Persistences, as defined above can now be 'attached' to the item like this:

#+begin_src lisp
  (defitem 'myitem "My Item" 'integer
    :initial-value 0
    (binding :initial-delay 0.1
             :delay 30
             :pull (lambda () (do-some-http-get))
             :push (lambda (value) (do-some-http-post))
             :call-push-p t)
    :persistence '(:id :simple
                   :frequency :every-change
                   :load-on-start t)
    :persistence '(:id :influx
                   :frequency :every-20s))
#+end_src

It is possible to attach multiple. In the case above both have different purposes. The =:simple= persistence is used to just store the latest value and can recover from it when told so using =:load-on-start=.

The =:influx= persistence will just store every value change to the database.

The =:frequency= defines how often the value is stored. =:every-change= will store the value to the persistence on every change of the item value.
=:every-20s= (form example) stored the value every 20 seconds recirring. The notation here is =:every-N<s|m|h>= where N is the number, s (seconds), m (minutes) and h (hours).

***** Defining rules

Rules are scripts that are run on certain triggers. Triggers are the change of one or more item values or one or more cron definitions. Example:

#+begin_src lisp
  (defrule "My Rule"
    :when-item-change 'my-item
    :when-cron '(:minute 0 :hour 0)
    :do (lambda (trigger)
          (case (car trigger)
            (:item
             (let ((item (cdr trigger)))
               (format t "Item changed: ~a~%" item)
               ;; asynchronously do something with the value
               (future:fcompleted (item:get-value item)
                   (value)
                 (do-some-action-with-value value))))
            (:cron
             (format t "Cron triggered: ~a~%" (cdr trigger))
             (do-some-action))))
#+end_src

Rules can be triggered by item value changes. To subscribe to certain items one has to use =:when-item-change= with the item id of the item definition. Use multiple =:when-item-change= to subscribe to multiple item changes.

The other trigger is cron. The lowest granularity is minutes. Specify cron triggers with =:with-cron=. Also multiple triggers can be defined. The =cdr= of the trigger variable is the cron expression.

When a cron trigger is specified as '=(:boot-only t)= then this means that the rule is called immediately after initialization, but only once.

**** Redefining persistences, items and rules

Those elements can be redefined, meaning re-evaluated to update a changed configuration. The re-evaluation usually happens in Emacs in the same way as functions are re-evaluated using =C-c C-c= or by reloading the whole script.

Caveeat: if a persistence is changed and re-evaluated also the items where it is attached have to be re-evaluated.

**** Logging

The project uses log4cl. You can change the log level for '=(chipi)= to suite your needs of granularity. The underlying Sento actor framework will log extensively on =:debug= level so a good idea is to silence this via =(log:config '(sento) :warn)=.

*** Available binding plugins

**** KNXNet/IP binding: see [[https://github.com/mdbergmann/chipi/blob/main/bindings/knx/README.md][README]]



knx-conn

binding-knx

cl-mock

fiveam

usocket-server

binding-knx/tests

alexandria

binding-arrows

cl-cron

drakma

local-time

parse-float

sento

timer-wheel

yason

chipi/tests

cl-base64

cl-ppcre

com.inuoe.jzon

hunchentoot

ironclad/digest/sha256

ironclad/kdf/bcrypt

ironclad/kdf/scrypt

ironclad/mac/hmac

marshal

snooze

chipi-web

Test system for chipi-web, the web API for chipi.

chipi-web/tests

chipi

Upstream URL

Author

License

binding-knx

Version:

License:

Authors:

Dependencies:

binding-knx/tests

Authors:

Dependencies:

chipi

Version:

License:

Authors:

Dependencies:

chipi/tests

Authors:

Dependencies:

chipi-web

Version:

License:

Authors:

Dependencies:

chipi-web/tests

Authors:

Dependencies:

Dependencies (20)

Dependents (0)