configuration.options README

1Introduction

The configuration.options system provides

• data structures and functions for hierarchical configurationschemata and options
• sources of option values (builtin sources are configuration files,environment variables and commandline options)
• and handling of changes of option values

All of these aspects are extensible via protocols.https://travis-ci.org/scymtym/configuration.options.svg

2STARTED Tutorial

    (defun first-line-or-less (string)
      (let ((end (min 70 (or (position #\Newline string) (length string)))))
        (subseq string 0 end)))

    (defun provider-table (service)
     (loop :for (name . provider) :in (service-provider:service-providers/alist service)
        :collect `(,name ,(first-line-or-less (documentation provider t)))))

    (defun provider-table/sorted (service)
      (sort (provider-table service) #'string< :key #'first))

Implementing configuration processing using the configuration.options system involves at least three steps:

1. *Specifying a Schema
2. *Constructing and Populating a Configuration based on the schema
3. *Querying a Configuration

2.1Names

Since options (and the corresponding schema items) are organizedinto a hierarchy, option names are a sequence of multiplecomponents. The notation COMPONENT₁.COMPONENT₂.… is used whenrepresenting names as strings.

"Wildcard names" are names in which one or more components is :wild or :wild-inferiors.

The following functions deal with names:

     (mapcar (lambda (example)
               (destructuring-bind (function arguments) example
                 (let ((*package* (find-package :configuration.options)))
                   (list (format nil "(~(~A~)~{ ~S~})" (symbol-name function) arguments)
                         (prin1-to-string (apply function arguments))))))
             `((configuration.options:parse-name      ("a.b.\"c.d\""))
               (configuration.options:make-name       ("a.b.c"))
               (configuration.options:make-name       ("a.*.c"))
               (configuration.options:make-name       ("a.**.c"))
               (configuration.options:make-name       (("a" "b" "c")))
               (configuration.options:name-components (,(configuration.options:make-name "a.**.c")))
               (configuration.options:name-equal      (,(configuration.options:make-name "a.b.c")
                                                        ,(configuration.options:make-name "d.e.f")))
               (configuration.options:name-matches    (,(configuration.options:make-name "d.**.g")
                                                        ,(configuration.options:make-name "d.e.f.g")))
               (configuration.options:name-equal      (,(configuration.options:make-name "a.b.c")
                                                        ,(configuration.options:make-name "d.e.f")))
               (configuration.options:merge-names     (,(configuration.options:make-name "a.b.c")
                                                       ,(configuration.options:make-name "d.e.f")))))

Form	Result
(parse-name "a.b.\"c.d\"")	("a" "b" "c.d")
(make-name "a.b.c")	("a" "b" "c")
(make-name "a.*.c")
(make-name "a.**.c")
(make-name ("a" "b" "c"))	("a" "b" "c")
(name-components #<WILDCARD-NAME a.**.c {100D6E5FE3}>)	("a" :WILD-INFERIORS "c")
(name-equal ("a" "b" "c") ("d" "e" "f"))	NIL
(name-matches #<WILDCARD-NAME d.**.g {100D6EFFC3}> ("d" "e" "f" "g"))	T
(name-equal ("a" "b" "c") ("d" "e" "f"))	NIL
(merge-names ("a" "b" "c") ("d" "e" "f"))	("a" "b" "c" "d" "e" "f")

2.2Specifying a Schema

A schema can be defined in multiple ways:

• "Manually" via multiple function and method calls
• Declaratively using configuration.options:eval-schema-spec
• Declaratively using configuration.options:define-schema

Since the third method is likely the most commonly used (and uses the same syntax as the second method), it is probably sufficient to only discuss configuration.options:define-schema. Here is an example:

     (configuration.options:define-schema *my-schema*
       "Configuration schema for my program."
       ("logging"
        ("appender"                :type    '(member :file :standard-output)
                                   :default :standard-output
                                   :documentation
                                   "Appender to use.")
        ((:wild-inferiors "level") :type    '(member :info :warning :error)
                                   :documentation
                                   "Package/module/component log level.")))

The above code creates a schema object and stores it in theparameter *my-schema*. The schema consists of two items:

1. logging.appender with allowed values :file and:standard-output and default value :standard-output
2. a "template" option named logging.**.level with allowed values:info, :warning and :error and without a default value

     (describe *my-schema*)

   #<STANDARD-SCHEMA  (2) (C 0) {1002EE8E03}>

   Tree:
     <root>
     │ Configuration schema for my program.
     └─logging
       ├─appender
       │   Type    (MEMBER FILE STANDARD-OUTPUT)
       │   Default :STANDARD-OUTPUT
       │   Appender to use.
       └─**
         └─level
             Type    (MEMBER INFO WARNING ERROR)
             Default <no default>
             Package/module/component log level.

2.2.1STARTED Types

As demonstrated above, each schema item has an associated type that describes the allowed values of associated options, as types tend to do. In addition to that, types are used to control the parsing and unparsing of option values. For better or worse, schema item types are specified using Common Lisp type specifiers such as (member :info :warning :error) in the above example. The validation, parsing and unparsing behavior for types is implemented using an extensible protocol. This protocol is used by, for example, the configuration.options-and-puri system to add support for additional types.

The builtin types are:

      (sort
       (remove-duplicates
        (alexandria:mappend
         (lambda (method)
           (let ((specializer (third (c2mop:method-specializers method))))
             (when (typep specializer 'c2mop:eql-specializer)
               (let* ((type-specifier (c2mop:eql-specializer-object specializer))
                      (documentation  (documentation type-specifier 'type))
                      (description    (cond
                                        ((eq (symbol-package type-specifier)
                                             (find-package '#:common-lisp))
                                         "«standard»")
                                        (documentation
                                         (first-line-or-less documentation))
                                        (t
                                         "«not documented»"))))
                 (list (list type-specifier description))))))
         (c2mop:generic-function-methods #'configuration.options:value->string-using-type))
        :test #'eq :key #'first)
       #'string-lessp :key #'first)

AND	«standard»
BOOLEAN	«standard»
CONFIGURATION.OPTIONS:DIRECTORY-PATHNAME	A pathname syntactically suitable for designating a directory.
CONFIGURATION.OPTIONS:FILE-PATHNAME	A pathname syntactically suitable for designating a file.
INTEGER	«standard»
LIST	«standard»
MEMBER	«standard»
NULL	«standard»
OR	«standard»
PATHNAME	«standard»
STRING	«standard»

2.2.2TODOSub-schemata

2.3Constructing and Populating a Configuration

Configurations are created from schemata by first creating an emptyconfiguration object and then populating it with option objectscorresponding to schema item objects in the schema:

     (defparameter *my-configuration* (configuration.options:make-configuration *my-schema*))

The created configuration is empty:

     (describe *my-configuration*)

   #<STANDARD-CONFIGURATION  (0) {10076052B3}>

   Tree:
     <empty>

There are several ways to create option objects from schema item objects:

1. "Manually", options can be created using the make-optiongeneric function (this also works if the corresponding to schemaitems have wild names):

           (let* ((name        "logging.mypackage.myparser.level")
                  (schema-item (configuration.options:find-option
                                name *my-schema*
                                :interpret-wildcards? :container)))
             (setf (configuration.options:find-option name *my-configuration*)
                   (configuration.options:make-option schema-item name)))
   

   #<STANDARD-OPTION  logging.mypackage.myparser.level: (MEMBER INFO WARNING ERROR) <no value> {100B8A4FE3}>

   Note that the schema item named logging.**.level matches the requested name because of its :wild-inferiors name component. Also note that creating an option object does not automatically assign a value to it (even if the schema item specifies a default value).

   The schema item lookup and make-option call in the above code can be done automatically, shortening the example to:

           (configuration.options:find-option
            "logging.mypackage.mylexer.level" *my-configuration*
            :if-does-not-exist :create)
   

   #<STANDARD-OPTION  logging.mypackage.mylexer.level: (MEMBER INFO WARNING ERROR) <no value> {100B8DD5C3}>

2. Using a "synchronizer" which integrates data from sources suchas configuration files into configuration objects:

           (defun populate-configuration (schema configuration)
             (let ((synchronizer (make-instance 'configuration.options:standard-synchronizer
                                                :target configuration))
                   (source       (configuration.options.sources:make-source :defaults)))
               (configuration.options.sources:initialize source schema)
               (configuration.options.sources:process source synchronizer)))
   
           (populate-configuration *my-schema* *my-configuration*)
   

   The above example uses the simple "default values" source which instantiates option objects for all schema items with non-wild names and sets their values to the respective default values (if any) stored in corresponding schema items.

After creating these option objects, the configuration looks like this:

     (describe *my-configuration*)

   #<STANDARD-CONFIGURATION  (3) {1002F54013}>

   Tree:
     <root>
     └─logging
       ├─appender
       │   Type    (MEMBER FILE STANDARD-OUTPUT)
       │   Default :STANDARD-OUTPUT
       │   Value   :STANDARD-OUTPUT
       │   Sources DEFAULT:
       │             :STANDARD-OUTPUT
       │   Appender to use.
       └─mypackage
         ├─mylexer
         │ └─level
         │     Type    (MEMBER INFO WARNING ERROR)
         │     Default <no default>
         │     Value   <no value>
         │     Package/module/component log level.
         └─myparser
           └─level
               Type    (MEMBER INFO WARNING ERROR)
               Default <no default>
               Value   <no value>
               Package/module/component log level.

In a more realistic setting, populating the configuration would be done exclusively using a synchronizer but with a "cascade" of sources [fn:1] instead of just the "default values" source.

2.4TODOQuerying a Configuration

2.5TODOTracking Changes of Option Values

2.6More on Sources

*Constructing and Populating a Configuration introduced the"source" and "synchronizer" concepts by demonstrating the defaultvalues source.

In more realistic settings, a combination of multiple sources like (from highest to lowest priority)

1. Commandline options
2. Environment variables
3. Configuration file(s) and directories
4. Default values

will be used. Cascades of this kind can be constructed by instantiating the :cascade source with appropriate subordinate sources:

     (configuration.options.sources:make-source
      :cascade
      :sources '((:commandline)
                 (:environment-variables)
                 (:config-file-cascade :config-file "my-program.conf"
                                       :syntax      :ini)
                 (:defaults)))

#<CASCADE-SOURCE  (4) {100B9D0953}>

A similar cascade of sources is constructed by the :common-cascade source without the need for manually specifying the involved sources.

     (configuration.options.sources:make-source
      :common-cascade :basename "my-program" :syntax :ini)

#<COMMON-CASCADE-SOURCE  (4) {100B962693}>

Currently available sources are:

     (provider-table/sorted 'configuration.options.sources::source)

Name	Documentation
:CASCADE	This source organizes a set of sources into a prioritized cascade.
:COMMANDLINE	This source obtains option values from commandline arguments.
:COMMON-CASCADE	This source implements a typical cascade for commandline programs.
:CONFIG-FILE-CASCADE	This source implements a cascade of file-based sources.
:DEFAULTS	This source assigns default values to options.
:DIRECTORY	Collects config files and creates corresponding subordinate sources.
:ENVIRONMENT-VARIABLES	This source reads values of environment variables.
:FILE	This source reads configuration data from files.
:STREAM	This source reads and configuration data from streams.

The :stream (and therefore :file, :config-file-cascade and :common-cascade) source supports the following syntaxes:

     (provider-table/sorted 'configuration.options.sources::syntax)

Name	Documentation
:INI	Parse textual configuration information in "ini" syntax.
:XML	This syntax allows using some kinds of XML documents as

2.7STARTED Configuration Debugging

With multiple configuration sources such as environment variables and various configuration files, it can sometimes be hard to understand how a particular option got its value (or did not get an expected value). This is true in particular for users who cannot poke around inside the program.

To alleviate this problem, the configuration.options system provides a simple configuration debugging facility aimed at users. This facility can be enabled by calling

(configuration.options.debug:enable-debugging STREAM)

To enable debug output to STREAM unconditionally

(configuration.options.debug:maybe-enable-debugging PREFIX :stream STREAM)

To enable debug output to STREAM if the environment variable PREFIXCONFIGDEBUG is set

The intention is that a program using this system calls one of these functions before configuration processing starts.

For example, using the schema defined above:

     (setf (uiop:getenv "MY_PROGRAM_LOGGING_APPENDER") "file")

     (configuration.options.debug:enable-debugging *standard-output*)

     (let* ((schema        *my-schema*)
            (configuration (configuration.options:make-configuration schema))
            (synchronizer  (make-instance 'configuration.options:standard-synchronizer
                                          :target configuration))
            (source        (configuration.options.sources:make-source
                            :common-cascade :basename "my-program" :syntax :ini)))
       (configuration.options.sources:initialize source schema)
       (configuration.options.sources:process source synchronizer))

   Configuring COMMON-CASCADE-SOURCE with child sources (highest priority first)

     1. Environment variables with prefix mapping
        MY_PROGRAM_LOGGING_APPENDER=file (mapped to logging.appender) -> "file"

     2. Configuring CONFIG-FILE-CASCADE-SOURCE with child sources (highest priority first)

        1. Current directory file "my-program.conf" does not exist

        2. User config file "/home/jmoringe/.config/my-program.conf" does not exist

        3. System-wide config file "/etc/my-program.conf" does not exist

3STARTED Integration with the architecture.service-provider System

The architecture.service-provider system allows defining services and providers of these services. The integration described here adds the ability to automatically define a configuration schema for a given service and use a configuration object to choose, instantiate and configure a provider:

    digraph {

      rankdir="LR"
      node [shape="box"]

      service
      provider1 [shape=record,label="provider 1|slot a: boolean"]
      service -> provider1 [arrowhead="none",weight=1000]
      service -> provider2 [arrowhead="none"]

      derive_schema [label="derive schema",shape="ellipse"]
      schema [shape="record",label="schema|item provider: (member 1 2)|item 1.a: boolean"]
      service -> derive_schema -> schema

      make_configuration [label="make configuration",shape="ellipse"]
      configuration [shape="record",label="configuration|option provider = 1|option 1.a = t"]
      schema -> make_configuration
      make_configuration -> configuration
      provider1 -> configuration [style="dashed",label="selects",arrowtail="normal",dir="back",weight=0]

      instantiate_provider [label="instantiate provider", shape="ellipse"]
      instance [shape="record",label="instance|slot a = t"]
      configuration -> instantiate_provider
      instantiate_provider -> instance
      provider1 -> instance [arrowtail="emptytriangle",dir="back",weight=0]
    }

This functionionality is provided in the separate configuration.options-and-service-provider system:

    (asdf:load-system :configuration.options-and-service-provider)

3.1STARTED Deriving a Schema for A Schema

     (service-provider:define-service my-service)

     (defclass my-provider () ((a :initarg :a :type string)))
     (service-provider:register-provider/class
      'my-service :my-provider :class 'my-provider)

     (describe
      (configuration.options.service-provider:service-schema
       (service-provider:find-service 'my-service)))

   #<STANDARD-SCHEMA  (1) (C 1) {10083C2913}>

   Tree:
     <root>
     │ Configuration options of the MY-SERVICE service.
     ├─provider
     │   Type    (PROVIDER-DESIGNATOR-MEMBER MY-PROVIDER)
     │   Default <no default>
     │   Selects one of the providers of the MY-SERVICE service for
     │   instantiation.
     └─my-provider
       │ Configuration of the MY-PROVIDER provider.
       └─a
           Type    STRING
           Default <no default>

3.2STARTED Creating a Configured Provider

     (let* ((schema        (configuration.options.service-provider:service-schema
                            'my-service))
            (configuration (configuration.options:make-configuration schema)))

       (populate-configuration schema configuration)
       (setf (configuration.options:option-value
              (configuration.options:find-option "provider" configuration))
             :my-provider
             (configuration.options:option-value
              (configuration.options:find-option "my-provider.a" configuration))
             "foo")

       (describe (service-provider:make-provider 'my-service configuration)))

   #<MY-PROVIDER {1007F19023}>
     [standard-object]

   Slots with :INSTANCE allocation:
     A  = "foo"

3.3TODOTracking Service Changes

4TODOReference

5TODORelated Work

• https://github.com/Shinmera/universal-config/
• https://github.com/Shinmera/ubiquitous
• https://docs.python.org/3/library/configparser.html
• cl-config

6Settings :noexport:

7Footnotes

[fn:1] See *More on Sources