3bgl-shader
2024-10-12
CL-hosted CL-like DSL for generating GLSL
3bgl-shader: a Common Lisp DSL for generating GLSL shaders
Features
- looks more-or-less like CL
- type inference
- hooks for interactive use
- automatic overloaded functions
Minimal Example shader program
Just transform the vertex, and send color to be interpolated. (More examples can be found here.)
;; define a package for the shader functions, :USEing :3BGL-GLSL/CL (cl:defpackage #:shader (:use :3bgl-glsl/cl) (cl:in-package #:shader) ;; vertex attributes, need to specify types for all 'external' ;; interfaces and globals (inputs, outputs, uniforms, varyings) (input position :vec4 :location 0) (input color :vec4 :location 1) ;; final output (output out-color :vec4 :stage :fragment) ;; model-view-projection matrix (uniform mvp :mat4) ;; interface between vertex and fragment shader (interface varyings (:out (:vertex outs) :in (:fragment ins)) (color :vec4)) ;; vertex shader (names are arbitrary) (defun vertex () (setf (@ outs color) color gl-position (* mvp position))) ;; fragment shader (defun boring-fragment () (setf out-color (@ ins color))) ;; not quite CL:defconstant, need to specify a type (defconstant +scale+ 2.0 :float) ;; a helper function (defun invert-and-scale (x) ;; RETURN is GLSL return rather than CL:RETURN, and is required for ;; functions that return a value (return (* +scale+ (- 1 x)))) ;; an alternate fragment shader (defun inverted-fragment () (setf out-color (invert-and-scale (@ ins color))))
Convert to glsl
(3bgl-shaders:generate-stage :vertex 'shader::vertex) (3bgl-shaders:generate-stage :fragment 'shader::boring-fragment) (3bgl-shaders:generate-stage :fragment 'shader::inverted-fragment)
results:
// vertex shader: #version 450 out varyings { vec4 color; } outs; layout(location = 1) in vec4 color; uniform mat4 mvp; layout(location = 0) in vec4 position; void main () { outs.color = color; gl_Position = (mvp * position); } // boring fragment shader: #version 450 out vec4 outColor; in varyings { vec4 color; } ins; void main () { outColor = ins.color; } // inverted fragment shader #version 450 out vec4 outColor; const float SCALE = 2.0; in varyings { vec4 color; } ins; vec4 invertAndScale (vec4 x) { return (SCALE * (1 - x)); } void main () { outColor = invertAndScale(ins.color); }
Hooks for interactive use
Programs using 3bgl-shader can add a function to
3bgl-shaders::*modified-function-hook*
, which will be called when
shader functions are redefined. It will be passed a list of names of
updated functions. For example in the shaders above, if the
(defconstant +scale+ 2.0 :float)
form were recompiled in slime with
C-c C-c
, the hook functions would be passed the list
(SHADER::INVERTED-FRAGMENT SHADER::INVERT-AND-SCALE)
since
invert-and-scale
depends on the constant, and inverted-fragment
depends on the function invert-and-scale
. The hook function could
then see one of the fragment shaders it is using had been modified,
and arrange for a running program to try to recompile the shader
program next frame.
Current status
The compiler and type inference mostly work, including some fairly complicated shaders.
Built-in functions/types/variables from glsl version 4.50 are available, and older versions might work to the extent they are compatible with the subset of 4.50 used in a particular shader. The type inference doesn't distinguish between versions, so might allow casts that wouldn't be valid in an older version.
Error messages are mostly horrible, so most type inference failures will give incomprehensible errors.
CL style type declarations are allowed and should mostly be respected.
API isn't completely finished, so some parts may change (in particular
the base types like :vec3
may be renamed to 3bgl-glsl:vec3
at some
point.)
The external API needs more work, in particular some way to query uniforms, inputs, outputs, etc.
Currently no way to translate line/column numbers from glsl error messages back to source.
Performance is acceptable for shaders I've tested it on, but not sure
how it scales. It currently WARN
s if it takes more than 2000 passes
for type inference, which may need adjusted or disabled for larger shaders.
Currently all functions that depend on a function/global will be recompiled when things they depend on are recompiled, which can make changing function signatures or types difficult if they aren't compatible with the uses.
Recompilation may be more aggressive than it needs to be, for example if the value of a constant is changed, it shouldn't need to re-run type inference of functions that use that constant if the type didn't change.
Dependencies on uniforms are sometimes missed, dumping a bare reference to it in main function is a simple workaround.
Misc notes
Concrete types
GLSL types are currently named with keywords (though that may change
in the future), like :vec2
, :vec3
, :vec4
, :mat2x4
,
:sampler-2d-array-shadow
etc. see the
source
for details for now, though most are fairly obvious.
Component swizzles
Components of GLSL vector types like :vec4
can be accessed with
'swizzle' functions like .xyz
, so for example glsl someVec.rraa
would be (.rraa some-vec)
. Type inference should correctly use the
swizzle to determine minimum size of the vector if not specified.
Structure/interface slots
(@ var slot-name)
is a shortcut for (slot-value var 'slot-name)
,
and either will compile to var.slot
. GLSL doesn't allow specifying a
slot through a variable, so slot name must be a quoted compile-time
literal.
RETURN
Functions are required to use RETURN
to return values, they will not
return the value of the last form as in CL. A function without a
RETURN
will have a void
return type. (return (values))
can also
be used to force a void
return type, and for early exit from a
void
function.
Overloaded functions
If a function doesn't have a specific derived or specified type, it can be used with any compatible types, and the generated GLSL will have a version for each type.
For example the previous code could have had
;; X can be any type that works with scalar `*` and `-` (defun invert-and-scale (x) (return (* +scale+ (- 1 x)))) (defun inverted-fragment () (setf out-color (+ (invert-and-scale 1) ;; call 'int' version (invert-and-scale (@ ins color))))) ;; call 'vec4' version
which would generate the glsl code
#version 450 out vec4 outColor; const float SCALE = 2.0; in varyings { vec4 color; } ins; // returns a vec4 because the input is vec4 vec4 invertAndScale (vec4 x) { return (SCALE * (1 - x)); } // returns float because SCALE is a float float invertAndScale (int x) { return (SCALE * (1 - x)); } void main () { outColor = (invertAndScale(1) + invertAndScale(ins.color)); }
Type declarations
CL-style type declarations are allowed, and should interact correctly with type inference.
for example
(defun foo (x y) (declare (values :float) (:float x)) (let ((a (+ x y))) (declare (:vec2 a)) (return (.x a))))
specifies that foo
returns a float
, the first argument is also
specified to be float
, while the second isn't explicitly
restricted. The local variable A
is specified to be a vec2
, which
implicitly restricts Y
to also be something that casts to vec2
.
(declare (values))
can be used to explicitly specify void
return
type for a function.
Uniforms, input, output, interface
Uniforms are specified with (UNIFORM name type &key stage location layout qualifiers)
.
:stage
specifies in which shader stages (:vertex
,:fragment
etc)
the uniform is visible (by default the uniform is visible in all
stages, though will only be included in generated GLSL for stages in
which it is referenced).
:location N
is a shortcut for specifying the location
layout qualifier.
:layout (...)
allows specifying arbitrary layout qualifiers, argument is a plist containing qualifier and value (specify value = t
for qualifiers that don't take arguments)
:qualifiers (...)
allows specifying other qualifiers like restrict
, argument is a list of qualifiers.
;; a simple 'int' uniform, location chosen by driver or GL side of API (uniform flag :int) ;; -> uniform int flag; ;; an image2D uniform, with format, location and `restrict` specified (uniform tex :image-2d :location 1 :layout (:rg32f t) :qualifiers (:restrict)) ;; -> layout(location = 1,rg32f) uniform restrict image2D tex; ;; an atomic counter, with binding and offset specified (uniform counter :atomic-uint :layout (:binding 0 :offset 0)) ;; -> layout(binding = 0,offset = 0) uniform atomic_uint counter;
Inputs and outputs are specified with (INPUT name type &key stage location)
and (OUTPUT name type &key stage location)
where stage
specifies in which shader stages (:vertex
,:fragment
etc) the input is visible, and location
is an integer which will be
output as layout(location = 1)
in GLSL.
Interfaces between stages are specified as (INTERFACE name (&key in out uniform) &body slots)
. slots
is a list of (slot-name type)
. in
, out
and uniform
specify how the interface will be
visible, and are either T
to make it visible to all stages as
name
, or a plist of stage names and names to use for the interface in that stage.
For example (interface varyings (:out (:vertex outs) :in (:fragment ins :geometry (ins "ins" :*))) ...)
will be visible as an output
named out
in the vertex shader, as an input array named ins
in the
geometry shader, and as an input named ins
in the fragment shader.
name
and slot-name
in uniform/input/output/interface can either be
a symbol which will be automatically converted from lisp-style
to
glslStyle
, or it can be a list of (lisp-name "glslName")
to
provide an explicit translation.
Running the example programs
Example program uses GLUT and GLU, and expects GLSL version 330. Most lisp dependencies should be available in quicklisp, aside from possibly mathkit.
Load 3bgl-shader-example.asd
through ASDF or Quicklisp, then run
(3bgl-shader-example:run-example)
. That should create a window with
a spinning teapot, hit 0
-5
keys to try the various example
shaders.
If that is working, you can open example-shaders.lisp in emacs and edit them and recompile as usual from slime (C-c C-c etc).
Getting names of uniforms/vertex attributes
In addition to generated GLSL source, GENERATE-STAGE
returns a list
of uniforms as 2nd value, and attributes in 3rd value. Both are in
form (lisp-name "glslName" TYPE)
for each entry. There isn't
currently any dead-code elimination, so listed names may not actually
be active in the final shader program.
Macros
DEFMACRO
and MACROLET
work as in CL code, and expansion runs on
host so can use arbitrary CL.
Array variables
There is partial support for arrays, though type inference doesn't work completely correctly on them and local array variables can't be initialized when bound.
Currently, array types are specified as (<base-type> <size>)
. (CL
style array/vector types may be supported at some point in the future)
(defun foo () (let ((a)) ;; can't currently initialize local array variables (declare ((:float 8) a)) ;; specify size/base type (setf (aref a 1) 1.23) ;; access as in CL (return (aref a 1)))
Compute Shaders
Compute shaders work pretty much like other stages, except you can't
specify input
s/output
s, and must specify the workgroup size for
kernel invocations. The workgroup sizes are specified with the
layout
declaration on the main kernel entrypoint. Compute shaders
also expose a number of constants describing an individual
invocation's relationship to the entire run: gl-local-invocation-id
, gl-global-invocation-id
, gl-work-group-id
, gl-num-work-groups
, and gl-work-group-size
, all :uvec3
, and gl-local-invocation-index
, an :int
.
;; define a kernel that runs in units of 8x8x8 blocks (defun some-kernel () (declare (layout (:in nil :local-size-x 8 :local-size-y 8 :local-size-z 8))) ;; xyz takes values from (uvec3 0 0 0) to (uvec3 7 7 7) (let ((xyz (.xyz gl-local-invocation-id))) ...))
Shared variables in compute shaders
Compute shader shared
variables are defined with SHARED
, which
takes a name and type (including array types) as arguments
;; define a shared array with 256 :float elements ;; can be accessed with (aref temp x) or (setf (aref temp x) ...) as in CL (shared temp (:float 256)) ;; a shared uint (shared foo :uint)
Shader Storage Buffer Objects
Limited support for SSBO, use (interface <name> (:buffer t) ...)
;; makes FOO and BAR available in shaders for read/write ;; BAR is an array of mat4, size depends on size of bound buffer (interface ssbo (:buffer t :layout (:binding 0 :std430 t)) (foo :vec4) (bar (:mat4 :*)))
Structures
Preliminary support for defining structures with |defstruct|, doesn't currently accept any of the extra options from |cl:defstruct|, and slot syntax is |(slot-name type)|.
Can't currently infer type of structs, so need to |declare| them by hand.
;; define a struct with a float, array of 8 int, and arbitrary ;; length array of vec4 (defstruct foo (a :float) (b (:int 8)) (c (:vec4 :*)))