optolab_wasm

What is this?

OptoLab is a program for creating interactive simulations and visualizations with instant feedback. It also lets you create animations or static images and render them in high quality. First and foremost it is a creative environment for programmers, mathematicians, and video bloggers making videos about programming or math. The program may interest you if you have used Shadertoy or Manim.

The foundation of this program is its own component system, extremely composable, where literally any parameter of any type can be parameterized: a number can be a formula, a matrix a product of other matrices, a choice between variants an If branch depending on other components, and shader text can be generated by templates in Jinja syntax. Components can be parameterized, substituted, and used inside one another with any depth of nesting.

There are two ways to use this component system:

Before reading the rest of the documentation it is highly recommended to play around with different scenes in the program.

Interface

Canvas

It occupies most of the screen; the main content is drawn here. It shows the image from the component named final_image. How that image is produced is explained in the “How drawing works” chapter.

Load

Existing scenes can be loaded through the Load menu. It has thumbnails, which are generated automatically through a setting in the scene. To load a user scene, go to “Edit scene” and press “Import” there.

Control scene

The “Control scene” window is the user interface. First comes the selection of the current subscene (internally, a subscene is a patch whose name starts with user.; patches are explained later), or you can pick an animation to watch, and below it are the user settings for the current subscene. The scene author designs this user interface themselves using the component named interface. The assumption is that the scene is opened by a user either completely unfamiliar with it or only weakly familiar; so you can put a description there, and all the settings should be quite friendly to the user. By the way, switching the subscene resets the values of all components to their defaults. Subscenes may share the same components, and without the reset the user would see not what the developer designed and tested, but the dirty state left over from twiddling settings in another subscene.

Edit scene

The “Edit scene” window is the interface for scene developers. There you can edit patches (explained later) and all the components in each patch.

The “Export”, “Export pretty”, “Import” buttons let you dump the scene into the internal text format and load it back. This format is quite human-readable and can even be edited by hand or by LLMs! It will be described later. It exactly mirrors how the scene looks in the interface: the recursive nesting of inline components, component values, and their order. The interface just gives you more convenient buttons for editing the values.

Next to the obvious “Save” and “Load” buttons there is also a “Load from memory” button: all built-in scenes are embedded right into the program, and this button loads the embedded copy - for the case when the scene file is not in the file system.

The first checkbox, “Allow scene editing”, is the switch from user mode to developer mode; it enables editing of patches and components. Once it is on, switching between subscenes (patches) resets only the computed state - your component edits are kept. Now, to switch between subscenes (patches), you use not Control scene but the “Select current patch” menu at the bottom.

“Components settings” - buttons for sorting components and patches (by name, by type, by computation time) and display checkboxes: whether to show the component type, whether to highlight special names with color, whether to show computation time next to the name.

Then comes the search menu; as you type text into it, it highlights all components whose name contains the entered substring. It also searches by component type. That is, you can type “Bool” and get all boolean components. You can jump into the search by pressing Ctrl+F anywhere in the program and starting to type.

Then comes the list of all patches, “Patches”, where each patch stores scene components. In short, a patch is a folder grouping several components; but patches can also be overlaid on top of other patches, and that will be explained separately.

A patch can be local (Inline) or read from another scene (FromFile). It can also have dependencies, and inside it live all the components belonging to that patch. The order in which patches and components are displayed in these lists affects nothing; the only order that matters is the order of patches inside a dependency list.

Settings

A small number of program settings.

By default the canvas takes up all available space. So the aspect ratio setting exists to let you see your scene the way it will look in animations. I usually set it to 16:9, because I render my animations with that aspect ratio. Once I had Free aspect ratio (i.e. no such setting at all), I made some scene, and when I rendered it, some elements went off screen and I could not understand why. The thing is, my window was not 16:9 but roughly 16:8.56 - the window title bar and the menu made the render area smaller, so everything looked slightly different. This setting exists so that this never happens again.

Next there is scene autosave into the file scene_dump.ron; it sits in the folder the program was launched from (usually next to the executable). If the program crashed or something, find this file and use it. WARNING: if you launch the program, this file gets overwritten! So after a crash, first copy it somewhere, and only then open the program.

Performance

Aggregated computation time for each component. Everything here is fairly obvious, except that the parentheses with an ellipsis show the minimum and maximum time. Besides CPU time there is a separate GPU column - the execution time of shaders on the video card. GPU measurements arrive asynchronously, with a frame or two of delay.

A very useful window even if you don’t write components in Rust and use at most the shader component. It lets you pick shader rendering settings such that it computes in optimal time. It lets you find the bottlenecks of your computations.

“us” means microseconds (a thousandth of a millisecond).

Memory

A very technical window that can be useful to you only if you develop your own components in Rust. It shows indirect indicators of memory usage per component: the number of allocations during the last computation and the allocation bandwidth (how much memory in total was allocated and deallocated during the computation). It cannot show the most useful thing - exactly how much memory each component holds right now - due to technical complexity; maybe I will add that in the future, but it will cost performance.

Errors

Just a window that shows the errors of all components in one place. Convenient so you don’t have to run around patches and components hunting for the cause of an error.

Components

For now let’s assume there are no patches and we simply have one set of components. What is a component, and how do you do something useful with them?

Each component is a unit of computation: it computes some data and can take other components as arguments for the calculation. All components are connected into one acyclic graph: there are components that depend on no one, there are those that depend on them, and so on - but dependencies never close into a circle. If you want cycles (to compute something repeating across frames), there is the @prev(target, init, clock) expression in the component name field, which will be described later.

Also, some component names have special meaning for the program itself: for example, final_image, time, mouse_state, interface. See the full list with descriptions in the special components documentation.

Let’s start with an example of the simplest component - Bool. When you create it in the interface, its Simple variant is created by default, which contains just a checkbox. This checkbox shows the current value of this component: true or false. At the bottom, the “Value” section of the interface shows the component’s current value. In scene syntax it looks like Bool(Simple(false)); I will use this syntax from here on. You can copy it with the “Copy RON” button and paste it into a component with the “Paste RON” button.

(
    patches: {
        "main": Inline((
            dependencies: [],
            local_components: {
                "bool": Bool(Simple(false)),
            },
        )),
    },
    current_patch: "main",
)
---
bool

If you change the Bool component’s variant to And, the component now starts requiring two other components of type Bool, to compute the logical AND of those two values. If you changed this in the interface, your component looks like this:

(
    patches: {
        "main": Inline((
            dependencies: [],
            local_components: {
                "bool_and": Bool(And(Inline(Simple(false)), Inline(Simple(false)))),
            },
        )),
    },
    current_patch: "main",
)
---
bool_and

What is going on here? I have to explain that there are two kinds of components - named (Named) and inline (Inline). Named components have a name by which other components can reference them. Inline components are, so to speak, “anonymous” components: they exist as a single instance right in the place where you embedded them. This is very convenient for keeping all sorts of small computations inside one component instead of creating a million named ones. If we change our And to use named components, it becomes:

(
    patches: {
        "main": Inline((
            dependencies: [],
            local_components: {
                "bool1": Bool(Simple(false)),
                "bool2": Bool(Simple(false)),
                "bool_and_named": Bool(And(Named("bool1"), Named("bool2"))),
            },
        )),
    },
    current_patch: "main",
)

In the interface, switching between inline and named components is done by pressing the 📌 button. Also, if you right-click that icon, you will see a menu where you can copy the inline component, look at its value, or extract it into a named one. In scene text, inline components are written as Inline(<component text>) and can have any nesting depth!

Component names may contain spaces, dots, and hyphens, but may not contain @, (, ) and , - these characters are reserved for expressions like @prev(...), described later.

The next example of a very important component is the real number component Float. With it you can do arithmetic using its variants: Simple, Sum, Sub, Mul, Div, Neg and so on.

For example, this component first adds 2 + 3, then multiplies the result by 4. Every argument is itself an inline Float component and can be changed or replaced with another computation.

(
    patches: {
        "main": Inline((
            dependencies: [],
            local_components: {
                "formula": Float(Mul(
                    Inline(Sum(Inline(Simple(2.0)), Inline(Simple(3.0)))),
                    Inline(Simple(4.0)),
                )),
            },
        )),
    },
    current_patch: "main",
)
---
formula

There are also ways to convert one component into another. Bool has a FromFloat variant, Float has FromBool, Vec2X (extract one Float out of a Float2) and so on.

Next is a list of the basic components with short descriptions; it is recommended to read it so you know where to look.

Basic data components:

System components:

Drawing components:

Computation order

The component system recomputes only what changed. This is one of the most important parts of the whole program. On each update, the only components recomputed are those that changed themselves or whose dependencies changed.

How can a component change at all? There are several ways:

Thanks to this you can build a complex system that computes something once at initialization, and then this computation does not depend on time or anything else, yet is actively used by other components.

Interactive cameras (Camera, Camera3D) have a special trait: by moving the camera with the mouse you change the component itself - the current position is written back into its fields. So when you export the scene, the camera position you currently see is what gets saved.

Each component has its own current value, called the cache. When a scene is initialized nobody has a cache yet, so all components are computed. On a subsequent computation the component is handed its current cache, which is used as the previous value. Usually a component compares its new value against it, and this is how it reports whether it changed. Thanks to this the component system does no extra recomputation when the dependencies changed but the value actually did not. A great example is Float::If: its inactive branch may hold a value that changes, yet the component itself reports that it did not change. The previous value can also be used to avoid new allocations and only modify the existing data (e.g. for a physics simulation).

But some components are not capable of knowing when they really changed - for example, the image component drawn by a shader. The shader’s dependencies - the variables passed into the shader - changed, so the whole shader is redrawn. But it may happen that not a single pixel changed during that redraw. Comparing the previous value with the current one is too expensive, so the shader reports it changed merely from the fact that it was redrawn, not from the fact that it really changed. This can play a dirty trick on you if you build a physics simulation on shaders and textures and use the @prev feature.

The previous value as physics simulation state is relevant only to those who develop their own components in Rust; for ordinary users there is @prev, which has its own chapter.

The cache is almost never copied; the original cache is always what gets passed into every component. This is a computation speed optimization, because the program was made from the start to operate on components whose cache can be tens of megabytes. The exception is the RememberClonable component, which can store copies of certain lightweight components that can afford it.

Computation errors

All errors are shown in the interface at the bottom of a component, or next to the field that causes the error.

There are two kinds of errors: structural errors and errors inside a component. Structural means a dependency references a nonexistent component, or the dependencies form a cycle. When such an error occurs, nothing is computed at all.

If an error occurs inside a component during computation (e.g. the type of a dependency component does not fit), it is attached to that component. The component’s value is lost, and all components depending on it start producing the error Failed to get cache for <component name>. Such error chains are normal: fix the root error, and the rest will disappear by themselves after recomputation.

So when an error appears you need to fix it, otherwise the scene either will not work or will work incorrectly.

How drawing works

The canvas always shows exactly one component: the Framebuffer named final_image. There is no other path to the screen - whatever you want to see, you build it out of components until the result lands in this framebuffer. If there is no final_image, the canvas has nothing to show.

A Framebuffer is an image stored in a GPU texture. There are two main ways to fill it:

The two ways compose freely. A framebuffer drawn by a shader can be placed into a RenderDrawable scene as a DrawFramebuffer object; and any framebuffer can be passed into a shader as a uniform and processed further. Multi-pass rendering is just a chain of framebuffers: one shader draws into a framebuffer, the next shader reads it as a texture, and so on, until the last one ends up in final_image.

Every framebuffer chooses its own resolution. The canvas shows final_image stretched to fill it, composited over the special background_color component. So for final_image you usually take the size from the component named screen_size, which the app fills with the current canvas size in pixels - this way one framebuffer pixel maps to one canvas pixel and nothing gets stretched.

Coordinate spaces

Three coordinate spaces are used throughout the components, and it helps to know them by name:

Matrix2::FromCamera converts a Camera into a screen-to-world matrix, and the GlslShader documentation has ready-made GLSL helpers for converting between all three spaces.

Aspect ratio

By default the canvas takes all available window space, so its aspect ratio is arbitrary. Since the shortest side of screen space always spans [-1, 1], the central square [-1, 1] x [-1, 1] is visible at any aspect ratio. So if you want a scene to work everywhere - wide 16:9, vertical 9:16, and anything in between - keep everything important inside that central square. The aspect ratio setting in Settings exists exactly to preview this: switch between 16:9 and 9:16 and check that nothing important goes off screen.

Starting a new scene

The built-in scene none (in the Load menu) is an empty skeleton with all the necessary starting components already wired up: final_image as a RenderDrawable framebuffer sized by screen_size, an empty draw collection, an interactive camera, time with render_to_time, mouse_state, interface, and the other special components.

The fastest first visible result with objects: load none, enable scene editing, create an Arrow component, and add it to the draw collection. With a shader: create a GlslShader component with the smallest shader body from the GlslShader documentation, then change final_image from RenderDrawable to DrawShader, keeping screen_size as the size and pointing shader at your new component.

Patches

Each patch stores a set of components. You can look at patches as a way to organize several components with a common meaning (a folder), or as a means of component reuse (create some recurring set of components in one file and use it in many other scenes via FromFile). But the main use is a literal patch of components, overlaid on top of your existing components.

Let’s look at an example. By default there is just one patch - main, which holds the main components. Say you built an interactive scene there where two spheres can be dragged around. The sphere positions are set by components a and b of matrix type Matrix3. Now what if you want to make an animation where these spheres move in a specific way? In the interactive scene these components use the “Simple” variant, which allows passing them into the user interface so the user can change them and play around. In an animation, you need these matrices to somehow depend on the time component “time” so your animation can work.

Then you can create a separate patch “anim.moving” in which you override these components! You create new components with the same names “a” and “b” and write your motion into them, with a dependency on the current time (the Parametric variant). You must also state in the patch that it depends on the other patch “main”. Then, when you select “anim.moving” as the current patch, all components will be taken from the main patch, but the components “a” and “b” will be taken from your animation patch, and the spheres will move the way you defined.

Here is a smaller version of the same idea. Switch to anim.override: its value replaces the one from main, so doubled changes from 2 to 6 even though doubled itself still comes from main.

(
    patches: {
        "main": Inline((
            dependencies: [],
            local_components: {
                "value": Float(Simple(1.0)),
                "doubled": Float(Mul(Named("value"), Inline(Simple(2.0)))),
            },
        )),
        "anim.override": Inline((
            dependencies: ["main"],
            local_components: {
                "value": Float(Simple(3.0)),
            },
        )),
    },
    current_patch: "main",
)

Patches can have several dependencies; if the same component is defined in several of them, the patch standing lower in the dependency list wins.

For convenience, the interface marks the current patch with green text (current), and the patches from its dependencies with blue text (used) after the patch name.

Also for convenience, each component’s interface shows whether it is currently active. If a component’s name is colored gray, it means this component is currently taken from another patch, and the bottom of the component says from where exactly. If the component’s color is white, then this exact component is active and its value is used in the scene.

Switching the current patch always recomputes the scene from scratch: all caches, compiled shaders, and @prev cycles are reset. In developer mode your component edits are kept - only the computed state is reset.

By default all patches are Inline - that is, they store their own components. There is also the FromFile patch type; it takes the components from a patch in another file, with an important restriction: such a patch cannot have dependencies. Component changes in a FromFile patch are not saved back to the file they came from. For convenience, modified components are shown with yellow text (changed). If you want to change or override them, copy the component to your own patch and modify it there.

All this patch dependency and component overriding business is very similar to how inheritance works in programming languages - including the classic complaint that with a lot of it, it becomes hard to track where a specific thing came from. So far, given that this is scene definition rather than code, and with this interface, patches have not been that confusing in practice; but only time will tell. I also plan lambda components with an explicit input and output interface as another reuse mechanism - as an addition to patches, not a replacement.

There are special patch names:

Animations and the thumbnail are covered later.

User interface

There are two kinds of users: you (the one reading this documentation) - the user of OptoLab, and the second kind - those for whom you make your scene. This chapter is about the interface for the second kind: those who will interact with your scene, not with OptoLab itself.

OptoLab has a very interesting system that I am proud of and have never seen anywhere before. Usually an interface is written in code while the program is being developed - a user of the finished program can no longer do that. In OptoLab, the interface is defined through another interface, right while the program is being used.

The user interface is defined through the interface component, which has type Interface. It has both ordinary interface elements like Label and Named (lets the user modify a named component), and interface combinators: Horizontal, Vertical, CollapsingHeader. For details see the Interface component documentation; I don’t want to duplicate it here.

The main variant is Named, where you specify a component name, and that component can then be modified in the user interface. Not all components and not all subcomponents can be passed here. For example, Bool::Simple supports the user interface, since it is just a checkbox, but Bool::And does not, because - what could the user even specify there? Names of other components? They don’t know the names of other components; they are a user, not a developer. To see all the possible components, look into scene “User interface showcase” in the “Technical” folder in the “Load” menu. For now, rely on intuition. If a variant looks like a simple control and does not dump internals outward (component names, formulas) - it is most likely supported. Some component variants are created exclusively for the user interface, e.g. Usize::Variants, which lets the user pick an integer just by pressing a button with a label on it.

Another feature of this interface is that you can make a parametric interface. That is, use the If variant. If your scene has settings expressed as Bool components, those same components can be used to turn pieces of the interface related to that setting on and off!

And since the interface is a component, its parts can also be extracted into named components and overridden through other patches.

Previous value @prev

Say you want to make a scene where, every new frame, some component uses its own value from the previous frame to do iterative computation: path tracing or a physics simulation.

Or you want to compute something that depends on itself. For example, passing gravitational forces to a body that itself changes that gravitational field. You have a physical body component that takes the gravitational field as an argument. And you have a gravitational field component that takes a set of massive bodies as an argument. You cannot naively make them depend on each other, otherwise you get a cycle, and it becomes impossible to decide how to compute them.

For these cases you need the feature of using some component’s value from the previous frame. In the place where you specify the name of a component your component depends on, you can use special syntax: @prev(value, init, time). All three parameters are component names. The first parameter is the thing whose previous-frame value you want. The second component is which value to take when the previous frame’s value does not exist (in other words - initialization). And the third is the cycle clock: usually the time component. init must differ from value. Referencing yourself through @prev is forbidden.

Under the hood, the computation order is arranged so that all components asking for @prev(value, ...) are computed before value itself: they read value’s still-untouched cache from the previous frame, so the previous value never needs to be copied anywhere (as said earlier, a cache copy operation does not exist in the program at all).

There is an important subtlety to these computations - if your component value changed, then on the next frame whatever uses its @prev() will be recomputed. This is logical behavior, but you must be careful with it. Because some components can say they changed (as mentioned earlier about the shader image component) when they actually did not. For example, you make a physics simulation on shaders, storing the simulation state in a texture. Your simulation receives the time change as an argument for its computations. But when you pause time, the shader receives dt = 0 - that is an argument change, so the texture is redrawn, although in fact not a single pixel of it changes. But your texture component will still say it changed, and on the next frame whatever depends on it will recompute too, and it will itself recompute as well (since in a physics simulation the state component must ultimately depend on itself). Thus you get stuck in an infinite cycle of pointless computations every frame that do nothing but eat your CPU and GPU.

The third argument, time, was added exactly for this. It dictates when the cycle makes a step. While time changes, the component that uses @prev(value, ..., time) is recomputed on every tick - even if value did not change (otherwise the cycle could stall). And if time does not change (paused), the cycle freezes - even if value reports changes, like that texture from the example above. Changing other inputs while time is stopped gives exactly one recomputation, but does not start the cycle. This way you can stop the chain of pointless computations by tying your computations to time. This is a reasonable feature, since 99% of all recurrent computations OptoLab is meant for are most likely something time-related (physics computations etc.).

Also, when the time value decreases (time reset to 0, rewinding), the value cache of this cycle is reset, and init is used as @prev again, even if the value value exists. This is the reset mechanism for this complex feedback system, without resetting the whole cache.

This example uses an ordinary Float::Progress as the clock so you can step the cycle manually. Open clock and move its slider forward: every change advances value by one. Stop changing it and the cycle freezes. Move it backwards and value resets through initial.

(
    patches: {
        "main": Inline((
            dependencies: [],
            local_components: {
                "initial": Float(Simple(0.0)),
                "clock": Float(Progress(0.0)),
                "previous": Float(Sum(
                    Named("@prev(value, initial, clock)"),
                    Inline(Simple(0.0)),
                )),
                "value": Float(Sum(Named("previous"), Inline(Simple(1.0)))),
            },
        )),
    },
    current_patch: "main",
)

There is also a simplified syntax - @prev(value, init) - without time. It is recommended only when you have made your own components in Rust and know that they report 100% correctly when they changed and when they did not. Or when you use the previous value to untangle a cycle in computations.

Feedback through shaders

The classic Shadertoy scenario - a shader reading its own texture from the previous frame: physics on textures, cellular automata, path tracing. There is a technical restriction here: a shader cannot read the texture it is currently drawing into. And @prev onto yourself, as said above, is forbidden.

So feedback through a single buffer needs one copy component, the CopyAnother variant of Framebuffer:

A live example is the game_of_life_shader scene; it is exactly this scheme.

If instead you have a chain of several buffers (A reads @prev(C, ...), C reads B, B reads A), the copy component is not needed - the cycle is already broken by the @prev between different components.

Animations

OptoLab natively supports offline rendering of animations into PNG or MP4 files.

Patch

All animations must be stored in a patch starting with anim..

The patch must have a component named render_to_time of type Float. It specifies the animation duration. There is also the render_stop_now component of type Bool - if it becomes true, rendering stops early.

If render_to_time equals zero, a static picture is rendered.

The render_to_time value is read once at render start, so it cannot depend on time or change over the course of the animation (unlike, say, motion_blur_frames, which you can change every frame if you like).

During animation rendering the is_rendering component is set to true. You can branch settings on it: compute fast in interactive mode for responsiveness, and crank up maximum quality in offline rendering.

For more quality you can use the antialiasing_size component; it sets how many times to enlarge the canvas while drawing, so it can later be shrunk back by averaging neighboring pixels.

There is also a bunch of other settings related to motion blur; see them in the special components section.

One remark about motion blur: you can use it as a way to effectively increase the simulation fps, without actually increasing the fps of the output video file. Useful when your computations are tied to @prev and you need more quality within one frame: be it path tracing or physics simulations.

Rendering

For rendering there are several options in the program’s console interface:

Each option has its own settings; see their --help.

By default an animation is rendered to MP4, and it is assumed that you have ffmpeg installed and on PATH.

If the animation is not a static picture, it is saved into three files: <name>.mp4, <name>.start.png, <name>.end.png. These frames are saved before motion blur is applied - that is, they are the true first and last frames. This is for convenience in video editing software, where the first and last frames come in handy very often.

Another quirk of video editors: for the colors in the video to match the colors in the PNGs, the video must be saved with special codec settings. There are two options for this: --davinci and --premiere-pro.

Sometimes, when rendering very heavy shaders at very high resolution, the operating system can freeze for a long time, or even kill the program doing the rendering. For this reason the GlslShader component has the ability to split drawing into several independent vertical strips. This adds some overhead to rendering, but the computer does not freeze, you can keep working on it, and the program does not get killed.

How rendering advances time

Offline rendering does not use the interactive clock: speed_ms is ignored, count_to is turned off, and the renderer directly writes each sample’s timestamp into the time component.

Timestamps are spaced uniformly. With duration T = render_to_time and render fps F, the renderer produces N = ceil(F * T) + 1 output frames with step D = T / N; output frame i gets the base time i * D. Two consequences worth knowing: the step is slightly smaller than 1 / F, and the last frame lands at (N - 1) * D, strictly before T. Fractional fps values are truncated to a whole number.

There are two separate fps settings:

Motion blur subdivides each output frame. With S = motion_blur_frames subframes and exposure E = motion_blur_exposure, subframe j (from 0 to S - 1) of output frame i is sampled at time (i + E * j / S) * D, and the output frame is the average of all S subframes. So the shutter opens exactly at the frame’s base time, and the exposure compresses the samples toward the beginning of the frame - they are not centered around it. With E = 0, all subframes sample the same base time.

With motion_blur_exposure_simulate_gaps enabled, the clock instead advances uniformly in K = max(S, ceil(S / E)) steps per output frame; only the first S steps are rendered and averaged, while the remaining steps simulate through the shutter-closed gap with the same timestep.

The saved .start.png and .end.png are single unblurred subframes, not averaged output frames: the first subframe of the first output frame (always at time 0) and the last subframe of the last output frame. If render_stop_now stops rendering early, .end.png captures the last subframe that was actually rendered.

Thumbnail

Your scene’s thumbnail is defined in the thumbnail patch; the program simulates the scene up to time render_to_time and then saves that frame as the thumbnail.

If there is no thumbnail patch, the current patch and time 0 are simply used for the preview.

To regenerate thumbnails, use the generate-thumbnails CLI command.

Text representation of scenes

All scenes are serialized in the RON format and exactly mirror how the components look in the interface.

A scene as text looks exactly like the “Edit scene” window: first comes the set of patches, and after it the name of the current patch. Each patch has dependencies and its own set of components. Each component either references a named one or contains an inline component directly. The file skeleton looks like this:

(
    patches: {
        "main": Inline((
            dependencies: [],
            local_components: {
                "brightness": Float(Simple(1.0)),
                "render_to_time": Float(Simple(0.0)),
            },
        )),
        "anim.demo": Inline((
            dependencies: ["main"],
            local_components: {
                "render_to_time": Float(Simple(2.0)),
            },
        )),
    },
    current_patch: "main",
)

This is just a skeleton, not a complete scene. For the minimal working set of components see the none scene - an empty skeleton that is convenient to start a new scene from. Any built-in scene can be dumped to text (with the “Export” button or the CLI command) and used as an example; a good, bigger sample is game_of_life_shader: shaders, feedback through @prev, an animation patch.

A FromFile patch looks like this in text:

"planets": FromFile((
    file: "scenes/torus.ron",
    patch: "planets",
)),

How a component looks in text form will be shown for each component in its own documentation. And the fastest way to learn the text of a specific configured component is to build it in the interface and press “Copy RON”.

A few syntax rules that are easy to trip over:

On top of RON there is my own format called bigstring. It exists to move multiline strings into a separate block at the end of the file. If a RON file looks like this:

(
    name: "demo",
    script: "let x = 1;\nprint(x);",
)

then in the BIGSTRING format it would look like this:

(
    name: "demo",
    script: <<<BIGSTRING 1>>>,
)
===BIGSTRINGS===
<<<BIGSTRING 1 1>>>
let x = 1;
print(x);
<<<END_BIGSTRING 1 1>>>

In general, every big string simply has its own unique number; at the place of use only that number is written (<<<BIGSTRING 1>>>), and at the place of definition also a second number (<<<BIGSTRING 1 1>>>). Usually the second number is 1, but if you nest one bigstring file inside another (e.g. scene text inside a scene), you can increase it so that the inner string’s end marker does not collide with the outer one’s.

The ===BIGSTRINGS=== section is optional: a scene without multiline strings is plain RON, and a multiline string can also be written as an ordinary RON string with \n - import accepts both. Export, however, always moves all multiline strings into bigstring and numbers them in order. The main convenience is that the text between the markers is stored as is: no escaping of quotes and backslashes, which is especially important for shaders.

Editing works like this: you saw <<<BIGSTRING 5>>> in a component field - look for the block <<<BIGSTRING 5 1>>> at the end of the file and edit the text between the markers as is. To add a new big string, take any free number. Every block in the section must be used in the scene body; unused blocks are an import error.

You can check an edited scene in two ways: in the interface - with the “Import” button, after which all problems will be visible in the Errors window; without the interface - by rendering one frame with the render-frame command, component errors will be printed to the console.

Comparison with Shadertoy

This program lets you do with shaders almost everything you can do on Shadertoy, and more.

Missing for now:

Same as on Shadertoy:

New features (implementing most of the Shadertoy.com Roadmap):

Component index

How to read this reference

Each component’s page starts with its RON schema: the exact shape accepted in scene text. The general syntax rules are in the “Text representation of scenes” chapter; the schemas themselves use a few conventions:

Many variants are marked “intended for user interfaces”: such a variant exists to give the component an editable control (checkbox, slider, color picker), which can also be shown to the end user in the Control scene window through Interface::Named.

List of components with short description

The listed component doesn’t have to be top-level with the type Float2::Simple or Matrix2::Simple. It goes through all inline subcomponents of every listed component and shows controls for each of the corresponding type.

The app reads the top-level component named screen_points (see the special components documentation). Groups can be nested: array may contain other ScreenPoints components, each with its own camera, or plain Collections.

Special components

Some top-level component names are read or overwritten by the application itself.

Some top-level component names are read or overwritten by the application itself. They fall into three groups.

Written by the app

The app overwrites these components; your components read them as inputs.

Read by the app

You define these components; the app reads them to know what to show.

Offline rendering settings

These are read by offline animation rendering; the Animations chapter explains how they interact.

Component reference

Bool

RON schema

enum Bool {
    Simple(bool),
    Variants(bool, string, string, enum VariantsType {
        Label,
        Radio,
        ComboBox,
    }),
    Not(Bool),
    And(Bool, Bool),
    Or(Bool, Bool),
    Xor(Bool, Bool),
    Less(Float, Float),
    Greater(Float, Float),
    EqualFloat(Float, Float),
    EqualUsize(Usize, Usize),
    FromUsize(Usize),
    FromFloat(Float),
    FromProgram(RhaiProgram),
    GlslNeedsRecompilation(GlslShader),
    MouseLeftButton(MouseState),
    MouseMiddleButton(MouseState),
    MouseRightButton(MouseState),
    MouseLeftDragButton(MouseState),
    MouseMiddleDragButton(MouseState),
    MouseRightDragButton(MouseState),
    OnceButton(bool),
    Button {
        text: Text,
        enabled: Bool,
        clicked: bool,
    },
    If {
        condition: Bool,
        then: Bool,
        otherwise: Bool,
    },
}

Documentation

Boolean value. Can be used in other components as a condition.

Usize

RON schema

enum Usize {
    Simple(usize),
    Variants(usize, Vec<(string, usize)>, enum VariantsType {
        Label,
        Radio,
        ComboBox,
    }),
    FromBool(Bool),
    FromFloat(Float),
    FromProgram(RhaiProgram),
    If {
        condition: Bool,
        then: Usize,
        otherwise: Usize,
    },
}

Documentation

Unsigned whole-number value.

Float

RON schema

enum Float {
    Simple(f64),
    Progress(f64),
    Angle(f64),
    Positive(f64),
    Slider {
        value: f64,
        min: Float,
        max: Float,
    },
    FromBool(Bool),
    FromUsize(Usize),
    FromTime(Time),
    CalculationTime(OnlyNamed),
    FrameCalculationTime(AlwaysChanged),
    FromProgram(RhaiProgram),
    Neg(Float),
    Abs(Float),
    Sum(Float, Float),
    Sub(Float, Float),
    Mul(Float, Float),
    Div(Float, Float),
    Max(Float, Float),
    Min(Float, Float),
    Mod(Float, Float),
    Sin(Float),
    Cos(Float),
    Exp(Float),
    Ln(Float),
    Lerp {
        a: Float,
        b: Float,
        t: Float,
    },
    ExpLerp {
        a: Float,
        b: Float,
        t: Float,
    },
    Easing(enum Easing {
        Linear,
        In,
        Out,
        InOut,
        InOutFast,
        ElasticOut,
    }, Float),
    LaterStart {
        t: Float,
        start: Float,
    },
    EarlyFinish {
        t: Float,
        time: Float,
    },
    VecLength(Float2),
    Vec2X(Float2),
    Vec2Y(Float2),
    Atan2(Float2, Float2),
    Dot(Float2, Float2),
    Vec3Length(Float3),
    Vec3X(Float3),
    Vec3Y(Float3),
    Vec3Z(Float3),
    Dot3(Float3, Float3),
    Vec4Length(Float4),
    Vec4X(Float4),
    Vec4Y(Float4),
    Vec4Z(Float4),
    Vec4W(Float4),
    Dot4(Float4, Float4),
    MouseZoom(MouseState),
    If {
        condition: Bool,
        then: Float,
        otherwise: Float,
    },
}

Documentation

64-bit floating-point value.

Float2

RON schema

enum Float2 {
    Simple(f64, f64),
    Zeros,
    Usize(usize, usize),
    Floats(Float, Float),
    Neg(Float2),
    Sum(Float2, Float2),
    Sub(Float2, Float2),
    Mul(Float2, Float),
    TransformPosition(Matrix2, Float2),
    ProjectPosition(Matrix2, Float2),
    TransformDirection(Matrix2, Float2),
    FixedPoint(Matrix2),
    NormalizeDirection(Float2),
    SolveQuadratic {
        a: Float,
        b: Float,
        c: Float,
    },
    FromProgram(RhaiProgram),
    Lerp {
        a: Float2,
        b: Float2,
        t: Float,
    },
    LerpPoints {
        points: Vec<Float2>,
        t: Float,
    },
    ScreenSize(()),
    MousePosition(MouseState),
    MouseDelta(MouseState),
    MouseZoomPosition(MouseState),
    If {
        condition: Bool,
        then: Float2,
        otherwise: Float2,
    },
}

Documentation

2D vector with 64-bit floating-point coordinates (x, y).

Float3

RON schema

enum Float3 {
    Simple(f64, f64, f64),
    Zeros,
    Floats(Float, Float, Float),
    Neg(Float3),
    Sum(Float3, Float3),
    Sub(Float3, Float3),
    Mul(Float3, Float),
    Cross(Float3, Float3),
    TransformPosition(Matrix3, Float3),
    ProjectPosition(Matrix3, Float3),
    TransformDirection(Matrix3, Float3),
    NormalizeDirection(Float3),
    FromProgram(RhaiProgram),
    Lerp {
        a: Float3,
        b: Float3,
        t: Float,
    },
    LerpPoints {
        points: Vec<Float3>,
        t: Float,
    },
    If {
        condition: Bool,
        then: Float3,
        otherwise: Float3,
    },
}

Documentation

3D vector with 64-bit floating-point coordinates (x, y, z).

Float4

RON schema

enum Float4 {
    Simple(f64, f64, f64, f64),
    Zeros,
    Floats(Float, Float, Float, Float),
    Neg(Float4),
    Sum(Float4, Float4),
    Sub(Float4, Float4),
    Mul(Float4, Float),
    TransformPosition(Matrix4, Float4),
    ProjectPosition(Matrix4, Float4),
    TransformDirection(Matrix4, Float4),
    NormalizeDirection(Float4),
    FromProgram(RhaiProgram),
    Lerp {
        a: Float4,
        b: Float4,
        t: Float,
    },
    LerpPoints {
        points: Vec<Float4>,
        t: Float,
    },
    If {
        condition: Bool,
        then: Float4,
        otherwise: Float4,
    },
}

Documentation

4D vector with 64-bit floating-point coordinates (x, y, z, w).

Matrix2

RON schema

enum Matrix2 {
    Simple {
        offset: (f64, f64),
        rotation: f64,
        scale: f64,
        mirror_x: bool,
    },
    Parametric {
        offset: Float2,
        rotation: Float,
        scale: Float,
        mirror_x: Bool,
    },
    Inverse(Matrix2),
    Mul(Matrix2, Matrix2),
    Teleport {
        from: Matrix2,
        to: Matrix2,
        what: Matrix2,
    },
    Lerp {
        a: Matrix2,
        b: Matrix2,
        t: Float,
    },
    Exact {
        m00: Float,
        m01: Float,
        m02: Float,
        m10: Float,
        m11: Float,
        m12: Float,
        m20: Float,
        m21: Float,
        m22: Float,
    },
    FromCamera(Camera),
    FromProgram(RhaiProgram),
    Identity,
    If {
        condition: Bool,
        then: Matrix2,
        otherwise: Matrix2,
    },
}

Documentation

2D transformation stored as a 3×3 homogeneous matrix. Its upper-left 2×2 part represents linear transformations such as rotation, reflection, scale, and shear, while the third column adds translation. A matrix whose last row is (0, 0, 1) is called affine: it can combine translation with linear transformations, but does not contain perspective or projection. To transform a 2D vector, represent a position as (x, y, 1) or a direction as (x, y, 0); Float2::TransformPosition and Float2::TransformDirection do this for you.

Matrix3

RON schema

enum Matrix3 {
    Simple {
        offset: (f64, f64, f64),
        rotate: (f64, f64, f64),
        scale: f64,
        mirror: (bool, bool, bool),
    },
    Parametric {
        offset: Float3,
        rotate: Float3,
        mirror: Float3,
        scale: Float,
    },
    Translate(Float3),
    Rotate {
        axis: enum Matrix3RotateAxis {
            X,
            Y,
            Z,
        },
        angle: Float,
    },
    OrbitFrame3D {
        radius: Float,
        azimuth: Float,
        elevation: Float,
        roll: Float,
    },
    PerspectiveY {
        fov_y: Float,
        near: Float,
        far: Float,
    },
    FromCamera3D(Camera3D),
    FromMatrix2(Matrix2),
    Inverse(Matrix3),
    Mul(Matrix3, Matrix3),
    Teleport {
        from: Matrix3,
        to: Matrix3,
        what: Matrix3,
    },
    Lerp {
        a: Matrix3,
        b: Matrix3,
        t: Float,
    },
    Exact {
        m00: Float,
        m01: Float,
        m02: Float,
        m03: Float,
        m10: Float,
        m11: Float,
        m12: Float,
        m13: Float,
        m20: Float,
        m21: Float,
        m22: Float,
        m23: Float,
        m30: Float,
        m31: Float,
        m32: Float,
        m33: Float,
    },
    Identity,
    FromProgram(RhaiProgram),
    If {
        condition: Bool,
        then: Matrix3,
        otherwise: Matrix3,
    },
}

Documentation

3D transformation stored as a 4×4 homogeneous matrix. Its upper-left 3×3 part represents linear transformations such as rotation, reflection, scale, and shear, while the fourth column adds translation. A matrix whose last row is (0, 0, 0, 1) is called affine: it can combine translation with linear transformations, but does not contain perspective or projection. Positions are represented as (x, y, z, 1) and directions as (x, y, z, 0); Float3::TransformPosition, Float3::TransformDirection, and Float3::ProjectPosition perform the corresponding operations.

In GLSL, a bound Matrix3 is a mat4. Transform a position without projection as (matrix * vec4(position, 1.0)).xyz; when the matrix contains projection, divide that XYZ result by the resulting W. Transform a direction as (matrix * vec4(direction, 0.0)).xyz only when the matrix has no projection.

Matrix4

RON schema

enum Matrix4 {
    Simple {
        offset: (f64, f64, f64, f64),
        rotate_xy: f64,
        rotate_xz: f64,
        rotate_xw: f64,
        rotate_yz: f64,
        rotate_yw: f64,
        rotate_zw: f64,
        scale: f64,
        mirror: (bool, bool, bool, bool),
    },
    Parametric {
        offset: Float4,
        rotate_xy: Float,
        rotate_xz: Float,
        rotate_xw: Float,
        rotate_yz: Float,
        rotate_yw: Float,
        rotate_zw: Float,
        mirror: Float4,
        scale: Float,
    },
    Rotate {
        plane: enum Matrix4RotatePlane {
            XY,
            XZ,
            XW,
            YZ,
            YW,
            ZW,
        },
        angle: Float,
    },
    PerspectiveY {
        fov_y: Float,
        aspect_x: Float,
        aspect_z: Float,
        near: Float,
        far: Float,
    },
    OrbitFrame4D {
        radius: Float,
        azimuth: Float,
        latitude: Float,
        elevation: Float,
        horizontal_roll: Float,
        vertical_roll_x: Float,
        vertical_roll_z: Float,
    },
    FromMatrix3(Matrix3),
    Inverse(Matrix4),
    Mul(Matrix4, Matrix4),
    Teleport {
        from: Matrix4,
        to: Matrix4,
        what: Matrix4,
    },
    Lerp {
        a: Matrix4,
        b: Matrix4,
        t: Float,
    },
    Exact {
        m00: Float,
        m01: Float,
        m02: Float,
        m03: Float,
        m04: Float,
        m10: Float,
        m11: Float,
        m12: Float,
        m13: Float,
        m14: Float,
        m20: Float,
        m21: Float,
        m22: Float,
        m23: Float,
        m24: Float,
        m30: Float,
        m31: Float,
        m32: Float,
        m33: Float,
        m34: Float,
        m40: Float,
        m41: Float,
        m42: Float,
        m43: Float,
        m44: Float,
    },
    FromProgram(RhaiProgram),
    Identity,
    If {
        condition: Bool,
        then: Matrix4,
        otherwise: Matrix4,
    },
}

Documentation

4D transformation stored as a 5×5 homogeneous matrix. Its upper-left 4×4 part represents linear transformations such as rotation, reflection, scale, and shear, while the fifth column adds translation. A matrix whose last row is (0, 0, 0, 0, 1) is called affine: it can combine translation with linear transformations, but does not contain perspective or projection. Positions are represented as (x, y, z, w, 1) and directions as (x, y, z, w, 0); Float4::TransformPosition, Float4::TransformDirection, and Float4::ProjectPosition perform the corresponding operations.

GLSL has no 5×5 matrix type, so a Matrix4 binding named foo generates foo_project_point(position) for the usual position transform and foo_project(position) when the homogeneous divisor is also needed. Use foo_project_direction(direction) when there is no projection, or foo_project_direction(position, direction) when projection makes the transformed direction depend on its position.

Color

RON schema

enum Color {
    Simple(u8, u8, u8, u8),
    Rainbow(Float),
    Viridis(Float),
    Blend(Color, Color),
    AddAlpha(Color, Float),
    Parametric {
        r: Float,
        g: Float,
        b: Float,
        a: Float,
    },
    Gradient(Color, Color, Float),
    FromProgram(RhaiProgram),
    If {
        condition: Bool,
        then: Color,
        otherwise: Color,
    },
}

Documentation

Color with 8-bit red, green, blue, and alpha channels.

Interface

RON schema

enum Interface {
    Empty,
    Label(string),
    RedLabel(string),
    Text(Text),
    Named(string, OnlyNamed),
    GetErrors(OnlyNamed),
    Horizontal(Vec<Interface>),
    Vertical(Vec<Interface>),
    HorizontalSeparator,
    VerticalSeparator,
    CollapsingHeader(string, Interface),
    If {
        condition: Bool,
        then: Interface,
        otherwise: Interface,
    },
}

Documentation

Defines the user interface shown in the Control scene window. The app renders the top-level Interface component named interface; other Interface components are shown only when referenced from this tree.

Named inserts another component’s compact user-interface control. The referenced component decides what control it displays; calculated variants generally report that they do not support the user interface.

Collection

RON schema

struct Collection {
    elems: Vec<enum CollectionElem {
        Named(OnlyNamed),
        If {
            condition: Bool,
            then: OnlyNamed,
            otherwise: OnlyNamed,
        },
    }>,
}

Documentation

An ordered group of existing named components. A collection does not convert its components or require them to have one type; the component that consumes the collection decides which types it accepts.

A Collection stores only references to each component’s cached result; it does not copy the results. Creating a collection is therefore cheap even when its components produce very large objects, although a component consuming the collection may choose to copy them later.

Nested Collection components are flattened recursively at their position in the list. The nested collection’s own name is removed, while its items keep their names and order. Duplicate items are otherwise preserved.

Element kinds

One entry in a Collection.

Text

RON schema

enum Text {
    Simple(string),
    SimpleMultiline(string),
    TextCode {
        language: enum TextCodeLanguage {
            PlainText,
            Glsl,
            Jinja,
            Rhai,
        },
        text: string,
    },
    TextWithVariables {
        variables: Vec<(string, OnlyNamed)>,
        text: Text,
    },
    RawFromVariables(Text),
    FromOnlyNamed(OnlyNamed),
    Add(Vec<Text>),
    FromBool(Bool),
    FromUsize(Usize),
    FromFloat(Float),
    FromFloatFormatted {
        value: Float,
        decimals: Usize,
        scientific: Bool,
    },
    FromProgram(RhaiProgram),
    FromMinijinja(MiniJinja),
    If {
        condition: Bool,
        then: Text,
        otherwise: Text,
    },
}

Documentation

Text value that can be edited, generated, selected, or assembled from multiple parts.

Besides its visible contents, Text stores which component produced each concatenated part. Components that compile or parse the result can use this provenance to report errors at the exact field that produced them.

Text can also carry named component variables. When several parts are combined with Add, their variables are unified, so a shader assembled from reusable parts can give each part its own bindings. GlslShader exposes these variables as uniforms, while RhaiProgram and MiniJinja provide them as inputs.

Generated text cannot preserve the provenance of the text used to generate it or attach variables to newly generated parts. RhaiProgram output loses both provenance and variables. MiniJinja also loses provenance and consumes its input bindings, although it forwards variables already carried by inserted Text or MiniJinja values. There is currently no way around these limitations.

MouseState

RON schema

enum MouseState {
    Simple(()),
    SwapButtons(MouseState),
    ZoomButton {
        left_button: bool,
        middle_button: bool,
        right_button: bool,
        other: MouseState,
    },
    If {
        condition: Bool,
        then: MouseState,
        otherwise: MouseState,
    },
}

Documentation

Mouse input over the rendered canvas. The top-level component named mouse_state is special: the app overwrites it every frame with the current input as Simple. Runtime input is not saved in the scene, so Simple is serialized as Simple(()).

Positions and deltas use centered screen space: (0, 0) is the screen center, the shortest side spans [-1, 1], X points right, and Y points down. Other MouseState components can route or modify the top-level input before passing it to cameras, shaders, or other interactive components.

Simple

Simple holds the cursor position, currently held buttons, movement since the previous frame in dpos, buttons held during that movement in drag_buttons, the scroll or pinch change in dzoom, and the cursor position around which that zoom occurred in dzoom_position. position and buttons are available while hovering over or dragging the canvas. dpos and drag_buttons are nonzero only while dragging. Positive dzoom zooms in, negative zooms out, and dzoom_position is (0, 0) when there is no zoom.

ScreenPoints

RON schema

struct ScreenPoints {
    camera: Camera,
    array: Collection,
}

Documentation

Allows you to change coordinates of Float2::Simple and Matrix2::Simple inside every component listed in array. Displays points coordinates according to camera.

RhaiProgram

RON schema

struct RhaiProgram {
    program: Text,
}

Documentation

Evaluates a Rhai script and returns its final expression. See the Rhai language reference for syntax and built-in operations.

The default program is Text::TextWithVariables: give each component input a name, then use that name as a variable in the script. For example, bind position to a Float2, then write:

let offset = vec2(1.0, 2.0);
position + offset

print(...) and debug(...) output is captured and shown in the component’s Value section.

Inputs

Component types are represented as follows:

Result

A RhaiProgram has no fixed result type. Its final expression can be passed unchanged into another RhaiProgram, or read by a component’s FromProgram variant; each such variant documents the type it expects. For example, Float::FromProgram expects a floating-point value, while Float2::FromProgram accepts either DVec2 or [x, y].

When a string result is converted to Text with Text::FromProgram, it has no attached variables and cannot preserve the provenance of the script parts that generated it. There is currently no way for a Rhai script to return that metadata.

Vectors

DVec2, DVec3, and DVec4 provide .x, .y, .z, and .w as applicable, numeric indexing such as v[0], and .to_array(). Construct them with vec2(x, y), vec3(x, y, z), and vec4(x, y, z, w).

Vector operations include:

Matrices

DMat3, DMat4, and DMat5 provide matrix multiplication and .to_array(), which returns nested row-major arrays. Build a matrix from the same array form with mat3(rows), mat4(rows), or mat5(rows).

Common methods are:

Constructor angles use radians unless their name ends in _deg:

Every scale parameter above accepts either one float or a vector of the corresponding dimension.

CpuFramebuffer

A CpuFramebuffer provides .width, .height, .format, and .channels. .pixel(x, y) returns its one or four channel values, with (0, 0) at the bottom-left. Rgba8 and Rgba32u channels are integers; other formats return floats. Reading a CpuFramebuffer requires copying the image from the GPU, so see that component’s documentation before using it for large or frequently changing images.

MiniJinja

RON schema

struct MiniJinja {
    template: Text,
}

Documentation

Renders a Text template. Use {{ expression }} to insert a value and {% ... %} for Jinja statements such as conditions and loops. See the MiniJinja template syntax for the complete language reference. Missing variables are errors instead of silently becoming empty text.

Inputs

The default template is Text::TextWithVariables: give each component input a name, then use that name in the template. For example, bind radius to a Float component named circle_radius, then write:

const float radius = {{ radius|fmt_float(3) }};

Component types are represented as follows:

Matrices use column-major indexing: matrix[column][row]. Nested collections are flattened by Collection; access a key that is not a simple identifier with brackets, for example values["group.radius"]. To iterate through every entry, use {% for name, value in values|items %}...{% endfor %}.

Output variables and errors

Rendering creates new text, so its original provenance is lost: an error in generated GLSL or Rhai code points to the MiniJinja component rather than the exact template part that produced it. Input bindings are consumed and do not become variables on the output. However, variables already carried by inserted Text or MiniJinja values are forwarded, including values nested in a Collection. There is currently no way to preserve provenance or create new output variable bindings from the template.

Filters

All built-in filters enabled by the app are described in the MiniJinja filter documentation.

Custom filters

fmt_float(decimals?) formats a number with at most 16 decimal places and removes trailing zeroes; it also turns rounded -0 into 0. pad_left(width) and pad_right(width) add spaces until the result reaches width. Filters can be chained: {{ value|fmt_float(1)|pad_left(4) }} renders 1.12 as ` 1.1`.

Time

RON schema

struct Time {
    elapsed: {
        secs: u64,
        nanos: u32,
    },
    started: bool,
    speed_ms: f64,
    count_to: Option<Float>,
}

Documentation

A controllable simulation clock that outputs time in seconds.

Time does not measure real elapsed time. While started is on in the interactive editor, it adds speed_ms milliseconds once per rendered frame and requests another frame. This fixed timestep makes simulations reproducible frame by frame; speed_ms = 16 advances the clock by 0.016 seconds per frame. Use the play, pause, stop, reset, Step, and Step/10 controls to run or inspect a simulation manually.

Without count_to, the output is the full elapsed duration. With count_to, the output wraps into 0 <= time < count_to, while elapsed continues increasing in the background. The editor also shows a slider for scrubbing within this range; moving it replaces elapsed with the selected time. A zero or negative count_to does not wrap.

For animations, count_to is usually Named("render_to_time"). This makes the time component loop over the animation’s duration and provides a convenient slider for watching and scrubbing the animation in the editor.

A scene normally has one top-level Time named time, which the app itself updates; you will almost never need another one.

GlslShader

RON schema

struct GlslShader {
    always_recompile: Bool,
    recompile: Bool,
    time: Option<Time>,
    outputs: Usize,
    glsl_text: Text,
}

Documentation

A fullscreen fragment shader. This component only compiles the GLSL program and holds its uniform values; it draws nothing by itself. To draw the picture, some other component must draw it:

The drawing component also chooses the resolution.

Shader source

Keep glsl_text in its default shape: an inline TextWithVariables holding the uniform bindings, with an inline TextCode (language Glsl) holding the code. Write only the body of the fragment shader: helper functions and void main(). Everything around it is generated: #version (writing your own is an error), precision statements, uniform declarations, and output declarations. The smallest shader:

void main() {
    out_color = vec4(v_uv, 0.0, 1.0);
}

Names available in every shader (all coordinates use the lower-left origin):

Since glsl_text is an ordinary Text component, the code does not have to be one literal: assemble it with Text::Add, switch versions with Text::If, or generate it with MiniJinja or RhaiProgram.

Screen and world coordinates

Pixel coordinates use a lower-left origin. Camera screen space is centered at (0, 0), its shortest side spans [-1, 1], and Y points downward. These helpers convert between pixels, screen space, and world space:

vec2 pixel_to_screen(vec2 pixel_pos, vec2 resolution) {
    float scale = 0.5 * min(resolution.x, resolution.y);
    return (pixel_pos - 0.5 * resolution) / vec2(scale, -scale);
}

vec2 screen_to_pixel(vec2 screen_pos, vec2 resolution) {
    float scale = 0.5 * min(resolution.x, resolution.y);
    return screen_pos * vec2(scale, -scale) + 0.5 * resolution;
}

vec2 pixel_to_world(vec2 pixel_pos, vec2 resolution, mat3 screen_to_world) {
    vec2 screen_pos = pixel_to_screen(pixel_pos, resolution);
    return (screen_to_world * vec3(screen_pos, 1.0)).xy;
}

vec2 world_to_pixel(vec2 world_pos, vec2 resolution, mat3 world_to_screen) {
    vec2 screen_pos = (world_to_screen * vec3(world_pos, 1.0)).xy;
    return screen_to_pixel(screen_pos, resolution);
}

Bind Matrix2::FromCamera as screen_to_world; bind its Matrix2::Inverse as world_to_screen. For the current fragment, pass current_pixel_pos and current_resolution.

For example, this draws a world-space grid using a Matrix2::FromCamera binding named screen_to_world:

void main() {
    vec2 world_pos = pixel_to_world(
        current_pixel_pos,
        current_resolution,
        screen_to_world
    );
    vec2 cell = fract(world_pos);
    float grid = max(1.0 - step(0.03, cell.x), 1.0 - step(0.03, cell.y));
    out_color = vec4(vec3(grid), 1.0);
}

When the source Camera allows changes and receives mouse input, dragging or zooming it moves the view over the grid.

3D projective camera

In 3D, a pixel corresponds to a viewing ray rather than one world-space position. Build and bind these Matrix3 components:

These helpers convert pixels to rays and world-space positions back to pixels:

vec3 project_position(mat4 matrix, vec3 position) {
    vec4 transformed = matrix * vec4(position, 1.0);
    return transformed.xyz / transformed.w;
}

vec2 pixel_to_clip(vec2 pixel_pos, vec2 resolution) {
    return (2.0 * pixel_pos - resolution) / resolution.y;
}

vec2 clip_to_pixel(vec2 clip_pos, vec2 resolution) {
    return 0.5 * (clip_pos * resolution.y + resolution);
}

vec3 pixel_to_world(
    vec2 pixel_pos,
    float clip_z,
    vec2 resolution,
    mat4 world_from_clip
) {
    vec2 clip_pos = pixel_to_clip(pixel_pos, resolution);
    return project_position(world_from_clip, vec3(clip_pos, clip_z));
}

void pixel_to_world_ray(
    vec2 pixel_pos,
    vec2 resolution,
    mat4 world_from_camera,
    mat4 world_from_clip,
    out vec3 ray_origin,
    out vec3 ray_direction
) {
    ray_origin = (world_from_camera * vec4(0.0, 0.0, 0.0, 1.0)).xyz;
    vec3 far_position = pixel_to_world(pixel_pos, 1.0, resolution, world_from_clip);
    ray_direction = normalize(far_position - ray_origin);
}

vec2 world_to_pixel(
    vec3 world_pos,
    vec2 resolution,
    mat4 clip_from_world
) {
    vec3 clip_pos = project_position(clip_from_world, world_pos);
    return clip_to_pixel(clip_pos.xy, resolution);
}

For PerspectiveY, clip_z = -1 is the near plane and clip_z = 1 is the far plane. A primary ray uses the far plane:

void main() {
    vec3 ray_origin;
    vec3 ray_direction;
    pixel_to_world_ray(
        current_pixel_pos,
        current_resolution,
        world_from_camera,
        world_from_clip,
        ray_origin,
        ray_direction
    );

    out_color = vec4(0.5 + 0.5 * ray_direction, 1.0);
}

When the source Camera3D allows changes and receives mouse input, dragging orbits the camera and zooming changes its distance. Unlike the 2D helpers, these helpers use the framebuffer height and do not flip Y because PerspectiveY defines a vertical field of view in Y-up coordinates.

Uniform bindings

To use other components inside the shader - numbers, vectors, matrices, colors, framebuffers, the mouse - add variables to glsl_text (TextWithVariables). Each binding is a pair (uniform name, component name). Uniform names must be valid GLSL identifiers, unique, and must not collide with the built-in names.

The bindings are attached as metadata to whatever text is inside, and they are real component references that participate in dependency tracking - not names parsed out of the shader source. The uniform declaration is generated from the component’s type automatically. When a bound component changes, the shader gets the new value without recompilation; recompilation happens only when the source text changes.

Component types map to GLSL as follows:

Framebuffer uniforms

A Framebuffer binding foo also generates foo_logical_resolution: vec2. The texture is often allocated larger than the image, with the image in its lower-left corner, so read it through the built-in helper, which accounts for that:

vec4 c = sample_framebuffer(foo, foo_logical_resolution, v_uv);

For usampler2D it returns uvec4. Raw texture(...), texelFetch(...) and textureSize(...) see the whole allocated texture; if you need them, apply foo_logical_resolution yourself.

A Framebuffers binding foo generates one sampler and one logical resolution per output: foo0, foo0_logical_resolution, foo1, … This samples the outputs directly, with no copy. Framebuffer::FromFramebuffers is only needed when one output must exist as a standalone Framebuffer.

Matrix4 uniforms

A Matrix4 binding foo generates the raw uniforms foo_linear: mat4, foo_translation: vec4, foo_h_row: vec4, foo_h_bias: float (the 5x5 homogeneous matrix split into blocks) and helpers:

MouseState uniforms

A MouseState binding foo generates the uniforms foo_position: vec2, foo_dpos: vec2, foo_dzoom: float, foo_dzoom_position: vec2, foo_buttons: int, foo_drag_buttons: int. Values come straight from the MouseState component; button masks are 1 for left, 2 for middle, and 4 for right. It also generates boolean helpers foo_left_button(), foo_middle_button(), foo_right_button(), plus the same three as foo_*_drag_button().

Recompilation

With always_recompile: true (the default), source changes compile immediately. Convenient while writing. With false, text edits leave the last successfully compiled program running until recompile fires. This is useful for shaders that compile slowly, or for editing several text pieces before applying them together. While the running program is older than the source, the Value section shows Needs recompilation: true.

recompile is the one-shot apply trigger. Bool::OnceButton fits best: it turns on for one recalculation and resets itself.

If compilation fails, errors are reported in your line numbers, and each error is also attached to the exact Text component that authored the failing line.

Time

When time is set, the shader gets two float uniforms: _time, the time value in seconds, and _dt, seconds since this shader’s previous calculation. _dt is 0 on the first frame and when time jumps backward. Time is an ordinary dependency, so the shader recalculates and redraws every frame while it is running. Leave it None for static shaders.

Multiple outputs

outputs sets the number of fragment color outputs and must be at least 1. The first output is always named out_color0, with out_color as an alias, so naming stays the same in single- and multi-output shaders.

The default 1 is for shaders drawn by Framebuffer::DrawShader. With N > 1, the shader additionally writes out_color1, …, out_colorN-1 and must be drawn by a Framebuffers component, which renders all outputs in one pass into a multi-target framebuffer. Read one output back as a normal framebuffer with Framebuffer::FromFramebuffers, or bind the whole Framebuffers component as a shader uniform. Do not share a multi-output shader with Framebuffer::DrawShader: it captures only output 0.

Portability and feedback

The program is compiled as GLSL 330 on desktop and GLSL ES 3.00 on web and Android, so stick to their common subset if the scene must run everywhere.

A shader can never read the framebuffer it is currently drawing into. To read your own previous frame (simulations, cellular automata, path tracing), use the @prev + Framebuffer::CopyAnother scheme from the general documentation; game_of_life_shader is a working example.

DrawnText

RON schema

struct DrawnText {
    position: Float2,
    text: Text,
    font_size: Float,
    color: Color,
    align: enum TextAlign {
        Left,
        Center,
        Right,
    },
    valign: enum TextVAlign {
        Top,
        Center,
        Bottom,
    },
    font: enum TextFont {
        Proportional,
        Monospace,
    },
    draw_background: Bool,
    background: Color,
    background_padding: Float,
}

Documentation

Draws a text label anchored at a position in world space. It is intended to be included in the draw collection of Framebuffer::RenderDrawable.

font_size and background_padding are measured in world units, so the label scales with camera zoom. align and valign choose which point of the label is placed at position.

Framebuffer

RON schema

enum Framebuffer {
    CopyAnother {
        source: Framebuffer,
        initial_size: Option<Float2>,
    },
    RenderDrawable {
        screen_size: Float2,
        draw: Collection,
        camera: Camera,
        margin: Float,
        initial_size: Option<Float2>,
        format: enum TextureFormat {
            Rgba8,
            Rgba16f,
            Rgba32f,
            Rgba32u,
            R32f,
        },
    },
    DrawShader {
        screen_size: Float2,
        shader: GlslShader,
        redraw_on_changes: Bool,
        initial_size: Option<Float2>,
        vertical_strips: Option<Usize>,
        format: enum TextureFormat {
            Rgba8,
            Rgba16f,
            Rgba32f,
            Rgba32u,
            R32f,
        },
    },
    FromFramebuffers {
        source: Framebuffers,
        index: Usize,
        initial_size: Option<Float2>,
    },
    FromImage {
        uri: Text,
        reload: Bool,
    },
    Random {
        size: Usize,
        seed: Usize,
    },
    FromProgram(RhaiProgram),
}

Documentation

A 2D image stored in a GPU texture. The app displays the top-level Framebuffer named final_image; other framebuffers can hold intermediate images and be bound as inputs to a GlslShader for multi-pass rendering.

Variants with initial_size retain a GPU texture that may be larger than the logical image. The allocation starts at least this large, never shrinks, and grows in 50-pixel steps, avoiding reallocations when the image size changes. In GLSL, use sample_framebuffer with the generated logical resolution so sampling ignores the unused allocated area.

CopyAnother

Copies source into current framebuffer.

For single-pass shader feedback, source can read @prev(target, init, clock) while target samples this copy, because a shader cannot sample the framebuffer it is currently drawing into.

RenderDrawable

Renders drawable objects into a framebuffer. A drawable object is a component whose calculated result can paint into a 2D framebuffer. Objects from draw are painted in collection order through camera; nested collections are flattened.

Before painting, the framebuffer is cleared to transparent black (0, 0, 0, 0). The special background_color is not used here - it is applied later, when the app shows final_image on the canvas.

Drawable components:

A raw Framebuffer in draw is a fullscreen drawable: it ignores the camera, fills the whole destination, must have exactly the same logical size as the destination (otherwise it is an error), and is composited over earlier objects using its alpha. To place an image in world space through the camera instead, use DrawFramebuffer.

DrawShader

Draws a fullscreen GlslShader directly into this framebuffer.

FromFramebuffers

Reads one output of a multi-output Framebuffers component and exposes it as a regular Framebuffer by copying the selected attachment.

FromImage

Loads a local image path at runtime and keeps the uploaded framebuffer around until you reload or change the URI.

Random

Generates a deterministic square R32f framebuffer filled with random values.

Framebuffers

RON schema

enum Framebuffers {
    DrawShader {
        screen_size: Float2,
        shader: GlslShader,
        redraw_on_changes: Bool,
        initial_size: Option<Float2>,
        vertical_strips: Option<Usize>,
        format: enum TextureFormat {
            Rgba8,
            Rgba16f,
            Rgba32f,
            Rgba32u,
            R32f,
        },
    },
    CopyAnother {
        source: Framebuffers,
        initial_size: Option<Float2>,
    },
}

Documentation

Several same-sized 2D images stored in GPU textures with the same format. They let one GlslShader produce related images in a single pass: set its outputs and write to out_color0, out_color1, …

Bind a Framebuffers component to another GlslShader as foo to sample its outputs as foo0, foo1, … Use Framebuffer::FromFramebuffers when one output is needed as a regular Framebuffer.

CpuFramebuffer

RON schema

struct CpuFramebuffer {
    source: Framebuffer,
}

Documentation

Pixels of a Framebuffer copied from GPU to CPU memory. This makes the CPU wait for the GPU, so use it only when a CPU-side component such as RhaiProgram needs rendered pixels. Only the logical image is copied; unused retained texture area is ignored.

In Rhai, a CpuFramebuffer has .width, .height, .format, and .channels; .pixel(x, y) returns its one or four channel values. Coordinates start at the bottom-left, as in GLSL. Rgba8 and Rgba32u channels are integers; other formats return floats.

DrawFramebuffer

RON schema

struct DrawFramebuffer {
    framebuffer: Framebuffer,
    matrix: Matrix2,
}

Documentation

Draws a Framebuffer as a transformed rectangle in a world-space scene. It is intended to be included in the draw collection of Framebuffer::RenderDrawable.

Before applying matrix, the image spans from -1 to 1 on X and is centered on Y, with its height chosen from the framebuffer’s aspect ratio. Transparent pixels blend over objects drawn earlier. The source and destination framebuffers must be different.

Camera

RON schema

enum Camera {
    Simple {
        position: (f64, f64),
        scale: f64,
        allow_change: bool,
        mouse_state: MouseState,
    },
    Parametric {
        position: Float2,
        scale: Float,
    },
    Interpolation(Camera, Camera, Float),
    MultiInterpolation {
        mode: enum CameraPathInterpolation {
            SegmentInOut,
            SmartSpline,
        },
        cameras: Vec<{
            position: f64,
            camera: Camera,
        }>,
        time: Float,
    },
    If {
        condition: Bool,
        then: Camera,
        otherwise: Camera,
    },
}

Documentation

A 2D camera that chooses the visible part of the world. position is the world point at the center of the screen. scale is half the visible world-space span along the shortest screen side, so a smaller value zooms in; Y points downward.

Use it in Framebuffer::RenderDrawable to draw a world-space scene, or use Matrix2::FromCamera to convert centered screen-space positions into world space.

Camera3D

RON schema

enum Camera3D {
    Simple {
        alpha: f64,
        beta: f64,
        distance: f64,
        allow_change: bool,
        mouse_state: MouseState,
    },
    Parametric {
        alpha: Float,
        beta: Float,
        distance: Float,
    },
    Interpolation(Camera3D, Camera3D, Float),
    MultiInterpolation {
        mode: enum CameraPathInterpolation {
            SegmentInOut,
            SmartSpline,
        },
        cameras: Vec<{
            position: f64,
            camera: Camera3D,
        }>,
        time: Float,
    },
    If {
        condition: Bool,
        then: Camera3D,
        otherwise: Camera3D,
    },
}

Documentation

A 3D orbit camera in a Y-up world. It looks at the origin from distance: with alpha = 0 and beta = 0, the camera is on the positive X axis; alpha moves it around the Y axis, and beta moves it above or below the horizontal plane. Angles are in radians, and beta should stay strictly between -pi/2 and pi/2.

It describes only the camera’s position and orientation, without a projection lens. Matrix3::FromCamera3D converts it to a camera-to-world frame; combine that with Matrix3::PerspectiveY as described in their documentation to project 3D positions.

Arrow

RON schema

struct Arrow {
    matrix: Matrix2,
    arrow_scale: Float,
    arrow_color: Color,
    stroke_thickness: Float,
    stroke_color: Color,
    centered: Bool,
    enabled: Bool,
}

Documentation

Draws a filled, optionally outlined arrow pointing along its local +X axis. It is intended to be included in the draw collection of Framebuffer::RenderDrawable.

RememberClonable

RON schema

struct RememberClonable {
    enable_remembering: Bool,
    clear_cache: Bool,
    to_remember: Collection,
}

Documentation

Accumulates snapshots of selected components, preserving earlier drawable states to create trails or show a history. It is intended to be included in the draw collection of Framebuffer::RenderDrawable; stored snapshots are drawn in capture order.

On the first evaluation or when to_remember changes, every current object is captured. Afterwards, while enable_remembering is true, a new snapshot is appended whenever an object changes. Existing snapshots remain until clear_cache is activated, and objects that cannot be copied are skipped.

Components that can be captured: