SIPL, in full.

SIPL Language Reference#

This is the complete language on one page. An AI model can use it as a system prompt.

1. The syntax decision: braces for structure, line breaks for statements#

The brace-versus-indentation argument comes down to two concerns:

• Brace programmers want structure that stays correct after copy-paste, reformatting, or token-by-token generation. Whitespace must never change meaning.

• Indentation programmers want clean code without semicolons and clutter, and they want indentation to always match the structure.

SIPL meets both needs:

The line-break rule follows Go: a line break ends a statement only when the line ends with a name, literal, ), ], }, return, break, or continue. So expressions can continue across lines after an operator, comma, or opening bracket. A line that starts with . continues the previous line:

let result = numbers
    .filter(fn(n) => n > 0)
    .map(fn(n) => n * 2)

2. Program structure#

// Comments start with //

let MAX_USERS = 100          // top level: only use, fn, type, enum, let, var

fn start() {                 // execution begins here
    print("ready")
}

Top-level functions can be declared in any order. Named test "description" { ... } blocks are available to sipl test in test sources; production run and build commands reject them. See Testing and API documentation.

3. Variables and types#

let name = "Ada"             // constant binding, can never be reassigned
var count = 0                // variable
count += 1                   // also -=, *=, /=
let ratio: Float = 0.5       // optional annotation, checked

let fixes the name. Lists and Maps that the name points to can still be changed with their mutating methods.

Annotations are enforced by sipl check whenever the source type is known. This includes initializers, assignments, function and method arguments, constructor fields, and return values. A function with a non-None return annotation must return on every structural path. Any is an explicit dynamic boundary: values flowing out of it retain the same runtime checks, including the permitted Int-to-Float conversion.

Annotation types: Any None Bool Int Float Number String List Map Range Function Type Fail Result Ai Bytes Regex Prng Instant Monotonic Duration Date LocalDateTime TimeZone Interval Cron Locale Translator, parameterized List[T], Map[K, V], Result[T], or the name of any user type. Generic types may nest, for example List[Map[String, Int]]. An Int is accepted where a scalar Float is expected and is converted automatically.

Generic collections and results#

fn total(values: List[Int]) -> Int {
    var sum = 0
    for value in values {
        sum += value             // value is known to be Int
    }
    return sum
}

fn start() {
    let scores: Map[String, Int] = {"Ada": 10}
    scores.set("Bob", 20)
    let names: List[String] = scores.keys()

    let answer: Result[Int] = result.ok(total([1, 2, 3]))
    print(names, answer.unwrap())
}

• List[T], Map[K, V], and Result[T] are invariant. List[Int] is not a List[Float]; this prevents mutation through an alias from violating the element type.

• Plain List, Map, and Result remain valid and mean that their contained values are dynamic.

• Literal contents, indexing, loops, keys(), values(), and unwrap() propagate their known types.

• Map keys may be None, Bool, Int, Float, Number, String, Bytes, or Any. Numeric map literals infer Number keys because 1 and 1.0 are the same key in SIPL.

• When a value crosses from Any into a generic type, the VM validates its contents. A validated collection remembers its descriptor, so later checks are O(1) and every mutation through any alias must preserve the type.

4. Operators (highest to lowest precedence)#

"Total: " + 5 is an error. Use interpolation instead: "Total: {5}".

5. Control flow#

if score >= 90 {
    grade = "A"
} else if score >= 80 {
    grade = "B"
} else {
    grade = "C"
}

while queue.length > 0 { ... }

for i in 0..10 { ... }            // Range → Int
for item in list { ... }          // List → item
for i, item in list { ... }       // List → index, item
for key in map { ... }            // Map → key
for key, value in map { ... }     // Map → key, value
for ch in "abc" { ... }           // String → one-character String

Conditions must be Bool. Write if not list.isEmpty, not if list.

A for loop over a Map may change the values of existing keys, but adding or removing keys during the loop is an error. To do that, loop over map.keys() instead.

5b. Closed enums and exhaustive match#

An enum declares a closed set of nominal variants. Payload fields are named and typed:

enum State {
    Loading
    Ready(value: String)
    Failed(code: Int, reason: String)
}

fn describe(state: State) -> String => match state {
    Loading => "loading"
    Ready(value) => "ready: {value}"
    Failed(code, reason) => "failed {code}: {reason}"
}

• A variant without a payload is a value: State.Loading. Do not call it.

• A payload variant is a constructor. It accepts positional or named arguments: State.Failed(7, "bad") or State.Failed(reason: "bad", code: 7).

• Enums are nominal. Two enums with identical variants are still different types, and enums in different modules are distinct.

• match evaluates its subject once and is an expression. The subject must have a statically known enum type.

• Every variant must appear exactly once. Missing, duplicate, unknown, incorrectly qualified, or incorrectly bound patterns are compile errors. There is deliberately no wildcard pattern.

• Payload bindings are immutable typed locals in declaration order.

• A pattern may be unqualified (Ready(value)), enum-qualified (State.Ready(value)), or module-qualified (network.State.Ready(value)).

• Expression arms must produce one compatible type. Int, Float, and Number arms merge to Number; incompatible results are compile errors. A block arm produces None unless it exits with return or fail.

• Because { ... } immediately after => is a block arm, parenthesize a Map literal result: Ready(value) => ({"value": value}).

• Enum values compare structurally, print with their variant and named payloads, expose their enum through .type, and retain runtime validation when crossing an Any boundary.

6. Functions#

fn add(a: Int, b: Int) -> Int {
    return a + b
}

fn square(x) => x * x                      // one-expression body

let double = fn(x) => x * 2                // lambda
let counter = fn() {                       // lambda with a block body
    return 1
}

add(1, 2)
add(a: 1, b: 2)                            // named arguments, any order
add(1, b: 2)                               // positional first, then named

Functions are values. Closures capture variables by reference, and each loop iteration gets a fresh variable. Parameters cannot be reassigned; copy one into a var if you need to change it.

Core native operations with shared contracts also accept named arguments; sipl api symbol --json lists their exact labels/types. Labels may reorder supplied arguments, but cannot skip earlier optional parameters or name a variadic parameter. APIs not yet in that inventory require positional calls. See API discovery for precise properties and inferred awaited types.

6b. Failures (try, fail, next)#

try {
    let text = file.read("settings.txt")
    print(text)
} fail {
    print("Could not read settings: {fail.why}")
} next {
    print("finished")
}

• try needs a fail block, a next block, or both, in that order.

• Inside the handler, fail is an immutable local Fail value. Its read-only properties are why (the message), kind ("fail", "runtime", "file", or "system"), and line (the originating source line).

• fail("reason") raises a failure. fail(fail) re-raises the same value, preserving its message and line.

• Runtime failures propagate through function calls and built-in callbacks to the nearest active handler. Syntax and compilation errors are not catchable.

• next runs after success or failure, including when return, break, or continue leaves the protected block or handler. Nested cleanup runs from inner to outer.

• A failure raised in a handler still runs its next block before propagating. A failure or return in next replaces the pending failure or return.

• Without a handler, an uncaught failure stops the program with a diagnostic and stack trace. There are no catch or finally keywords, and try is not an expression.

7. Types#

type Point {
    let x: Float          // `let` field: fixed after construction
    var label: String     // `var` field: can be reassigned

    fn length() -> Float => sqrt(self.x * self.x)

    fn toString() -> String => "{self.label} at {self.x}"   // optional override
}

let p = Point(3, "start")            // positional, in field order
let q = Point(label: "end", x: 5)    // or named
p.label = "moved"
print(p.length(), p.type)            // 3.0 Point

• Declared only at the top level. Names are PascalCase. Every field needs a type; there are no default values.

• The constructor takes every field and checks its type (an Int is accepted for a Float field).

• Inside methods, self is the instance. Fields are properties (p.x), and methods are called with ().

• Every type inherits the base Object. It may override toString() (used by print and interpolation) and equals(other) (used by ==, contains, indexOf). type, hash and copy are fixed.

• Default == compares the fields of two values of the same type. Values of different types are never equal.

• Instances cannot be Map keys (they are not hashable).

7b. Modules (use)#

// geometry.sipl
let PI = 3.14159

type Point {
    let x: Float
    let y: Float
}

fn distance(a: Point, b: Point) -> Float => sqrt(pow(a.x - b.x, 2) + pow(a.y - b.y, 2))

// main.sipl
use geometry                // geometry.sipl in the same folder
use shapes/circle           // shapes/circle.sipl
use ../shared/util          // a folder up

fn start() {
    let a = geometry.Point(0, 0)
    print(geometry.distance(a, geometry.Point(3, 4)), geometry.PI)
}

• One file is one module. Its name is the file name, and its members are always written module.name.

• use goes at the top level of a file. Paths are relative to the file that contains the use.

• Names starting with _ are private to their file.

• A module's vars can only be changed by that module's own code. Give it a function for outside changes.

• Each module is loaded once and initialized before the files that use it. Modules that use each other in a circle are an error; move the shared part into a third module.

• Types in different modules are different types even with the same name. p.type shows geometry.Point.

• Module members are resolved when compiling: no lookup cost at run time, and wrong names or argument counts are reported before the program runs.

Data and I/O modules#

• json.parse(text), json.isValid(text), and json.stringify(value, pretty?) convert between JSON and SIPL values. JSON objects become String-keyed Maps; null becomes none.

• result.ok(value) and result.err(fail) construct a Result. Results expose .isOk, .isErr, .value, and .error, plus unwrap(), unwrapOr(default), map(fn), andThen(fn), and mapError(fn).

• path provides join, normalize, isAbsolute, basename, dirname, and extension. fs provides text/binary-safe readText/writeText, exclusive empty-file createFile, exists, isFile, isDirectory, list, recursive sorted walk (relative files, no symlink traversal), size, makeDirectory, remove, and copy.

• http.request(method, url, options?), http.get(url, options?), and http.post(url, body, options?) return NetTask. Poll with net.poll() or block explicitly with .wait() to obtain a typed HttpResponse with .status, .headers, .body, and .url. Portable TLS, streaming Bytes, cancellation, redirect policy, proxies, pooling, certificates, and connect/read/total deadlines are covered in HTTP.

• tcp, udp, websocket, and framing share the NetTask/CancelToken model. See networking for bounded streams, socket services, and WebSocket client/server APIs.

• The web module provides routes, middleware, static files, and secure server-side sessions with OS-generated opaque IDs, bounded expiry, rotation, and destruction; see docs/WEB.md.

• auth, identity, and optional ldap provide password hashes, PKCE/OIDC/JWT validation, immutable Identity values, and native directory access. OidcClient and LdapDirectory network methods return NetTask. See identity and authentication for supported protocols, trust boundaries and optional SDKs. These module names follow the existing top-level reserved-name rule.

• The generic graphics module provides typed applications, retained scenes, transforms, cameras, input, and depth-tested drawing without exposing backend-specific names; see docs/GRAPHICS.md.

Logging and media#

log.open(options?) creates an explicit Logger; audio.open(options?) creates an AudioApp that owns AudioSound handles; audio.stream(path, options?) creates an AudioStream with native chunked playback for long music files; image creates owned Image values. Their contracts, limits and examples are discoverable through sipl api. See logging, audio, and images. The new built-in names log, audio, and image follow the existing top-level reserved-name rule; use a distinct name for a top-level variable or module member.

8. Formatting#

sipl fmt file.sipl rewrites a file in the one official style. sipl fmt --check only reports. The formatter keeps every comment and refuses to write anything that would change the program.

9. User interface (ui)#

fn start() {
    var count = 0
    var name = ""
    ui.open("Counter", 400, 300)          // title, width, height
    while ui.frame() {                    // false when the window closes
        ui.heading("Hello")
        ui.text("Count: {count}")
        if ui.button("Add") {             // true on the frame it is clicked
            count += 1
        }
        name = ui.textInput("Your name", name)
    }
}

The interface is redrawn from your variables on every frame. Widgets return the new value, so there are no callbacks and nothing to keep in sync. Positions are in logical pixels from the top-left corner. Colors are Strings: "#RGB", "#RRGGBB" or "#RRGGBBAA".

For continuous game movement, ui.keyDown(key) returns whether a key is held while the window has focus. It accepts letters (case-insensitive), digits, "left", "right", "up", "down", "shift", "ctrl", "alt", "space", "enter", "escape", and "tab". Use ui.keys for per-frame key events and shortcuts; use the native gamepad module for controllers. Tests can enqueue held-key changes with ui.setKeyDown(key, down) on an open window. On macOS, Command also maps to ctrl. Ctrl shortcut events additionally support digits, plus/equal and minus. ui.moveMouse(x, y) queues pointer movement for UI automation without pressing a button.

ui.tooltip(text, style?) attaches to the preceding widget. A one-second monotonic hover delay precedes display; clicks, dragging, leaving the widget, focus loss and popup capture suppress stale hints. Tooltips are painted in a final overlay above panels and later drawing, independent of their parent's clip. Long hints wrap and stay inside the window. Idle native windows wake once at the hover deadline rather than continuously animating.

ui.titleBar(height, dragWidth, interactive?) optionally replaces the native Windows caption with a client-drawn region and returns whether the platform supports it. Height is 0..128 (0 restores the native caption), drag width is 0..16384 logical pixels. interactive is a list of up to 64 rectangles {x,y,width,height} that retain normal client input. All remaining space in the top region is a native caption hit target: dragging moves the window and double-clicking maximizes it. The OS continues to own frame resizing. ui.windowAction("minimize"|"maximize"|"close") requests a native window action; maximize toggles restore. Both APIs return false on unsupported backends; ordinary windows retain their existing behavior unless explicitly opted in. Applications using client chrome must draw their own caption controls.

Buttons and styled entry/choice widgets accept paired integer x and y options for absolute positioning. For example, ui.textInput("Name", name, {"x": 220, "y": 100, "width": 240}) places the input at that location without advancing the normal layout cursor.

Text drawn with ui.text, ui.heading, or ui.drawText is selectable by default: drag across it, then use Ctrl+C. Multiline text is one selection block, so a drag can span lines and the copied String preserves newline characters. For decorative labels that should not accept selection, pass the explicit option {"unselectable": true} (ui.drawText takes it as the fifth argument). ui.textInput and ui.textArea support click/drag selection, Shift+arrow selection, and the usual Ctrl+A/C/X/V shortcuts. Multiline text areas scroll with the wheel, keep the caret visible, and show a thin scrollbar only when their content exceeds the visible rows. Inside a frame, ui.textAreaWrap(text, before, after) wraps the last drawn text area's selection with two strings and returns the updated text. With no selection, it inserts both strings at the caret and leaves the caret between them; formatting buttons can call it even when the text area was drawn in the previous frame.

Shared widget styles#

ui.setStyle(source) replaces the shared style with a validated snapshot of a Map. The source can be exported by a SIPL module or loaded with json.parse(fs.readText("theme.json")). Updating the original Map has no effect until setStyle is called again. An empty Map or ui.resetStyle() restores built-in defaults. Styles can be installed before opening a window and persist across window reopenings.

let theme = {
    "accent": "#8250DF", "text": "#202030", "radius": 8,
    "widgets": {
        "button": {"background": "#8250DF", "text": "#FFF", "padding": 20},
        "textInput": {"background": "#FAF8FF", "width": 320},
    },
}
ui.setStyle(theme)
// Inside a frame:
ui.button("Save")
ui.button("Delete", {"background": "#CF222E", "hover": "#A40E26"})
let reusable = {"radius": 2, "borderWidth": 2, "border": "#8250DF"}
ui.textInput("Name", name, {"style": reusable, "width": 400})
ui.withStyle({"accent": "#1A7F37"}, fn() {
    ui.checkbox("Section accent", true)
})

Every flow widget accepts an optional trailing style Map, including text, headings, choices, status widgets, text entry, scroll views, tooltips, and separators. Buttons, boxes, panels, scrollbars, context menus, and video controls accept style fields in their existing options Map. A reusable Map can be supplied as the style option. Resolution order is built-in defaults, common source fields, the source's widgets[widgetName], reusable style, then direct options. ui.withStyle(source, fn) temporarily overlays the common and widget source fields; nested scopes restore both after completion or an error. Widget-specific fields still override common fields within a scope. A box's local style affects its frame; use withStyle to style its children as well.

ui.getStyle() returns a detached Map of resolved common values; ui.getStyle("button") returns resolved defaults for one widget. These are useful for custom drawing and style editors. They exclude per-call overrides; use the original source Map to save or reload the entire theme. Unknown source fields and widget names, malformed colors, and invalid types or ranges raise an error before replacing the active source. Existing non-style behavior options remain valid.

Colors accept #RGB, #RRGGBB, and #RRGGBBAA, including transparent fills. Fields apply where a widget draws that part: for example, shadow belongs to context menus, and thumb to sliders, toggles, radios, and scrollbars. Explicit panel, box, scrollbar, and video geometry stays in their existing arguments; container padding controls child layout. Larger text can be paired with an explicit control height. Disabled containers disable their own resizing or scrolling; child controls resolve their own enabled values. Color-picker swatches, HSV gradients, and contrast samples retain their actual color values.

Widget names for widgets and getStyle are text, heading, button, checkbox, toggle, radio, slider, select, tabs, collapsible, progress, spinner, badge, colorPicker, tooltip, textInput, textArea, box, panel, scrollbar, contextMenu, scrollView, separator, and drawVideo.

Action backgrounds follow accent and action text follows onAccent unless explicitly supplied. Focus and selected text follow accent; tracks follow border; placeholders and disabled text follow muted. Hover and pressed colors remain separately configurable. See examples/ui_styles.sipl for a live theme whose accent can be changed with a color picker.

Toolbar buttons and context menus share a C menu-item parser. Define actions in SIPL as {"id": "createFile", "label": "Create file", "shortcut": "Ctrl+N", "enabled": true}. Only label is required; id defaults to the label, shortcut to empty, and enabled to true. IDs and labels contain 1–512 bytes; shortcut hints at most 128. A String item uses that String as both ID and label. SIPL dispatches the selected ID and implements keyboard shortcuts; C draws shortcut hints without installing global key bindings. Disabled items cannot activate from mouse or keyboard.

ui.button(item, options?) returns whether it was clicked. Its default placement still follows flow layout. Options x and y together select absolute placement; width and height override its measured size. Options also accept enabled, background, hover, pressed, text, and muted colors. Coordinates are 0–16384 and dimensions 1–16384. Button identity follows the item's ID rather than its display label. align accepts "left", "center" (default), or "right"; left/right alignment uses the button's padding.

Data tables#

ui.table(id, columns, rows, style?) draws a native table with left-aligned headers and cells. Drag header dividers to resize adjacent columns while keeping the table width fixed. Resizing defaults to true; pass {"resizable": false} to disable the handles.

var widths: List[Int] = []
while ui.frame() {
    let view = ui.table("results", ["Name", "Score"], [["Ada", 9.5]],
        {"width": 600, "columnWidths": widths})
    widths = view["columnWidths"]
}

Keep the returned columnWidths for the next frame, as with resizable panel geometry. The result also contains clickedColumn, clickedRow (zero-based, or -1), resizing, x, y, width, and height. Dragging a divider consumes the pointer event and does not click a header or row. sortable: false disables header clicks. Sorting and row actions remain application decisions.

Options include paired x/y, width, height, minColumnWidth (default 48), headerHeight (36), rowHeight (29), selectedRow, stripe color, and the shared widget styles. selected and selectedText style the header. Widths adapt to the available space; minimums shrink only when necessary to fit. The widget accepts 1–256 columns and up to 10,000 rows, with one value per column in every row. Cells are clipped to their columns, rendered on one line, and show None as NULL. Page or scroll large results in the application.

ui.contextMenu accepts 1–64 items and returns {open, selected, x, y, width, height}. Assign open back to application state; selected is the chosen item's ID or empty when nothing was selected. Mouse clicks or Enter activate an enabled item; Up/Down skip disabled items. Escape, an outside click, or window focus loss closes the menu. It fits within the window and scrolls longer lists with the wheel or keyboard. Options are minWidth and background, text, border, hover, muted colors. Only one popup is active at a time. The optional openingPress: true ignores the press used to open a popup in that frame; pass false afterward so outside clicks dismiss normally.

Call the menu before processing underlying inputs. While open, it consumes mouse presses, typed text, keys, and wheel input. C paints it above later widgets at presentation and screenshot time, independent of the enclosing clip. The same event path handles right clicks and focus changes on Windows and Linux; ui.rightClick injects matching headless events.

let create = {"id": "createFile", "label": "Create file"}
// The same item works in either control; application code owns its action.
let popup = ui.contextMenu("explorer", menuX, menuY, menuOpen, [create])
menuOpen = popup["open"]
if popup["selected"] == "createFile" {
    beginCreate()
}
if ui.button(create) {
    beginCreate()
}

Panels and boxes share optional resize handles. Pass a stable id and a resize List containing any of "left", "right", "top", "bottom"; the default is no handles. Each handle occupies six pixels inside its edge, highlights on hover, shows a centered grip, and captures a drag until release, including outside the panel. Windows and Linux show the corresponding resize cursor. Reserve these edge pixels when drawing custom children. enabled: false disables resizing. Options also accept minWidth, maxWidth, minHeight, maxHeight, and background, border, accent colors. Default minimum dimensions are 32 pixels (or the window size if smaller); default maxima are the window size. Limits must satisfy 1 <= min <= max <= 16384. Panels are additionally constrained to their enclosing clip.

ui.panel draws an absolute surface and returns {x, y, width, height, resizing}. Keep the returned geometry in application state; derive adjacent rectangles from it each frame. Left/top drags move the origin while keeping the opposite edge fixed. A box with options returns the same geometry and automatically moves subsequent flow widgets when its size changes. Flow boxes keep their layout origin for all handle sides. A three-argument box retains its None return. Three-argument boxes require dimensions of at least 32 pixels; boxes with options accept dimensions from 1 to 16384. Call resizable controls before custom child input: captured presses are removed from ui.isMousePressed, and native widgets drawn afterwards cannot also activate from that press.

ui.row(fn() {
    let pane = ui.box(sidebarWidth, 400, fn() { ui.text("Explorer") },
        {"id": "explorer", "resize": ["right"], "minWidth": 160, "maxWidth": 400})
    sidebarWidth = pane["width"]
    ui.box(max(32, ui.width - sidebarWidth - 48), 400, fn() { ui.text("Editor") })
})

ui.scrollbar uses application-defined units for offset, viewport, and content, so a code editor can scroll in rows or columns. It clamps to 0..max(0, content - viewport), supports dragging the thumb and clicking the track, and returns the updated offset. Options are horizontal (default false), thickness (4–32, default 12), enabled, and background, thumb, accent colors. Length and viewport must be positive; content may be zero. A bar with no overflow draws a full thumb and does not capture input. Stable IDs preserve capture across frames. ui.drag queues a press, movement, and release on successive input frames for deterministic desktop/headless tests.

select and tabs take a non-empty List and return one of its values. Radio buttons are composable: set the shared choice when a radio returns true. A scroll view keeps no hidden application state, so assign its returned offset each frame:

offset = ui.scrollView(200, offset, fn() {
    for item in items {
        ui.text(item)
    }
})

colorPicker starts as a compact current-color display and 16-swatch palette. Its Expand control opens the full editor inline in the same SIPL window: saturation/value field, hue and opacity strips, current and previous colors, RGB/HSL/HSV values, tint and shade suggestions, color harmonies, and white/black WCAG contrast ratios. Compact returns to the small state. The widget returns the selected #RRGGBB or #RRGGBBAA String every frame.

When hosted in an openFixed window, the picker automatically switches the client area between its fixed 420×150 compact size and 740×450 expanded size. The fixed window has a close button but no minimize, maximize, or resize controls.

The three picker functions use native Windows dialogs or built-in canvas dialogs on Linux. Linux dialogs let you browse folders, type or paste a path (Ctrl+L), choose with Enter, and cancel with Escape. Saving an existing file requires an explicit replacement confirmation. They return none on cancellation and in headless test mode.

Linux windows use X11/XWayland and the same canvas and fonts as Windows. Desktop window decorations follow the host theme. make SIPL_UI_X11=0 builds a runtime for headless use without X11 development packages. The macOS implementation hosts the same widgets in a Cocoa window, including Retina scaling, clipboard and file panels. Command shortcuts use the existing ctrl+ key names. See Mac build and acceptance for its deployment target and pending native qualification.

Key names: backspace delete enter tab escape left right up down home end pageUp pageDown, the same with shift+ in front (shift+left), and ctrl+a … ctrl+z, ctrl+space, and ctrl+enter. Ctrl combinations also preserve Shift, for example ctrl+shift+g. Input arrives in the order it happened: a frame never mixes a click with keyboard input, and holds typed text followed by at most one key.

Files and the system#

sipl run program.sipl a b makes system.args == ["a", "b"].

system.securityRevision is a read-only Int reporting the runtime's security patch level (currently 2). Toolkit's runtime compatibility check requires at least that revision; rebuild old runtimes and packaged applications to update them. SIPL programs run with the current user's OS permissions. File/process APIs and Toolkit Run commands are not a sandbox for untrusted code.

Set SIPL_UI=headless to run without a window: tests, CI and servers. In headless mode ui.frame() returns false once there is no input left to process.

10. Built-in names (the complete list)#

Every value (base Object)#

.type · .toString() · .equals(x) · .hash() · .copy()

Int / Float#

.toInt() (truncates) · .toFloat()

String#

Properties: .length · .isEmpty

.toUpper() · .toLower() · .trim() · .trimStart() · .trimEnd() · .split(sep) / .split() (splits on whitespace) · .contains(s) · .startsWith(s) · .endsWith(s) · .indexOf(s) (-1 if missing) · .replace(old, new) (replaces all) · .replaceFirst(old, new) · .repeat(n) · .slice(start, end?) · .chars() · .reverse() · .toInt() / .toFloat() (none if invalid) · .padStart(width, fill?) · .padEnd(width, fill?)

List[T]#

Properties: .length · .isEmpty · .first · .last (none if empty)

Mutating: .push(x, ...) · .pop() · .insert(i, x) · .removeAt(i) · .remove(x) (→ Bool) · .clear()

Non-mutating: .contains(x) · .indexOf(x) · .slice(start, end?) · .chunk(size) · .flatMap(fn) · .groupBy(fn) · .join(sep) · .reverse() · .sort() · .sortBy(fn) · .map(fn) · .filter(fn) · .reduce(fn, initial) · .find(fn) · .any(fn) · .all(fn) · .sum() · .min() · .max()

Batching, flattening, and grouping#

fn start() {
    let numbers: List[Int] = [1, 2, 3, 4, 5]
    let batches: List[List[Int]] = numbers.chunk(2)
    print(batches) // [[1, 2], [3, 4], [5]]

    let words = ["red blue", "green"].flatMap(fn(line) => line.split())
    print(words) // ["red", "blue", "green"]

    let groups = numbers.groupBy(fn(n) => n % 2)
    print(groups[1]) // [1, 3, 5]
    print(groups[0]) // [2, 4]
}

These methods allocate new containers and leave the source structure unchanged. Copies are shallow: mutable objects inside them retain their identity. chunk and groupBy preserve validated input item types in their batches, including through Any aliases. flatMap results are dynamic because a callback can return different item types; an explicit List[T] annotation validates them.

flatMap and groupBy iterate a shallow snapshot taken before the first callback. A callback may clear, extend, or reorder the original list without changing which items are visited. Callback failures propagate normally; callback side effects are not rolled back. chunk takes O(n) time and space; groupBy takes expected O(n) time and O(n) space; flatMap takes O(n + m) time and space plus callback work, where m is the output count. Named arguments are available: numbers.chunk(size: 2) and numbers.groupBy(key: fn(n) => n % 2).

Map[K, V]#

Properties: .length · .isEmpty

.get(key, default?) · .set(key, value) · .contains(key) · .remove(key) (→ Bool) · .keys() · .values() · .clear() · indexing map[key] (error if missing) and map[key] = value

Result[T]#

Properties: .isOk · .isErr · .value (none on failure) · .error (none on success)

.unwrap() · .unwrapOr(default) · .map(fn) · .andThen(fn) · .mapError(fn)

Range#

Properties: .length · .isEmpty · .start · .end

.contains(n) · .toList()

Function#

Read-only properties: .name, .callCount (successful calls), .loopCount (loop backedges executed), .jitStatus, and .nativeCallCount.

Type (a type itself) and instances#

Type property: .name. Instances: every field as a property, the type's methods, and the base Object members.

Global functions#

print(...) · input(prompt?) · assert(cond, message?) · abs(x) · sqrt(x) · pow(a, b) · floor(x) · ceil(x) · round(x) (these three return Int) · min(a, b, ...) · max(a, b, ...) · random() · randomInt(min, max) (max excluded) · exit(code?)

Standard-library types and modules#

Bytes, Regex, Prng, Instant, Monotonic, Duration, Date, LocalDateTime, TimeZone, Interval, Cron, Locale, and Translator support type annotations. The native bytes, unicode, regex, url, encoding, math, time, i18n, env, and config modules are available globally. String stores UTF-8 bytes; its core length, indexing, and slicing use bytes. Unicode scalar/grapheme operations are explicit. HTTP bodies use Bytes. The callable time() has been removed; use time.now() for UTC timestamps and time.monotonic() for elapsed measurements. See the standard-library API contracts for limits and examples.

The explicit secrets.load(path) / secrets.save(path, map) APIs encrypt local JSON settings with Windows DPAPI. secrets.userPath() locates the current user store; project stores live under the project's .sipl-toolkit. Toolkit remembers OpenAI/Google keys with these APIs; see encrypted settings.

11. Names SIPL does not use#

12. AI cheat sheet#

fn start() {
    var totals = {:}
    let lines = ["a,3", "b,5", "a,2"]
    for line in lines {
        let parts = line.split(",")
        let key = parts[0]
        let amount = parts[1].toInt()
        totals[key] = totals.get(key, 0) + amount
    }
    for key, total in totals {
        print("{key}: {total}")
    }
}

API discovery and the coding loop#

Native contracts for the language core and modules including networking, time, HTTP/HTTPS serving, logging, audio, images, gamepads, PDF, document collections, and linear algebra share one inventory: src/native_api.def. Compiler/editor inference, native named arguments, option-key/type validation, and discovery consume that inventory. Operation-specific native checks still enforce ranges, nested data, security, resource bounds, and ownership. APIs outside that inventory, including many UI, graphics, worker, and AI operations, retain their module references and require positional arguments. Use sipl api to check which contracts are available in the runtime you are using.

Find a contract or complete program#

sipl api http.get
sipl api web.serve
sipl api time --json --limit 10
sipl api HttpResponse --json
sipl api --task http --json
sipl api --verify

Select an exact symbol, a module, or a native type. No selector lists the inventory. JSON output has schemaVersion: 1, runtimeVersion, entries, shapes, matched, returned, truncated, and ok. Each entry includes its kind, named parameters, signature, return/awaited type, effects, option schema, limits, failure guidance, availability, example, and reference path. Parameters are structured objects with name, type, optional, and variadic. Property entries include mutable: native sound controls have checked setters, while image dimensions and other read-only metadata cannot be assigned. Options and named Map shapes use semicolon-separated field:Type contracts; option keys are optional unless the operation's reference explicitly requires them. | means alternatives, not new SIPL union-annotation syntax. Generic placeholders T, K, V, and Self refer to the receiver's type. Dynamic JSON, unions, and callback-dependent results remain conservative Any in the compiler.

--limit accepts 1–1024 and explicitly reports truncation. Unknown symbols return ok: false and exit 1. Usage errors exit 2. --verify compares required core/module/type operations and arities against the actual runtime tables; the regression suite also executes self-contained contract examples and checks their result types. Discovery does not execute example code or contact a server.

--task returns an exact, complete recipe embedded from examples/recipes/: files, json, http, cli, service, gui, scheduling, clock_tests, library, logging, audio, or image. JSON includes task, description, command, path, and source. command is the driver (run, test, or doc), not a shell command. Read the recipe's explicit argument assertions before running it. Recipe tests compare embedded bytes with source and exercise file outputs, loopback HTTP/service behavior, scheduling, headless GUI, documentation, and normal/GC-stress execution. After changing a recipe, regenerate its embedding:

python3 scripts/embed_api_recipes.py

Names and awaited types#

Known native operations accept the same argument-label syntax as user functions:

let date = Date(day: 29, year: 2024, month: 2)
let stamp = time.format(zone: TimeZone("UTC"), instant: time.fromUnixNanoseconds(0))

Evaluation stays in source order; native dispatch reorders the resulting values. Labels may reorder supplied parameters but cannot skip an earlier optional parameter or name a variadic parameter. Unknown/duplicate labels fail. Native operations without metadata continue to require positional arguments.

Awaited types come from the same contracts: await http.get(...) produces an inferred HttpResponse; awaiting Interval.next() produces Monotonic. Annotated async user functions preserve their declared return type through their task and await. Task[T]/NetTask[T] describe inferred editor facts; they are not additional user annotation syntax. Input/domain failures remain catchable native errors, including inside negative tests.

Generate, check, repair, prove#

1. Retrieve the relevant symbol or task, then make a small change.

2. Run sipl fmt --check file.sipl and sipl check --json file.sipl.

3. Apply only a verified suggested fix, or use its code/range/help to repair.

4. Run sipl test tests --filter "relevant case" --json build/tests.json.

5. Keep a full make test / run_tests.ps1 verification gate.

The editor service caches compiler facts, invalidates changed files and known dependents, and runs cancellable background analysis. Ordinary CLI checks are fresh, compile-only checks; they do not use a persistent cache or execute SIPL. Test selection is an explicit filter, not inferred affected-test selection. See LSP.md, TESTING.md, and DEBUGGING.md for structured diagnostics, reports, and crash data.

Reproducible measurements#

python3 bench/agent_usability.py build/sipl --output build/agent-baseline.json
python3 bench/agent_usability.py build/sipl --baseline build/agent-baseline.json

On Windows use python and build/sipl.exe. The suite pins its version, runtime version/binary hash, and canonical recipe source hashes, then measures median discovery/check latency with a monotonic clock. Baseline comparisons require matching task source. This is a tooling baseline, not a claim of measured model success or token savings. Semantic recipe tests are a separate gate.

Optional --agent-results imports actual measurements without calling a model, spending tokens, or executing generated code. Input is an object with pinned model, promptVersion, and a nonempty tasks list. Each task has task, Boolean firstCheckPassed/success, and nonnegative finite repairs, inputTokens, outputTokens, toolCalls, and codingMilliseconds. The report aggregates success rates and repair/token/tool/time totals. Real agent runs must pin prompts, environment, and task inputs themselves; never substitute canonical recipe checks for first-generation success.

Everyday standard library#

Native processes and completed files#

system.exec(arguments: List[String]) -> Map launches an executable with a literal argument list. The first item names the executable; remaining items are its arguments. It inherits the current directory and environment, supplies closed input, waits for completion, and captures combined standard output/error. Use it inside a SIPL worker for long jobs in a desktop application.

The result contains exitCode, output, and outputTruncated. Capture retains up to 8 MiB while continuing to drain the child. Limits are 1..4096 arguments and 1 MiB of UTF-8 argument bytes; Windows additionally limits its encoded command line to 32,766 characters. NUL bytes and an empty executable are rejected. Executable lookup uses PATH. Missing executables report exit code 127. Arguments are passed directly to the process, with Windows CRT quoting where required. Descendants retain their SIPL worker's process group or Job Object, so worker cancellation terminates the encoder as well as its SIPL parent.

fs.move(source, destination, replace?: Bool) publishes a completed regular file on the same volume. By default an existing destination causes failure and is preserved. Explicit replace: true performs an atomic replacement, useful for progress snapshots. Stage temporary files beside their destination. On POSIX, exclusive publication uses a hard link followed by source removal; if removing the source fails, both names remain and the call reports failure.

let result = system.exec(["ffmpeg", "-version"])
assert(result["exitCode"] == 0, result["output"])
fs.move("finished.tmp", "finished.mp4")

The native modules bytes, unicode, regex, url, encoding, math, time, i18n, env, and config are available in every program. No import or package download is required. New annotation names are Bytes, Regex, Prng, Instant, Monotonic, Duration, Date, LocalDateTime, TimeZone, Interval, Cron, Locale, and Translator.

The native auth, identity, and optional ldap libraries provide password hashes, secure random tokens, OAuth/OIDC, JWT validation, typed identities and directory access. Their APIs, LDAP SDK requirements, ui.passwordInput and security boundaries are documented in the identity guide, with working examples. LDAP availability is explicit through ldap.available; other standard libraries remain usable without a directory SDK.

The optional native webview module hosts the installed Windows WebView2 engine. webview.available reports availability; webview.run(url, options?) opens a same-origin window without a SIPL/JavaScript authority bridge. The reusable meeting library supplies one-call meeting server/desktop setup, WebRTC signaling, admission, chat and file/PDF sharing. It builds on web.serve with its optional literal IPv4 host and server-owned window.

Invalid arguments and resource limits raise catchable Fail values. APIs do not silently replace malformed text, discard unknown regex flags, or read unrequested environment variables. Tables below state parameters, returns, failures, and a canonical example; each section states effects and complexity.

This prerelease changes HTTP request/response bodies to Bytes and removes the callable global time(). Use time.monotonic() for measurements and time.now() for UTC timestamps. Examples and Toolkit use these APIs directly.

Bytes and binary I/O#

Bytes is immutable binary storage, limited to 16 MiB per value. Indexing returns an Int from 0 to 255; negative indices count from the end. Out-of-range indices fail. Iteration yields bytes, or index/byte pairs with two loop names. Equality compares contents. Bytes can be Map keys. .length is an Int and .isEmpty is a Bool; assignments to bytes or their properties fail.

Slices use an exclusive end, clamp bounds, and return empty Bytes when the end precedes the start. File read limits default to 16 MiB and must be between zero and 16 MiB. Text APIs fs.readText/file.read and fs.writeText/file.write validate UTF-8; use binary APIs for arbitrary data.

Constructors, slices, conversions, content equality, and hashing take O(n) time; new values use O(n) space. Length, indexing, and emptiness take O(1). secureEquals visits every byte for equal-length inputs without data-dependent early exit; different lengths return false immediately. It is intended for comparing secrets of a known, fixed length. Binary reads and writes perform filesystem I/O and take O(n) time and O(n) read space; writes can create or truncate files. Other Bytes APIs only allocate values.

fs.createFile creates an empty file exclusively and never replaces an existing entry, including a symlink. Its parent must already exist. It performs one filesystem create/close operation with O(path length) conversion space on Windows. Explorer uses it after inline filename confirmation, so concurrent changes cannot cause an existing file to be truncated.

fn start() {
    let response = http.post("https://example.com/data", Bytes([0, 128, 255])).waitBlocking()
    fs.writeBytes("response.bin", response.body)
    // Decode only when the application expects UTF-8 text:
    print(response.body.decodeUtf8())
}

HTTP options body and http.post bodies require Bytes. Response body is Bytes. Web handler request bodies are Bytes; responses accept String text or Bytes binary data. See HTTP and Web.

ZIP archives#

zip uses the bundled MiniZip and zlib sources. zip.open(path) opens an existing ZIP or creates an empty one and returns a ZipArchive. Call zip.close(archive) when done; later calls with that handle fail. All methods perform blocking filesystem I/O.

fn start() {
    let archive: ZipArchive = zip.open("notes.zip")
    zip.add(archive, "notes.txt", "docs/notes.txt")
    print(zip.list(archive))
    let contents: Bytes = zip.read(archive, "docs/notes.txt")
    print(contents.decodeUtf8())
    zip.extract(archive, "docs/notes.txt", "copied-notes.txt")
    zip.remove(archive, "docs/notes.txt")
    zip.close(archive)
}

Added entry names must be relative paths using /, without .. segments. zip.add and zip.remove rewrite the archive through a temporary file in the same folder and replace the original only after a successful write. Writers and streaming extraction retain their file and directory handles inside a private staging directory; they do not close and reopen a shared temporary filename. The final commit stays anchored to the opened destination parent and replaces the destination entry rather than following a destination symlink. Rewriting copies existing stored and deflated entries without expanding them. Encrypted entries and other compression methods fail without changing the original. ZIP metadata other than entry names, timestamps, attributes, comments, and archive comment is not retained when rewriting. zip.read returns Bytes, so use .decodeUtf8() for text. The 16 MiB limit applies to one in-memory Bytes value, not to the archive or files added to it. Use zip.extract for larger entries.

Unicode text model#

String stores UTF-8 bytes. Its core .length, indexing, .chars(), .slice(), and .reverse() operate on bytes, without implicit normalization. A byte slice can cut a multibyte character; Unicode APIs reject invalid results. Use Bytes for binary storage and the explicit Unicode APIs for text units. String .toUpper()/.toLower() map ASCII only, independently of machine locale. Unicode casing is available through the unicode module.

unicode.version reports the supported Unicode data version as a String. Scalar values exclude surrogates and are between zero and U+10FFFF; embedded U+0000 is preserved. Graphemes are extended grapheme clusters, including combining sequences, emoji ZWJ sequences, and regional-indicator pairs. All defaults are locale-free.

Normalization accepts NFC (default), NFD, NFKC, and NFKD. Case folding uses full Unicode mappings, including expansions such as ß to ss; it does not implicitly normalize. Unicode upper/lower use simple scalar mappings, which differ from full string casing where a character expands. Code-point slicing uses exclusive/clamped/negative bounds like Bytes slicing.

Effects are allocation only. Validation and conversions take O(n) time and O(n) output space; validation alone uses O(1) additional space. Code-point slicing scans the input even for a small slice. Normalization is normally linear but canonical ordering can take O(n²) for a long combining sequence. Inputs and resulting text are limited to 16 MiB.

Encodings and CSV#

Binary encoders take Bytes. Their inverse returns Bytes; text conversion is a separate decision. JSON remains the existing json module. Configuration loading uses that same parser. There is no second JSON or general TOML API.

Hex decoding accepts either case. Base64 is the standard alphabet with required padding, no whitespace, and canonical unused bits. Percent encoding preserves ASCII letters, digits, and -._~; other bytes become uppercase %HH. Decoding leaves + literal; it is not form-urlencoded decoding.

CSV uses a one-byte delimiter, default comma, excluding quote, newline, carriage return, NUL, and non-ASCII bytes. Quoted fields may contain newlines and doubled quotes. Parsing accepts CRLF, LF, and CR row endings, preserves empty fields, performs no numeric conversion, and limits input to 16 MiB and 100000 fields. Empty input has zero rows. Writing emits CRLF and quotes fields as needed, including a sole empty cell; output is limited to 16 MiB.

Effects are allocation only. Each API takes O(n) time and O(n) output space. Encoded strings may be larger than the 16 MiB binary input limit; decoded Bytes remain bounded. CSV also allocates a List for each row and a String for each field. CSV writing does not enforce the parser's 100000-field limit.

Compiled regular expressions#

regex.compile(pattern: String, flags?: String, limits?: Map) -> Regex compiles a regular-expression pattern in Unicode mode. Flags are i (case-insensitive), m (multiline anchors), s (dot matches newline), and x (extended syntax). Unknown or repeated flags fail. Syntax errors report the pattern offset and compiler message. Named captures use (?<name>...). Pattern size is at most 4096 bytes, nesting at most 128, and capture count at most 64.

Canonical compile example: regex.compile("(?<word>\\w+)", "i").

Unknown limit keys fail. Every matching call checks the input limit and shares one work budget across candidate starting positions and all global matches. Automatic callouts and engine match limits stop excessive backtracking; engine depth is capped at 128 and engine heap at 4 MiB. Matching does not use JIT. Returned captures/replacement output have a 16 MiB budget. Hitting any budget raises Fail rather than returning a partial result. maxWork counts engine work checks, not milliseconds or an exact cross-version instruction count.

.pattern and .flags are read-only Strings. Match Maps contain text, start, end, groups, and named. Start/end are UTF-8 byte offsets, with exclusive end; groups include the whole match at index zero, and unmatched optional captures are None. Named is a String-keyed Map. Global operations support empty matches, retry a nonempty alternative at the same position, then advance by a whole Unicode scalar when necessary.

Replacement templates accept $0, numbered captures such as $1, ${name}, and $$ for a literal dollar. SIPL string interpolation requires escaping the braces in a named template: "$\{name\}". Unknown capture references fail. Replace processes all nonoverlapping matches.

Effects are allocation only; compiled values are GC-managed. Compilation cost depends on pattern expansion. Matching is bounded backtracking, not a linear-time guarantee: work may grow rapidly but is stopped by the configured budget. Result construction takes O(total returned bytes + captures) time and space. Programs should choose smaller limits for smaller workloads.

URLs#

URL helpers handle absolute URLs and relative references. Inputs/outputs are limited to 65536 bytes, ports to 0..65535, and query Lists to 1000 pairs. Unescaped controls, spaces, backslashes, malformed escapes, invalid schemes, and invalid bracketed IP literals fail. Hosts use ASCII DNS names or IP literals; supply international DNS names in ASCII IDNA form. No implicit DNS lookup, filesystem access, or network request occurs.

Parsed fields are scheme, authority (Bool), userinfo, host, port, path, query, fragment, encoded (true), and ipLiteral (Bool). Missing optional components are None; path is always String. Host excludes IP brackets. Encoded components retain their percent escapes.

Builders accept raw component Strings by default, encode their UTF-8 bytes, and lowercase the ASCII scheme/host. An explicit host introduces authority; its nonempty path gets a leading slash when needed. Port/userinfo require a host. A path beginning // without a host fails to avoid changing its meaning. The encoded: true marker returned by parse preserves existing escapes when rebuilding; use raw values without that marker when composing new URLs. ipLiteral: true preserves brackets for IPvFuture literals.

Prefer query pairs over a query String when composing values. Duplicate keys and their order are preserved. Plus signs are literal, missing = means an empty value, and a trailing & adds no extra pair. Helpers do not resolve dot segments, infer default ports, or apply browser-specific URL rewriting. Effects are allocation only; time and output space are O(n).

Math and deterministic random objects#

math.pi, math.tau, and math.e are read-only Float constants. Angles use radians and scalar inputs accept Int or Float. math.clamp preserves Int precision when all three arguments are Int; otherwise it returns Float.

Scalar functions take O(1) time/space and perform no I/O. Floating-point overflow may yield infinity; these are Float operations, not exact decimal arithmetic. Libm rounding can differ slightly across platforms.

Prng uses xoshiro256** with SplitMix64 seeding and 53-bit Float draws. Equal seeds and equal call sequences produce equal outputs across supported platforms. .seed is a read-only Int. Draws mutate that object's state only; objects share no hidden random state. Integer draws use rejection sampling to avoid modulo bias (expected O(1)); Float draws are O(1), and byte generation is O(n) time/space. This generator is for simulation/tests, not cryptographic keys or secrets. Decimal and arbitrary-precision integer types remain future work.

Instants, monotonic points, and durations#

These immutable types keep wall-clock timestamps separate from elapsed time. Instant is signed Int64 nanoseconds since the Unix epoch in UTC. Monotonic is a point on the system monotonic clock with an unspecified origin; it is suitable for measurements within the running environment, not persistent timestamps. Duration is a signed Int64 count of nanoseconds. Precision of the actual clocks depends on the platform.

Read-only properties: Instant .unixNanoseconds: Int, .unixSeconds: Float; Monotonic .nanoseconds: Int; Duration .totalNanoseconds: Int, .totalSeconds: Float, and .totalMilliseconds: Float. Wall-clock differences can be negative when the clock changes. Arithmetic checks Int64 overflow. Use monotonic points for elapsed measurements. Instant differences compare stored timestamps and may reflect wall-clock corrections.

All operations take O(1) time/space apart from sleep's elapsed wait. Clock reads observe the OS; sleep blocks the calling thread (rounded up to a millisecond on Windows). Constructors/arithmetic only allocate values.

fn start() {
    let began: Monotonic = time.monotonic()
    print("UTC timestamp", time.now().unixSeconds)
    print("Elapsed milliseconds", time.elapsed(began).totalMilliseconds)
}

Calendar values and explicit timezones#

Date and LocalDateTime are immutable proleptic Gregorian values, years 1..9999. A local datetime has nanosecond precision and carries no timezone. Converting it to an Instant requires an explicit TimeZone; the narrower Instant range is 1677-09-21T00:12:43.145224192Z through 2262-04-11T23:47:16.854775807Z. A timezone is an exact, case-sensitive IANA identifier. Neither TZ, the machine timezone, nor the process locale supplies an implicit default.

SIPL bundles public-domain IANA tzdata 2026c, including historical offsets, compatibility aliases, and future recurring DST rules expanded across the entire Instant range. The executable and packaged programs contain this data on every platform. zone.version identifies the snapshot; future political rule changes require updating the bundle. Regenerate it with scripts/embed_tzdata.py from a leap-free TZif tree compiled by zic. The generator validates every transition against Python's independent TZif reader and does not run during ordinary builds.

Date/local read-only properties are .year, .month, .day, and .weekday (ISO Monday=1 through Sunday=7). Local datetimes also expose .hour, .minute, .second, .nanosecond, and .date. Zones expose .id and .version. Calendar equality compares fields; timezone equality compares the supplied identifier. Aliases have the same rules but retain their own identifiers.

ISO output is locale-invariant ASCII. Parsing follows the Gregorian extended ISO-8601 / RFC 3339 profile above: seconds are required, fractions have 1..9 digits, offsets use Z or ±HH:MM, and lowercase t/z are accepted. -00:00 identifies a UTC instant but its unknown-local-offset metadata is not retained. Leap seconds, 24:00, week and ordinal dates, whitespace separators, and excess precision fail. Output uses uppercase T/Z and trims trailing fractional zeros without losing precision. Offsets containing historical seconds cannot be emitted in RFC 3339; format the Instant in UTC instead.

For overlapping local times, the default "reject" policy fails; "earlier" and "later" select the chronological occurrence. Nonexistent local times always fail. Calendar-day and local arithmetic operate on calendar fields; an elapsed 24-hour Duration added to an Instant may land at a different local hour after a DST change.

Calendar arithmetic is O(1). Zone lookup is O(log Z); offset lookup is O(log T) for a zone's T transitions. Local conversion tests its fixed set of distinct offsets, each with a binary search. No conversion mutates OS timezone settings.

Timers, intervals, and cron#

Schedules create cancellable NetTask waits on the existing async event loop. They inherit lexical task ownership, admission bounds, failure propagation, and scope-exit cancellation described in ASYNC.md. The caller supplies the work in an async function; native scheduling never invokes user callbacks.

Duration timers and interval phases use only the monotonic clock. Interval ticks retain their original phase and skip missed ticks; one pending wait per schedule prevents overlapping consumers and unbounded queues. Native timers have no hidden network timeout, so long waits survive until completion or owner cancellation. Resolution depends on the OS and event loop (approximately one millisecond); nanosecond values do not promise nanosecond wakeups.

Calendar alarms and cron use UTC wall time to decide when a civil deadline is due and follow wall-clock corrections. A past time.at target completes on the next event-loop turn. This is calendar scheduling, not elapsed measurement.

Cron has five numeric fields: minute 0..59, hour 0..23, day-of-month 1..31, month 1..12, weekday 0..7 (0 and 7 mean Sunday). Fields support *, comma lists, inclusive ranges, and positive /step values on wildcards, ranges, or a numeric start (e.g. 5/15 selects 5, 20, 35, 50 in the minute field). Names, seconds, and extended operators are unsupported. When either day field begins with *, both day masks must match, including stepped wildcards. Otherwise day-of-month and weekday combine with OR. DST gaps are skipped and both occurrences of an overlapping local minute are eligible. Computation scans at most eight years of calendar days and the selected hours/minutes, testing offsets without touching the wall clock in nextAfter. next() chooses from the current wall time and skips past runs.

async fn start() {
    let everySecond: Interval = time.interval(time.seconds(1))
    for i in 0..3 {
        let tick = await everySecond.next()
        print("Scheduled monotonic tick", tick.nanoseconds)
    }
    let morning: Cron = time.cron("30 9 * * 1-5", TimeZone("America/New_York"))
    let occurrence: Instant = await morning.next()
    print(time.format(occurrence, TimeZone("UTC")))
}

Explicit locale formatting and translations#

i18n never reads or sets the process-wide locale. Locale objects identify an explicit supported profile: en-US, en-GB, de-DE, fr-FR, ja-JP, or es-ES. Other names fail instead of falling back to a machine default. These are deliberately bounded formatting profiles, not a full CLDR/ICU implementation.

Number formatting defaults to two fractional digits; Int values keep their full 64-bit precision. Float rounding follows fixed-decimal C formatting in an explicit C numeric locale scoped to the operation. French grouping uses U+202F. Date formatting defaults to "short"; "long" uses localized month names. An Instant is converted using the supplied zone. Date/local values retain their calendar fields, and local/converted values append a 24-hour clock time.

Locale .id and Translator .locale are read-only. Locale equality compares the profile ID. Catalog keys are at most 128 UTF-8 bytes and messages at most 16 KiB; messages and supplied String parameters must be valid UTF-8. Templates substitute {name} literally; {{ and }} escape braces. Values requiring number/date formatting should be formatted explicitly before substitution. Missing translations raise Fail; there is no implicit fallback language or global translation catalog. Number/date formatting is O(output length), and translation is O(template plus output length) with Map lookups for parameters.

fn start() {
    let locale: Locale = i18n.locale("de-DE")
    let messages: Translator = i18n.translator(locale, {"price": "Preis: \{amount}"})
    print(messages.translate("price", {"amount": i18n.number(1234.5, locale)}))
}

Explicit environment and configuration#

Allowlist arguments are required. Environment names must be portable ASCII identifiers, 1..128 characters; allowlists hold at most 256 entries. Missing variables produce None, and values must be valid UTF-8 of at most 1 MiB. The APIs use platform environment-name matching; use consistent spelling, especially on case-insensitive Windows.

Configuration allowlists hold at most 256 paths and compare exact path spelling, without searching directories or expanding environment variables. Read limits default to 1 MiB and accept 1..16 MiB. Allowlisting selects the requested file; it is not a filesystem sandbox or a symlink restriction. Loading uses the existing JSON parser and its nesting/node limits. Merge copies entries, leaves its arguments unchanged, and shares nested values. There is no automatic global configuration, implicit dotenv load, or hidden precedence chain.

Environment reads observe process state and allocate O(total returned bytes) space; allowlist scanning and value copying take O(names + bytes) time plus platform lookup cost. Configuration loading reads the filesystem in O(file bytes) time/space plus JSON parsing costs. Merge only allocates a new Map, using expected O(base entries + override entries) time and space.

Structured logging#

Logging is explicit: create a Logger, pass it to code that needs it, and derive child contexts. There is no global logger configuration. Console output defaults to stderr, leaving stdout free for program results and CLI protocols.

fn start() {
    let logger: Logger = log.open({"sink": "json", "level": "info"})
    let request = logger.with({"traceId": "trace-1", "requestId": "request-1"})
    request.info("ready", {"port": 8080})
    logger.close()
}

Use sipl api log --json and sipl api Logger --json for the same named arguments, options, return types, and limits used by editor discovery.

Sinks and levels#

log.open(options?: Map[String, Any]) -> Logger accepts:

File maxRecordBytes must not exceed maxBytes. Options, context types, and limits are validated before opening or creating a file. Unknown options fail. Maps require String keys; structured serialization is bounded to depth 16 and 4096 visited values. Cycles, excessive nesting, and oversized records fail before that record reaches a sink. Non-JSON objects are represented as a type marker without invoking user callbacks; nonfinite numbers are strings.

logger.write(level, message, attributes?), debug, info, warn, and error return None. Sink I/O is synchronous, including from async tasks; buffering reduces write frequency but is not an async/nonblocking transport. Messages are String and attributes are Map[String, Any]. JSON records have timestamp, level, message, context, and attributes. Timestamps use the temporal wall-clock provider and UTC RFC-3339 with nanoseconds; they never use the machine timezone. Elapsed measurements remain the caller's explicit monotonic-clock responsibility. test.clock controls timestamps in tests.

logger.with(context) returns a Logger with a redacted context snapshot; child keys override parent keys without changing the parent. Trace/request IDs are ordinary explicit structured context fields, not ambient task globals. Child loggers share a sink, minimum level, buffer, rotation, and history.

Read-only properties are closed: Bool, level: String, and bufferedBytes: Int. history() -> List[String] returns redacted JSON records oldest-first; filtered levels never enter history. A buffered record enters history when accepted, not when committed to disk.

Lifecycle, buffering, and rotation#

flush() synchronously checks pending writes and fflush; it is not an fsync or durable-storage guarantee. close() flushes and closes the shared sink for all children, is idempotent, and does not close process stderr. Writes, flushes, and deriving new children after close fail. History remains readable after close.

On normal process exit, pending sinks receive a best-effort checked flush. The last garbage-collected child also performs a best-effort flush. Abrupt termination or a power failure can lose buffered records: explicitly flush or close when correctness depends on observing a write error. Sink I/O failures raise a file failure and invalidate the sink, rather than repeatedly retrying a potentially partially written record.

File sinks append to existing files. Before a record would exceed maxBytes, the old file moves to path.1, previous archives advance to path.2 and beyond, and a fresh current file opens. The oldest configured archive is removed. Each physical file is bounded by the configured maximum after new writes; an existing oversized file is rotated before the next record. The logger owns the current path and path.1 through path.maxFiles; do not use these names for unrelated data. Concurrent writers to the same path and cross-process rotation are not supported in this beta. Large media/metrics/tracing exports are not part of this module.

Default privacy#

Console, JSON, file, and history outputs all apply default redaction. Sensitive attribute/context key names (case and separators ignored) cover passwords, tokens, authorization/cookies, secrets/API/private/signing keys, credentials, email/phone/address/birth-date/card/social-security fields. Nested maps/lists are processed; sensitive-key values are replaced completely. Obvious email addresses, Bearer credentials, and password=.../token: ... forms in strings are also redacted. There is no switch to disable this in the beta.

Crash reports and structured test failure reports apply the same default rules to sensitive local/captured variable names, map keys, instance/enum field names, and textual values. Explicit interactive debugger inspection retains its normal raw-value semantics; treat an interactive debug session as privileged access.

This is best-effort detection, not a guarantee that arbitrary prose, opaque strings, source paths, user-defined secret field names, or every locale's PII are safe. Do not log secrets, raw HTTP bodies, or personal data; pass safe identifiers and structured allowlisted attributes. Terminal/source diagnostics and explicit print output are not automatically sanitized by the logger.

Phase-2 metrics, tracing spans, health/profiling/GC statistics, asynchronous export sinks, and OpenTelemetry remain optional follow-up packages.

Authentication, LDAP, and identity#

SIPL provides directory integration through ldap, password and OAuth/OIDC helpers through auth, and an immutable, annotatable Identity type. These are libraries, without new language syntax. Networking uses NetTask and the existing event loop. Cryptography uses the runtime's existing OpenSSL-compatible library; LDAP uses a maintained platform SDK.

This is an initial implementation with a deliberately bounded feature set. Windows and Linux builds and controlled-provider tests are included. A real AD domain, Entra tenant, or Google application needs deployment-specific testing; mock tests do not establish compatibility with every organization policy.

Identity and authorization#

let user: Identity = identity.create({
    "id": "employee-42",
    "provider": "local",
    "name": "Ada Example",
    "email": "ada@example.com",
    "groups": ["operations"],
    "roles": ["sales", "service"],
    "permissions": ["sales.read", "sales.write", "service.read"],
    "expiresAt": floor(time.now().unixSeconds) + 3600,
    "claims": {"department": "Customer services"}
})
assert(identity.hasRole(user, "sales"))
identity.requirePermission(user, "sales.write")

identity.create(values) accepts trusted application data. It does not authenticate arbitrary claims, validate a password, or turn browser-supplied roles into trusted permissions. Verify credentials/tokens first, then obtain roles and permissions from an application-controlled policy or allowed-user table. OIDC identities start with no roles or permissions. CRM-ERP performs this mapping using its own users and role tables.

Properties are id, provider, name, email, groups, roles, permissions, claims, expiresAt (integer Unix seconds), and isAuthenticated. Missing optional strings/lists/claims default to empty; expiry defaults to one hour. id must be a nonempty string. Lists allow 1,024 entries; strings and copied JSON have bounded sizes. Lists and claims returned by properties, and toMap(), are defensive copies.

Methods: hasRole(name), hasGroup(name), hasPermission(name), requirePermission(name), and toMap(). The first four also have module forms: identity.hasRole(user, name), etc. Expired identities never grant permissions. Matching is exact and case-sensitive. A role named admin has no universal meaning in the runtime: the application defines that policy. CRM-ERP defines admin as all present and future application permissions.

Passwords and random values#

PBKDF2 password strings accept 1–1,024 UTF-8 bytes; applications set their own password policy. CRM-ERP requires at least 12 bytes. Hashes are self-describing sipl-pbkdf2-sha256$iterations$salt$key values. Verification accepts 600,000 to 2,000,000 iterations, bounding resource use. Password hashing is synchronous CPU work; directory and HTTP I/O run asynchronously. Rate-limit login attempts and use a dummy hash for unknown usernames, as CRM-ERP does.

Bcrypt is bundled in every build using pinned Openwall crypt_blowfish 1.3, the implementation family used by PHP. The upstream reentrant implementation and per-hash self-test are retained; salts use the existing OpenSSL/LibreSSL CSPRNG. There is no optional SDK, host crypt() dependency, or bcrypt DLL. system.buildFeatures["bcrypt"] records this required support.

let hash = auth.bcryptHash(password) // cost 12, new random salt
let accepted = auth.passwordVerify(password, hash)
// Existing PHP $2y$ hashes can be passed to passwordVerify directly.

New bcrypt passwords must contain 1–72 UTF-8 bytes, without NUL; oversized passwords raise an error rather than being silently truncated. The limit counts encoded bytes, not characters. Verification retains bcrypt's historical 72-byte truncation for compatibility with existing PHP/Apache hashes, with the existing 1,024-byte input bound. Empty passwords can be verified against legacy bcrypt hashes but cannot be generated by bcryptHash.

Bcrypt generation and verification accept costs 4–16; the default for new hashes is 12. Hashes requesting a larger cost, unsupported prefixes such as $2x$, or noncanonical salt/digest encodings return false before derivation. The cost ceiling bounds CPU work. .htpasswd retains its existing cost ceiling of 14. passwordHash() continues generating PBKDF2, and passwordVerify() recognizes both formats, so existing accounts remain usable. Password generation/verification are synchronous CPU work and do not log passwords.

Implementation provenance: Openwall crypt_blowfish, pinned source and checksum.

ui.passwordInput(label, value, style?) reuses native text-input editing, draws one mask character per Unicode code point, and disables copy/cut. It returns the actual password string. Passwords and tokens are ordinary SIPL values and can be exposed by explicit logging or serialization: do not log them. The language does not yet have a redacting Secret value or guaranteed erasure of all GC-managed string copies.

OAuth 2.0 and OpenID Connect#

let client: OidcClient = auth.oidc({
    "issuer": "https://identity.example.com",
    "clientId": "registered-client-id",
    "clientSecret": "load-from-private-settings"
})

async fn start() {
    let metadata = await client.discover()
    let transaction = client.begin("http://127.0.0.1:8082/auth/callback")
    // Store transaction in this browser's SERVER-SIDE session.
    // Redirect the browser to transaction["url"].
}

In the callback, obtain code and state from the query and retrieve that browser's stored transaction:

async fn finish(client: OidcClient, transaction: Map, code: String, state: String) -> Map {
    let tokens = await client.exchange(transaction, code, state)
    let verified: Identity = tokens["identity"]
    // Look up an allowed user by (verified.provider, verified.id).
    // Rotate the application's session and store its local user ID.
    return tokens
}

Configuration fields are issuer, clientId, optional clientSecret, caFile, timeout (100–30,000 ms, default 10,000), and allowLoopbackHttp (false). Unknown fields fail. Production endpoints require HTTPS with chain and hostname validation, no URL credentials or fragments, and no automatic redirects. The explicit test option permits HTTP only for literal loopback hosts. Callback URLs may use HTTP on loopback for local apps.

discover() returns NetTask[Map], reads issuer/.well-known/openid-configuration, verifies exact issuer equality, and checks the authorization, token, and JWKS endpoints. It snapshots metadata internally; editing the returned Map cannot redirect credential requests.

Authorization requests always use S256 PKCE, without plain or non-PKCE fallback. When discovery lists PKCE methods, it must include S256 to begin a sign-in. Providers such as Microsoft Entra omit this discovery extension despite supporting S256; an absent field is accepted and S256 is still required in every request. Configure a provider with documented S256 support. Machine client-credentials grants do not depend on that authorization-flow capability. See Microsoft's PKCE flow.

begin(redirectUri, scopes = "openid profile email") returns a transaction with authorization URL, random state/nonce/verifier, redirect URI, and expiry. Keep the whole transaction server-side. It expires after ten minutes; at most 128 pending transactions are retained per client. Scopes must begin with openid. Redirect URIs must also be registered exactly with the provider.

exchange(transaction, code, returnedState) returns a NetTask. It compares state against the session transaction and the client's private transaction store, consumes the transaction once, sends the PKCE verifier, exchanges the code, fetches the provider's JWKS over validated HTTPS, verifies the ID token, and returns:

• accessToken, tokenType: "Bearer", integer expiresAt;

• optional refreshToken and scope;

• a verified identity with issuer and subject, name/email, claims, and expiry.

Returned tokens are intended for server-side use. Client secrets use the client_secret_post token-endpoint method; public PKCE clients omit the secret. The implementation does not use password or implicit grants.

refresh(refreshToken, originalIdentity?) returns a NetTask. When an ID token is returned, the original Identity is required; its issuer, subject, and original nonce must match. OAuth-only refreshes may return access tokens without an identity. Refresh-token rotation is returned to the application; persist the replacement securely when using durable tokens.

clientCredentials(scopes = "") requires a client secret and returns a NetTask with a machine token, never a fabricated user Identity. These helpers share SIPL's bounded HTTP connection pool and cancellation behavior. Buffered provider JSON is capped at 256 KiB. Client configuration stays private inside the native object; it has no public secret property.

Microsoft Entra ID and Google#

let microsoft: OidcClient = auth.microsoft({
    "tenant": "00000000-0000-0000-0000-000000000000",
    "clientId": "registered-client-id"
})
let google: OidcClient = auth.google({"clientId": "registered-client-id"})

Both return the same OidcClient API. Microsoft requires the explicit tenant GUID and uses its v2.0 issuer; shared tenant aliases and domain aliases are not accepted. Google pins https://accounts.google.com. Other clouds, specialized policies, and standards-compliant providers use auth.oidc with their exact issuer. Register your application, scopes, redirect URL, and appropriate public/confidential client type in the provider's administration interface. No cloud credentials or example tenant IDs are shipped.

JWT verification boundary#

auth.verifyJWT(token, jwks, options) returns verified claims. Required options are issuer and audience; optional values are nonce, clockSkew (0–120 seconds, default 30), accessToken, and code. It accepts only RS256 with 2048–4096-bit RSA keys or ES256 with P-256 keys. It verifies signatures before trusting claims, then checks subject, exact issuer, audience and authorized party, integer expiration/issued-at/not-before dates, expected nonce, and at_hash/c_hash when present. Multiple audiences require matching azp.

The verifier rejects unsigned/HMAC tokens, ambiguous keys, duplicate JSON keys, unsupported critical headers and remote key URLs. JWT size is at most 64 KiB, with at most 64 JWKs. It never fetches a key URL supplied by a token. Standalone callers must obtain JWKS from a trusted issuer, and must not use this ID-token helper as a generic authorization policy for arbitrary access tokens.

LDAP and Active Directory#

let directory: LdapDirectory = ldap.connect({
    "host": "dc.example.com",
    "port": 636,
    "baseDN": "DC=example,DC=com",
    "userAttribute": "sAMAccountName",
    "bindDN": "service@example.com",
    "password": "load-from-private-settings"
})
async fn start() {
    let valid = await directory.authenticate("ada@example.com", "entered-password")
    if valid {
        let user = await directory.findUser("ada")
        let group = await directory.findGroup("Sales")
        let member = await directory.isMember(user["dn"], group["dn"])
        print(member)
    }
    directory.close()
}

ldap.connect(options) configures a directory without network I/O. Options:

All connections require validated TLS. StartTLS never falls back to plaintext. Windows also rejects negotiated protocols below TLS 1.2 and ciphers below 128 bits. OpenLDAP requires a valid chain and matching subject alternative name, with TLS 1.2 minimum. Referral chasing is disabled so credentials are not forwarded to another server. Windows caFile is rejected explicitly: install the directory's issuing CA through Windows trust management.

authenticate(username, password) validates credentials on a separate connection, returning NetTask[Bool]: invalid credentials return false; transport/TLS/policy errors raise a failure. Empty usernames/passwords are rejected before I/O. It does not replace the service identity used for search. bind(username, password) validates and replaces that directory's service credentials only after success.

findUser(name) and findGroup(name) return NetTask[Map|None]; an ambiguous lookup fails. Returned entries contain dn and attributes, a Map of attribute names to lists of values. Valid non-NUL UTF-8 values become strings; binary values become Bytes. Helpers return common user attributes and memberOf, objectGUID, and objectSid when supplied by the directory.

isMember(userDN, groupDN, options?) searches the group entry's member attribute. Set attribute for uniqueMember or a directory-specific schema. Set nestedAD: true for AD's transitive matching rule 1.2.840.113556.1.4.1941. This is an explicit AD feature, not a generic recursive group walker. POSIX primary groups, AD tokenGroups expansion, ranged member attributes and forest-wide authorization need application-specific handling.

search(baseDN, filter, options?) returns NetTask[Map] containing entries, a Bytes cookie, and estimatedTotal. Options are scope (base, one, sub; default sub), attributes (1–64 names), pageSize (1–500; default 100), and a returned cookie (at most 4 KiB). An empty cookie finishes the paged search. Reuse the same directory and query for subsequent pages; the native connection stays open for server paging state. Each extracted page is bounded to 1 MiB. Size-limit errors fail rather than silently returning partial search results. Paged searches require server support.

let safe = ldap.escapeFilter(username)
// Escape VALUES, never an entire constructed filter or an attribute name.
let task = directory.search(baseDN, "(uid={safe})", {"pageSize": 100})

ldap.escapeFilter implements RFC 4515 value escaping; ldap.escapeDN escapes DN components under RFC 4514. Helpers validate attribute names and escape lookup values automatically. A raw search filter is deliberately an advanced, trusted application input, not a safe place to concatenate browser input.

LDAP jobs use native worker threads, capped at eight jobs and 64 directories. SIPL values remain on the VM thread. Await a directory operation before reusing that same directory; independent directories can run concurrently. Cancelling a NetTask ends the await, while an already executing SDK operation finishes under its configured deadline before native resources are reclaimed. close() releases the connection once no directory operation is running.

Windows integration and build support#

On Windows, ldap.available and ldap.integratedAvailable are true. The Windows SDK's WLDAP32/SSPI implementation performs directory authentication; integrated: true uses the current process account with negotiate semantics. OS/domain policy controls Kerberos versus permitted fallback. This does not add HTTP Negotiate/NTLM middleware, impersonation, delegation, or a full PrincipalContext replacement. Validate your domain's channel-binding and signing requirements in the intended deployment.

On POSIX desktop builds, OpenLDAP headers and libraries are required; Ubuntu's SDK package is libldap-dev. Make fails if they are absent, and rejects SIPL_LDAP=0. Windows uses its system SDK. Runtime directory configuration and a reachable directory server are only needed when an application uses LDAP.

Web sessions and secret storage#

Use web.sessions, rotate with web.sessionRotate after authentication, and store only a local user identifier, authorization version and expiry in the server-side session. Resolve current permissions from trusted storage on every protected operation. Sessions should be Secure and HttpOnly in production. OIDC redirect callbacks require SameSite=Lax rather than Strict, with state and nonce validation; keep ordinary API writes protected against foreign origins. CRM-ERP includes these integration patterns for its local, LDAP and OIDC logins.

Windows has secrets.load/save backed by current-account DPAPI. The existing API deliberately fails on unsupported platforms instead of storing plaintext. For portable deployments, use explicitly allowlisted environment variables provided by a deployment secret manager. Tokens default to process memory; the runtime does not automatically write them to a configuration file or add them to logs. A general POSIX keychain and redacting Secret type remain roadmap work, rather than an insecure file fallback.

Verification and references#

sipl api auth --json
sipl api ldap --json
sipl api Identity --json
sipl api --verify
sipl test tests/test_auth.sipl --timeout 60000
python3 tests/identity_provider_tests.py path/to/sipl
python3 tests/identity_ldap_tests.py path/to/sipl path/to/isolated-openldap-root

The HTTPS fixture signs its own RS256/ES256 tokens and tests PKCE, state replay, nonce and claim mismatches, invalid signatures, ambiguous keys, duplicate JSON, refresh, client credentials, issuer mismatch and certificate rejection. The OpenLDAP fixture tests real LDAPS/StartTLS, bind validation, searches, membership, binary attributes, filter injection and pagination. Fixtures run normally and with GC stress, using temporary data without cloud accounts or installed services. Both drivers also launch an isolated CRM-ERP copy to verify allowed-user mapping, additive roles, denial of unauthorized operations, and session revocation through its real HTTP handlers. The OIDC driver exercises the browser callback, state/replay checks and session rotation. The LDAP driver verifies that choosing LDAP cannot fall back to a local password. Both standalone examples run against these fixtures as well.

Protocol references: OAuth security BCP / RFC 9700, OIDC Core, PKCE / RFC 7636, OpenLDAP TLS, Microsoft LDAP SSL.

Images#

The first-party image module provides typed, owned Image values on Windows, Linux and headless runtimes. It does not require graphics hardware. PNG/JPEG are built in; WebP is included by default and can be disabled at build time. PDF generation is available through the pdf module; video and animated images have separate APIs. Office documents are separate future work.

fn start() {
    let original: Image = image.open("photo.jpeg", {"maxPixels": 4000000})
    let thumbnail = original.resize(320, 240)
    thumbnail.save("thumbnail.png", "png")
    thumbnail.close()
    original.close()
}

Use sipl api image --json and sipl api Image --json for the exact signatures, effects and option contracts. See the runnable recipe.

Ownership and formats#

image.open(path, options?) reads a bounded regular file. image.decode(bytes, options?) decodes existing Bytes. The default pixel format is RGBA: row-major, tightly packed, unsigned 8-bit channels, straight (unpremultiplied) alpha. Explicit "format": "RGB" requests three channels. Unknown formats fail; there is no automatic byte-order guessing or reinterpretation.

image.create(width, height, options?) creates zeroed pixels (transparent black for RGBA, black for RGB). Options include format and a color List containing exactly three or four Ints from 0 to 255. image.fromPixels(width, height, format, bytes) requires an exact byte count and copies the source buffer.

Read-only properties are width, height, format and closed. pixels() returns a detached copy as Bytes; pixel(x, y) returns a List of channels. Neither exposes a dangling view of native memory. crop, resize and convert return independent Images. convert("RGB") explicitly drops alpha; convert("RGBA") adds opaque alpha to RGB pixels. All pixel operations fail after close(). Close is idempotent; dimensions/format remain inspectable. Garbage collection is a fallback, not a substitute for closing large images.

Geometry, drawing and compositing#

crop(x, y, width, height) requires a positive rectangle entirely inside the image. resize(width, height, filter?) supports "nearest" and the default "bilinear". Bilinear interpolation uses premultiplied colors internally to avoid transparent-edge halos, then returns straight-alpha bytes. It interpolates encoded channel values; linear-light/HDR resizing is not promised.

setPixel(x, y, color), fill(color), rect(x, y, width, height, color) and line(x1, y1, x2, y2, color) mutate pixels. Drawing replaces channels; it does not implicitly alpha blend. Coordinates and rectangles must be inside the image; line drawing is deterministic, one-pixel-wide and not antialiased.

clone() makes an independent pixel copy for edit history. stroke(x1, y1, x2, y2, size, color, mode) paints circular brush stamps along a line, clipped to the image; mode is "paint" (source-over alpha) or "erase" (reduces alpha). Size is 1..256. floodFill(x, y, color, tolerance) replaces the contiguous region matching the starting pixel within 0..255 per channel and returns the number of changed pixels. shape(kind, x1, y1, x2, y2, thickness, color, filled) draws a line, rectangle, or ellipse. adjust(kind, amount?) changes RGB channels in place while preserving alpha; kinds are invert, grayscale, sepia, brightness, and contrast, with amount -100..100 for the last two. flip("horizontal"|"vertical") changes pixels in place. rotate90(clockwise) returns an independent image with swapped dimensions.

composite(source, x, y, opacity?) performs source-over alpha blending with finite opacity in 0..1. It clips to the destination bounds, supports RGB and RGBA sources/targets, and safely snapshots self-compositing before mutation. Positions must be within -16384..16384. No source pixels are changed.

Drawing in the UI and Toolkit#

ui.setIcon(picture) sets an open window's native icon on Windows and Linux/XWayland. Use an RGBA Image no larger than 256 x 256 pixels. The platform copies the pixels, so the Image can be closed immediately after the call.

Call ui.drawImage(x, y, width, height, picture) inside a UI frame to draw an open Image into a rectangle in logical pixels. It supports RGB and RGBA, respects ui.clip(), applies the display scale, and blends transparency over the existing canvas. Scaling uses bilinear interpolation with premultiplied colors to avoid transparent-edge halos. Coordinates must be finite and sizes nonnegative; zero-sized and offscreen rectangles draw nothing. The operation borrows pixels only for the call and neither changes nor closes the image. Closed images fail clearly.

fn start() {
    let picture = image.open("photo.png")
    ui.open("Image preview", 640, 480)
    while ui.frame() {
        ui.drawImage(20, 20, 600, 400, picture)
    }
    picture.close()
}

Toolkit opens .png, .jpg, .jpeg, and .webp files from the Explorer or Open dialog in read-only image tabs. Images fit the available area while preserving aspect ratio; small images retain their native size. A checkerboard shows transparency, and the footer reports dimensions and scale. The tab's Reload button reads changes from disk. Unsupported, corrupt, or oversized images show a preview error instead of opening their binary data as text. Closing an image tab releases its pixel storage. Preview behavior is written in toolkit/image_preview.sipl; the canvas primitive works without Vulkan.

Encoding and metadata#

encode(format, options?) returns Bytes; save(path, format, options?) writes those bytes using an explicit "png", "jpeg" or "webp" codec. Extensions do not silently choose an encoder. PNG is lossless and does not accept quality. JPEG uses quality 1..100, default 90; it cannot encode alpha and rejects nonopaque RGBA unless the caller explicitly converts or composites onto a background. JPEG cannot be made lossless with a flag.

WebP defaults to lossless encoding, preserving alpha and transparent pixels' hidden RGB values. {"lossless": false, "quality": 90} selects lossy WebP. The available backends are always inspectable with image.codecs(), which returns {"png": true, "jpeg": true, "webp": Bool}. An absent backend fails clearly; it does not encode a different format. WebP is required in desktop builds; feature-off switches are rejected. Vendor-backed builds do not download anything.

EXIF orientations 1..8 in PNG, JPEG and WebP are applied by default, including mirroring and width/height exchange. Decode option applyOrientation: false preserves source coordinates. Orientation parsing only reads bounded TIFF IFD0 fields; malformed EXIF, invalid tag types/counts and invalid orientations fail even when orientation application is disabled. Encoding emits fresh pixel data without EXIF, GPS, comments, ICC profiles or other source metadata. ICC color management is not implemented; channels are used as supplied. High-bit-depth PNG and HDR are explicitly unsupported instead of silently reducing precision. Animated PNG/WebP are rejected instead of silently returning one frame.

Native editing operations#

Image.effect(kind, options?) stages its output before committing pixels. Validation or scratch-allocation failure leaves the original unchanged. Kinds are saturation, hue, gamma, exposure, threshold, posterize, solarize, temperature, tint, vibrance, levels, autoLevels, gaussianBlur, sharpen, unsharpMask, findEdges, emboss, pixelate, noise, vignette, motionBlur, highPass and colorBalance.

Options are numeric amount (-100..100; default 25), integer radius (1..32; default 3), levels (2..256; default 6), seed (0..2147483647; default 1), angle (-180..180 degrees), gamma (0.1..10; default 1), black (0..254), white (1..255; black must be below white), and red, green, blue (-100..100 offsets). Unknown keys and kinds fail explicitly. Hue amount spans -180..180 degrees; exposure amount 25 is one stop; threshold/solarize map -100..100 onto 0..255. Hue is an RGB matrix rotation; these are 8-bit RGB adjustments, without ICC/HSL color management.

Gaussian/motion blur and mosaic use premultiplied alpha to avoid hidden-color halos. Other effects retain original alpha. Gaussian uses separable sampling, edge clamping and at most 64 MiB of float scratch plus 16 MiB of staged output at the image pixel ceiling. Run large operations in a worker to keep UI input responsive. Noise uses a local deterministic seed.

composite(source,x,y,opacity?,blend?) adds optional normal, multiply, screen, overlay, darken, lighten, difference, exclusion, hardLight, softLight, colorDodge and colorBurn blending. Existing four-argument Normal compositing keeps its previous behavior. Blend colors use source-over alpha; RGB/RGBA formats can be mixed explicitly as before. mix(source, amount) interpolates premultiplied pixels including alpha, requires matching dimensions/formats and accepts amount 0..1.

gradient(x1,y1,x2,y2,from,to,kind) paints a linear or radial color gradient; colors match the image's channel count, and a zero-length vector does nothing. text(text,x,y,color,size?) rasterizes up to 4096 UTF-8 bytes using the shared UI font renderer, with size 8..256 (default 24), preserving surrounding transparency. Text is a raster operation, not an editable text object.

Untrusted-media limits#

image.limits() exposes the hard ceilings. Each image is limited to 16384 per dimension, 4,194,304 pixels and 16 MiB of pixel bytes. Encoded input and output are each limited to 16 MiB. Decode options maxInputBytes, maxWidth, maxHeight and maxPixels can lower these limits but cannot raise them. Create supports the same dimension/pixel limits. Encode option maxOutputBytes lowers the output limit. All limits are positive Ints; unknown option keys fail.

PNG chunk lengths, CRCs, zlib checksum and exact decoded scanline sizes, JPEG segment/frame/scan boundaries, and WebP RIFF chunks/features are validated before pixel decoding. Dimensions and checked pixel counts are validated before allocating destination pixels. stb PNG/JPEG decoding/encoding uses a 64 MiB temporary-allocation budget. WebP uses checked upstream allocation and bounded dimensions, single-threaded encoding and a bounded output callback; its internal scratch allocations are not governed by the stb budget. Operations are synchronous and bounded by these resource limits, not cancellable background tasks or hostile-code sandboxing. Applications should use smaller limits for thumbnails and other untrusted uploads and close resources promptly.

Image errors use SIPL's normal runtime failure handling; file failures retain the file kind. Native codec buffers never contain SIPL references. Tests cover all EXIF transforms/endian variants, exact PNG/lossless WebP, JPEG tolerances, draw/resize/composite behavior, truncated input, malformed metadata, oversized headers, decompression budgets, forged JPEG termination markers and packaged programs in normal/GC-stress modes.

Dependencies are pinned in stb's image pin and libwebp's pin, with upstream license and patent notices kept alongside the sources.

UI fonts#

SIPL uses a smooth embedded monospace font by default. Labels, controls, drawn text, and the Toolkit editor share the same measured character grid. Packaged applications include the default font, so users do not need to install a font or place a font file beside the executable.

Glyphs are cached by Unicode scalar and display size, with at most 512 entries and 8 MiB of cached mask data. Changing fonts clears that cache.

Toolkit font choice#

Choose View → Font → Smooth for scalable text or View → Font → Pixel (5x7) for SIPL's compact pixel style. The selection applies to the current Toolkit session. Smooth text is selected when Toolkit starts.

Font APIs#

ui.setFont(path?: String or None, pixelSize?: Int) -> None selects a monospace TrueType/OpenType font. Omit the path, or pass none, to use the embedded font. Omit the size to use 16 logical pixels; accepted sizes are 8..32. Font files must be nonempty and at most 16 MiB. Paths must be nonempty Strings without NUL bytes and at most 4096 bytes. The font must have a Unicode character map and a fixed character width. Proportional fonts fail because the editor, text-entry controls, and selection calculations share a monospace grid.

A failed font change leaves the current font and metrics unchanged. A successful change clears text caches and entry scroll offsets and requests a redraw. Select a font before opening or drawing the UI.

fn start() {
    ui.setFont("assets/MyMonoFont.ttf", 18)
    ui.open("Scalable text", 640, 360)
    while ui.frame() {
        ui.heading("Custom UI font")
        ui.text("Readable text at your display's resolution")
        ui.button("Continue")
    }
}

ui.usePixelFont() -> None selects the built-in 5x7 renderer. It clears text caches and entry scroll offsets and requests a redraw. Normal text uses 12-pixel cells and 18-pixel lines. Call ui.setFont() to switch back to the embedded smooth font and default size.

Metrics and Unicode#

Use these metrics for custom layouts, caret positions, and selections instead of assuming a fixed character width.

UTF-8 code points supported by the selected font render directly. Missing glyphs use the replacement glyph or ?. The fixed grid advances once per scalar, including combining characters. Complex-script shaping, bidirectional layout, grapheme-aware editing, automatic font fallback, and color emoji are not currently supported.

Web Server#

Server-rendered .sipl.html pages compile to restricted SIPL bytecode and use context-aware output rules. See SIPL Pages for the <^ ... ^> syntax, safe data boundary, compilation model, and traffic plan.

Start#

fn start() {
    web.serve()
}

Defaults: the html folder relative to the working directory, port 8080, and the loopback address 127.0.0.1. The root must already exist.

web.serve("public", 9000)
web.serve(none, 9000)        // keep the default root

serve blocks until stopped or interrupted. Port 0 selects an available ephemeral port. web.port reports the bound port; web.isOpen reports whether the server is running. A handler can call web.stop() to stop accepting requests and drain queued responses. Ctrl+C terminates the process.

Registered routes run before the static fallback. Therefore a web.get("/", home) route can render pages/index.sipl.html even though the static server still uses html/index.html when no / route returns a response. The page filename is not discovered by the static server; web.render selects it.

Try the bundled example from the repository root:

.\build.bat
.\build\sipl.exe run examples\web.sipl

The bundled example listens at http://127.0.0.1:8082/.

Listener address and optional native window#

The default listener remains loopback. To select another IPv4 interface, pass {"host": "0.0.0.0"} (all interfaces) or a specific literal IPv4 address in the fourth argument. Hostnames and IPv6 listeners are not accepted. Opening an interface changes network exposure; configure HTTPS and application access as appropriate. TLS itself does not change the selected address.

A Windows WebView2 build can own a desktop window with {"window": {"title": "My app", "width": 1280, "height": 850}}. SIPL opens the window after binding the listener, serves requests on the VM thread, and stops the listener when the window closes. web.stop() also closes an owned window. No window work occurs unless this option is supplied. Window options also include profile (engine data path), url (override the default actual loopback listener URL), and closeAfterMs (0 by default, up to 120000). HTTPS or loopback HTTP is required for a web window, and navigation is confined to its initial origin. Unsupported builds fail and close the listener. See the meeting guide for the native API and one-call meeting library.

HTTPS#

web.serve(root?, port?, handler?, options?) accepts certFile and keyFile in its fourth argument. Supply both to serve HTTPS directly on Windows and Linux:

fn start() {
    web.get("/health", fn(request) => "healthy")
    web.serve("html", 8443, none, {
        "certFile": "certs/server.pem",
        "keyFile": "certs/server.key",
    })
}

Open https://127.0.0.1:8443/. The existing html directory is the public static root; the certs directory is separate. Certificate/key paths are relative to the process working directory, or may be absolute. certFile is a PEM certificate chain with the server certificate first, followed by any intermediate certificates. keyFile is its matching, unencrypted PEM private key. Use a certificate valid for the hostname or IP clients connect to, and keep private keys outside the public static root with restricted file access. Encrypted key files are rejected without an interactive password prompt.

TLS 1.2 and TLS 1.3 are supported. Older protocols, TLS compression, and renegotiation are disabled. The listener defaults to 127.0.0.1; HTTPS does not change the address selected by host. It accepts encrypted traffic only. TLS handshake and encrypted read/write/shutdown operations are nonblocking; a stalled handshake expires after 10 seconds and does not hold up other connections. The HTTP routes, Pages, static fallback, sessions, keep-alive and request limits work over either transport. web.isSecure is true while an HTTPS listener is active and false after it stops. The startup message prints the correct http:// or https:// URL, including the actual port when using port 0.

Omit both options (or pass an empty Map / none) to retain HTTP. Supplying only one, an invalid path, an unreadable certificate, a malformed/encrypted key, or a key that does not match the certificate fails startup and leaves no listener open. Unknown options are errors. There is no automatic downgrade to HTTP. Certificates are loaded at startup; stop and restart the server to renew them. Certificate issuance, browser trust installation, ACME renewal, mutual TLS, and HTTPS redirects are not managed by this API.

For a temporary local development certificate, use an installed OpenSSL CLI:

mkdir certs
openssl req -x509 -newkey rsa:2048 -nodes -days 7 \
  -subj "/CN=localhost" -addext "subjectAltName=DNS:localhost,IP:127.0.0.1" \
  -out certs/server.pem -keyout certs/server.key

The generated certificate is self-signed; browsers will warn until you explicitly trust your development certificate. Public deployments should use a certificate issued by a trusted authority. Production session cookies should keep their default secure: true setting when using direct HTTPS or a TLS reverse proxy. The runtime uses its existing OpenSSL-compatible TLS dependency: static LibreSSL on MSVC Windows builds and the system TLS libraries on Linux.

Optional .htaccess compatibility#

Apache 2.4 .htaccess compatibility is separately opt-in through {"htaccess": true}. It applies to native SIPL routes and .sipl.html Pages, static files, and optional FastCGI applications. It can be selected per named host using virtualHosts. PHP is not required. With compatibility enabled, directory indexes default to index.sipl.html, index.html, then the optional FastCGI index. Native Page indexes can resolve through exact registered routes while their template source remains private. See Apache compatibility and migration for supported directives, security behavior, limits, and a native SIPL example.

FastCGI applications#

SIPL includes a native C FastCGI client with no additional libraries. Enable it through web.serve to send requests to an independently running PHP-FPM, php-cgi, or another FastCGI responder:

fn start() {
    web.serve("public", 8080, none, {
        "fastcgi": {"host": "127.0.0.1", "port": 9000, "timeout": 30000},
    })
}

For a local PHP example, start php-cgi -d cgi.fix_pathinfo=0 -b 127.0.0.1:9000 in a separate terminal, then run sipl run examples/web_fastcgi.sipl and visit http://127.0.0.1:8080/. SIPL connects to the application; it does not launch or manage its processes. The backend must see the same absolute script paths as SIPL and have permission to read them.

The fastcgi Map accepts these options. Unknown names and invalid values fail before the listener opens.

For PHP-FPM over a Unix socket, use {"fastcgi": {"socket": "/run/php/php-fpm.sock"}}, substituting the configured pool socket path. Other languages can select their own suffix and index, such as {"fastcgi": {"port": 9001, "extension": ".py", "index": "app.py"}}. certFile and keyFile can be supplied alongside fastcgi for direct HTTPS.

Middleware, routes, and the optional SIPL handler run first. When they return none, a matching script runs through FastCGI. With htaccess disabled, a directory URL tries the configured script index first, then falls back to static index.html if that script does not exist. A missing or inaccessible matching script returns 404 and never falls through to serving its source. Only exact script filenames are supported: /index.php/extra does not invoke PHP or infer PATH_INFO. Omitting fastcgi on a later web.serve call disables it.

CGI parameters include the canonical SCRIPT_FILENAME and DOCUMENT_ROOT, SCRIPT_NAME, raw REQUEST_URI and QUERY_STRING, REQUEST_METHOD, CONTENT_TYPE, CONTENT_LENGTH, server/peer addresses and ports, SERVER_PROTOCOL, HTTPS, REQUEST_SCHEME, PHP's REDIRECT_STATUS=200, and request headers as HTTP_*. PATH_INFO is empty. Repeated request headers are joined with commas (cookies with semicolons). Hop-by-hop headers, Proxy, and header names containing underscores are excluded to avoid CGI variable aliases and HTTP_PROXY injection. The server uses its bound address for SERVER_NAME; the supplied Host header remains available as HTTP_HOST.

Scripts use the static server's confined file-opening checks. Traversal and hidden paths are rejected; Unix refuses symlinks, and Windows checks the final opened path against the canonical root. Keep the script tree writable only by trusted administrators: a separate FastCGI process reopens the checked filename, so SIPL cannot make that second open atomic with its own file check.

The client uses FastCGI v1, the responder role, and one request per connection. It implements 8-byte records, 1-byte/4-byte parameter lengths, 65535-byte stream chunks, padding, explicit stream terminators, interleaved stdout/stderr, and END_REQUEST. For PHP compatibility, END_REQUEST also closes output streams when the backend omits their empty terminator records. Data after an explicit stream terminator is rejected. Multiplexing, pooling, authorizer/filter roles, and application process management are outside this implementation. See the FastCGI specification and PHP's FastCGI implementation.

Backend sockets use the server's nonblocking poll loop, allowing other clients to progress during an upload or slow response. Request and response bodies are limited to 1 MiB, CGI response headers to 16 KiB and 128 fields. Responses are buffered until completion. Status, Location, binary bodies, repeated Set-Cookie, HEAD, and bodyless 204/304 responses are handled; SIPL computes HTTP framing and removes hop-by-hop response headers. Backend connection, protocol, size, and malformed-header failures return 502; overload returns 503; the deadline returns 504. Backend stderr is logged with escaped controls and a 16 KiB capture limit, never inserted into the HTTP body. Nonzero application exit status is logged while preserving a valid application response.

SIPL Pages#

Keep executable pages in a private folder separate from the static root:

fn home(request) {
    return web.render("index.sipl.html", {
        "title": "My SIPL page",
    })
}

fn start() {
    web.pages("pages", {"reload": true})
    web.get("/", home)
    web.serve("html", 8082)
}

web.pages(root, options?) validates and compiles every .sipl.html file. reload defaults to false; enable it during development to atomically load valid edits while retaining the last valid compiled page after an error. web.render(literalPath, model) returns a standard response Map containing status 200, an HTML content type, and the rendered body. The model must be an acyclic tree of page-safe scalar, List, and Map values. Render paths are literal, relative paths below the configured root.

Page source is never a static asset: the server returns 403 for any static request ending in .sipl.html, even if the static root is misconfigured to contain one. Use sipl page-check pages or Toolkit Scan to validate pages.

SIPL Backend#

The optional third argument is a fallback function called when no registered route matches:

fn handle(request) {
    if request["path"] == "/api/hello" {
        return "Hello from SIPL"
    }
    if request["path"] == "/api/echo" and request["method"] == "POST" {
        return {
            "status": 201,
            "body": request["body"],
            "headers": {"Content-Type": "text/plain; charset=utf-8"},
        }
    }
    return none
}

fn start() {
    web.serve("html", 8080, handle)
}

Request map fields:

A handler returns a String for a 200 text response, a Map with a String or Bytes body and optional integer status and Map headers, or none to use built-in static-file serving. Decode request text explicitly with request["body"].decodeUtf8() when expecting UTF-8. A Bytes response defaults to application/octet-stream unless headers specify another content type. The server owns Content-Length, Transfer-Encoding, and Connection; supplying them in a response produces 500.

Uncaught handler failures become generic 500 responses; details are written to stderr, not exposed to clients. The server remains available. Startup errors are catchable with SIPL's try/fail blocks.

Routes And Middleware#

Register routes before calling web.serve(). web.route(method, pattern, handler) accepts GET, HEAD, POST, PUT, PATCH, DELETE, or OPTIONS; web.get(pattern, handler) and web.post(pattern, handler) are shortcuts. Patterns match the decoded path exactly, and :name captures one path segment in request["params"]:

fn showUser(request) {
    let id = request["params"]["id"]
    return "User {id}"
}

fn start() {
    web.get("/users/:id", showUser)
    web.serve()
}

Use web.use(fn(request, continueRequest) { ... }) to register middleware. Calling continueRequest() runs the remaining middleware and then the matching route (or fallback handler). Middleware may return a response early, or modify the downstream response after continueRequest() returns. It may call the continuation at most once. A middleware that returns none without calling the continuation passes the request onward.

Secure User Sessions#

Call web.sessions() before web.serve() to enable server-side sessions:

fn login(request) {
    // Rotate after authentication so an anonymous ID cannot be fixed as the
    // authenticated session.
    web.sessionRotate(request)
    web.sessionSet(request, "userId", 42)
    return "Logged in"
}

fn dashboard(request) {
    let userId = web.sessionGet(request, "userId")
    if userId == none {
        return {
            "status": 302,
            "headers": {"Location": "/login"},
            "body": "",
        }
    }
    return "Welcome user {userId}"
}

fn logout(request) {
    web.sessionDestroy(request)
    return "Logged out"
}

fn start() {
    web.sessions()
    web.post("/login", login)
    web.get("/dashboard", dashboard)
    web.post("/logout", logout)
    web.serve()
}

The cookie contains only a cryptographically random 256-bit opaque ID. Session values stay in server memory and may be any SIPL value. IDs come from the operating system's secure random source, are replaced by sessionRotate, and are invalidated immediately by sessionDestroy.

Available operations, all of which require the current handler request:

web.sessions(options?) accepts:

Cookies are host-only and always use Path=/ and HttpOnly. Secure is on by default, SameSite=None requires it, and the __Secure- and __Host- cookie prefixes cannot be used with secure: false. Duplicate, malformed, unknown, rotated, expired, and destroyed session IDs are rejected and cleared. Access refreshes idle expiry but never extends the absolute lifetime. When capacity is full, the least recently seen session is evicted.

For plain-HTTP local development only, use a non-prefixed cookie with {"secure": false}. Production should retain secure: true over direct HTTPS or behind a TLS reverse proxy. Rotate the session immediately after successful authentication, use POST for logout, and add CSRF tokens to sensitive cookie-authenticated forms; SameSite is useful defense in depth, not a complete CSRF strategy.

Sessions survive web.stop() followed by another web.serve() in the same process. They are intentionally in-memory: they do not survive process exit and are not shared across server processes. Persistent or distributed session stores remain application infrastructure rather than part of this embedded server.

Scope And Limits#

• HTTP/1.0 and HTTP/1.1, with HTTP/1.1 keep-alive and ordered pipelining.

• Static GET and HEAD; trailing-slash paths use index.html. No directory listings.

• 128 simultaneous connections; at most 100 requests per connection.

• 16 KiB request headers, 64 headers, 1 MiB request and dynamic-response bodies.

• 10-second request deadline and 30-second response deadline; slow clients do not block socket I/O for others.

• Close-marked responses use a bounded one-second nonblocking input drain, avoiding TCP resets when rejected requests have unread bytes.

• Static files stream in 64 KiB chunks instead of being loaded into VM Strings.

• Percent-decoding is validated. Hidden and traversal path components, Windows separators/device-path syntax, and static .sipl files are rejected.

• Windows checks the final opened file handle's path against the canonical root before reading. Unix opens path components relative to the root directory descriptor without following symlinks.

• Chunked request bodies and Expect are deliberately unsupported and rejected with closed connections. Clients must provide Content-Length for bodies.

• Optional direct HTTPS with PEM certificate/key files, TLS 1.2 or newer, and a 10-second handshake deadline.

• No HTTP/2, HTTP/3, compression, ranges, cache validators, persistent/distributed session store, or database layer.

This is a bounded embedded HTTP/HTTPS engine. For public deployment, an established reverse proxy can provide rate limiting, certificate renewal, and additional request policy. Review authentication, logging, process isolation, and workload limits for your application. SIPL handlers and disk reads execute synchronously: blocking backend work or cold or slow storage can delay other requests. Handlers run in request order.

Toolkit workflow#

Toolkit recognizes .sipl.html files as SIPL Pages and highlights HTML and SIPL tags together. Use Scan to validate a page. Run the .sipl controller that calls web.pages() and web.render() to view it through the server.

Gather and the meeting library#

Gather is a standalone SIPL meeting application with a browser frontend and a Windows desktop frontend. It includes camera/audio calls, screen/window/tab sharing, chat, raised hands, invitations, a host-controlled waiting room, participant removal, file downloads and synchronized PDF presentation controls. It does not connect to Microsoft Teams.

The reusable library lives in libraries/meeting. Applications activate the complete service with one call:

use libraries/meeting/meeting

fn start() {
    meeting.app({"port": 8083})
}

This example is for an entry file in the repository root. Adjust the relative import for another directory, or copy libraries/meeting beside your app and use meeting/meeting. No registry, Node, Python, browser automation or launcher script is used by the application. The browser source and styles are embedded in an imported SIPL module and included in packaged executables.

meeting.serve({"port": 8083}) starts the same service for browsers without opening a desktop window. meeting.app({"url": "https://meet.example.com"}) opens a desktop client for an existing service. Remote mode accepts only url and the window options below; it does not start another server.

The serving call blocks until the server stops. In local desktop mode SIPL starts the listener before opening the window, runs the engine on its own native GUI thread, and stops the listener when the window closes. Application authors do not implement a signaling protocol or manage a second process. One server/window can be owned by a VM at a time.

Configuration#

Gather reads examples/projects/Gather/config.json from beside its entry file or executable. A packaged executable can run without this file using the defaults. Certificate paths in that file are resolved relative to the file, independently of the working directory. Library callers pass paths directly.

Unknown settings fail. Non-loopback listeners require TLS and a createKey of at least 16 characters. An explicit public origin also requires that key, including behind a reverse proxy. A public origin has no trailing slash, path, query, fragment or user information. Enter the actual origin clients use; the library rejects unrelated HTTP Host and Origin headers.

SIPL_GATHER_CREATE_KEY overrides the create key. The explicitly allowlisted SIPL_GATHER_TURN_USERNAME and SIPL_GATHER_TURN_PASSWORD override the respective fields in each configured ICE entry. These are read only when the service starts. Keep real credentials out of version control and use short-lived TURN credentials where your relay supports them. Approved and waiting clients receive the configured ICE settings after a valid invitation; keep those credentials limited to relay use.

Desktop app also accepts title, profile, width (640–3840), height (480–2160), and closeAfterMs (0–120000, zero disables automatic close). The example stores the system engine profile in data/browser beside the entry file. That profile holds browser permissions/cookies; meeting rooms, messages and shared files remain in the serving process's memory.

HTTPS and other computers#

Local testing works at http://127.0.0.1:8083. Browser media APIs require a secure context: use HTTPS for another machine. Browsers must trust the certificate; SIPL never bypasses certificate warnings or media permission prompts. A certificate for a public hostname must match that hostname.

For a direct HTTPS listener, edit the JSON configuration:

{
  "port": 8443,
  "host": "0.0.0.0",
  "publicOrigin": "https://meet.example.com:8443",
  "certFile": "certs/fullchain.pem",
  "keyFile": "certs/privkey.pem",
  "maxParticipants": 6,
  "iceServers": [],
  "createKey": "replace-with-a-long-private-server-key"
}

Supply the certificate/key, point DNS at the server, and allow the selected TCP port through the firewall. The hostname must resolve and be reachable from the hosting desktop too. When publicOrigin is set, a locally hosted native window uses it, allowing its certificate to match. Certificate renewal is a deployment responsibility; restart the listener after replacing its files.

Alternatively, put an existing HTTPS reverse proxy in front of SIPL's loopback HTTP listener. Set publicOrigin to its exact public HTTPS origin and set createKey; keep host at 127.0.0.1. Forward the original Host header and the /, /api and /api/file/... paths. Signaling uses short HTTP requests, so WebSocket upgrade configuration is unnecessary. Increase proxy body limits only enough for 256 KiB chunks. Do not expose the plaintext upstream publicly.

Signaling, mediation and relays#

SIPL owns the room/session broker: invitations, admission, member authentication, private offer/answer/ICE delivery, roster updates, chat, presentation state and file transfer. The bundled frontend owns WebRTC negotiation and reconnects. Neither part requires application-specific protocol code.

Media uses browser WebRTC with DTLS-SRTP between participants. This first version uses a small-room peer mesh, not an SFU/mixing server: each participant sends to the other participants. The server is trusted with room metadata, chat and files; those travel over the application's HTTPS connection. This does not provide application-level end-to-end encryption for chat or files.

The signaling server cannot by itself make media cross every firewall/NAT. For internet meetings configure reachable STUN/TURN services. For example:

"iceServers": [
  {"urls": "stun:relay.example.com:3478"},
  {
    "urls": ["turn:relay.example.com:3478?transport=udp", "turns:relay.example.com:5349?transport=tcp"],
    "username": "meeting-user",
    "credential": "replace-with-a-relay-credential"
  }
]

The client automatically uses the configured relay when ICE needs it. SIPL does not install or run coturn, provide a TURN server, allocate public relay addresses, configure firewalls, or provide an SFU. Use an existing managed or self-hosted relay. An empty ICE list can work on the same computer/LAN, but is not a guarantee of internet connectivity. The example performs no third-party STUN requests by default.

Meeting behavior and boundaries#

• Devices start off. Enabling a camera/microphone requests browser/OS permission. Disabling them stops the capture track. Speakers can be selected when the browser supports setSinkId. A click can be necessary to unlock audio playback.

• Screen sharing uses the browser/OS picker, including its visible stop controls. System/tab audio is shared only if the selected source/browser supplies it. Mobile browsers may not support screen capture. SIPL does not capture silently.

• A new meeting uses a waiting room by default. The host can admit/deny people, remove members, change admission for new arrivals or end the meeting. Turning off the waiting room does not automatically admit people already waiting. If the host leaves, the first remaining approved participant becomes host.

• Invitations are random capabilities in the URL fragment, outside HTTP request targets. Anyone with an invitation and any required passcode can request entry. Keep both private. There is no persistent directory/account integration in Gather yet; its meeting roles are separate from CRM-ERP roles.

• Browser HttpOnly/SameSite sessions and a per-tab bearer ticket jointly bind requests. Refreshing the same tab restores its session and renegotiates media. Reconnect media retries a failed connection. Expired/removed members must rejoin.

• Only admitted members receive chat, other approved member details or file data. Signaling is recipient-specific and cannot cross rooms. The waiting list is visible only to the host. The frontend renders participant/chat/file text as text, not HTML.

• PDFs use the system/browser PDF viewer, not an external document service. Use Gather's page controls for synchronized presentation. Scrolling or controls inside the native PDF viewer are local; PDF page counts are not extracted, and page controls accept 1–1000. A %PDF- prefix and .pdf extension identify a presentation candidate, not a comprehensive document validation/sanitization. Other files are download-only. The viewer can fall back to downloading.

• Uploaded files are at most 8 MiB each, 16 per room, and 32 MiB across the server. Chunks are exactly 256 KiB except the final chunk. Uploads are ordered, private to the meeting, and never interpreted as server paths. The owner or host can remove a file. Abandoned incomplete uploads expire after two minutes.

• At most 16 rooms exist. A room lasts six hours. Members expire after 45 seconds without requests; empty rooms expire after two minutes. Cleanup runs on API activity. Chat retains the latest 150 messages. Restarting/closing the hosting server removes every room, message and shared file. There is no recording, durable archive, calendar integration, telephony or large-conference SFU.

Native runtime API and builds#

The runtime adds optional webview.available and blocking webview.run(url, options?). web.serve accepts optional host and window options; window takes the same window settings plus optional url. The default is the actual loopback listener URL, including an automatically assigned port. The window may navigate only within its initial origin; popups and the SIPL/JavaScript native message bridge are disabled. No application authority is injected into pages. Media permissions remain browser/OS controlled.

MSVC Windows builds use the pinned Microsoft WebView2 SDK header/static loader and the installed Edge WebView2 Evergreen Runtime. No Chromium distribution or WebRTC stack is bundled with SIPL. SIPL_WEBVIEW=0 omits the native backend. webview.available is false without a supported backend/installed engine. Linux/macOS builds currently serve the web frontend; a native web window backend for those platforms is not implemented. Existing native SIPL ui is unchanged. Existing HTTP servers keep their loopback default and do no window work unless window is supplied.

Build the example's desktop and browser server executables with the current runtime. The desktop uses the Windows subsystem to avoid a console window:

sipl build examples/projects/Gather/main.sipl -o build/Gather.exe --subsystem windows
sipl build examples/projects/Gather/server.sipl -o build/GatherServer.exe

Copy config.json beside an executable to customize it. No source/assets or launcher scripts are needed there. Run Gather.exe --url=https://meet.example.com to open an existing service. Port overrides are accepted with --port=8084.

Developer verification#

sipl test tests/test_meeting.sipl
python tests/meeting_http_tests.py build/sipl.exe
python tests/meeting_http_tests.py build/GatherServer.exe --packaged
node tests/meeting_browser_tests.js build/sipl.exe
node tests/meeting_desktop_tests.js build/sipl.exe
node tests/meeting_desktop_tests.js build/Gather.exe --packaged
sipl doc libraries/meeting/meeting.sipl --check
python scripts/embed_meeting_assets.py --check

The browser/native tests run on Windows with Node and installed Chrome/WebView2. They use separate profiles and synthetic capture devices; those test flags are never set by the application. The browser test checks actual inbound audio bytes and decoded video frames in both directions, screen capture, admission, safe chat, file/PDF transfer, refresh, host transfer and cleanup. The native test opens the actual Win32 window, captures media, and checks server shutdown on close. Physical devices, public NAT/TURN connectivity, external reverse proxies and other browsers still need deployment/device testing. Backend/model tests run on Windows and Linux; model tests also run with GC stress and Linux sanitizer builds.

Edit the canonical frontend in libraries/meeting/web, then run python scripts/embed_meeting_assets.py. This generates assets.sipl; it is developer tooling only. The checked-in generated module keeps builds offline.

SIPL Pages#

Keep .sipl.html files outside the static html root. As defense in depth, the static server rejects requests for .sipl.html source with 403.

SIPL Pages add server-rendered HTML without turning an HTML file into an unrestricted program. A route loads data, passes one explicit model to a page, and the page renders it with a single tag pair:

<h1><^ model["title"] ^></h1>

<^ opens a SIPL Page tag and ^> closes it. There is no second output tag, raw-output shorthand, or alternate delimiter. Values are escaped by default. Tags contain one expression or page keyword and do not end with a semicolon. The pair is visually distinct, uncommon in HTML, and easy to identify without the EJS-style choice between silent, escaped, and raw tag variants. Page source is compiled away, so the delimiters never reach the browser.

The source extension is .sipl.html. This keeps HTML tooling useful while making executable page sources easy for people, AI tools, the server, and security scanners to identify.

Why this shape#

PHP-style file execution is convenient, but it mixes routing, database access, business logic, and rendering. It also encourages work to be repeated on every request and makes authority difficult to see.

SIPL Pages instead use an explicit controller/view boundary:

HTTP request -> SIPL route -> database/query -> page model
                                           -> compiled page -> response

The route is ordinary SIPL and owns side effects. The page owns presentation. This gives an AI a small, regular target: one model name, one delimiter pair, one escaping rule, and a short list of page constructs.

Pages are compiled, not reparsed and interpreted for every request. Production startup compiles the page tree and fails early on malformed pages. Rendering then executes cached bytecode and appends into a bounded response buffer.

API#

Quick start and Toolkit workflow#

The page is a view, not a standalone program. It needs a controller to provide its model and register an HTTP route. The bundled pair is:

• examples/web.sipl — executable controller

• pages/index.sipl.html — private compiled page

From a terminal:

.\build.bat
.\build\sipl.exe run examples\web.sipl

Then open http://127.0.0.1:8082/.

In the Toolkit, open pages/index.sipl.html and choose Scan to validate the page. To view it in a browser, open examples/web.sipl and choose Run. The Toolkit deliberately does not execute a page by itself because doing so would invent missing database/model values. Pressing Run while a page is selected now explains this in the Output panel.

After rebuilding SIPL, restart the Toolkit so its background worker uses the new executable. Windows cannot replace build\sipl.exe while the Toolkit is running, so close it before running build.bat.

Configure a private page root before starting the server:

fn start() {
    web.pages("pages")
    web.get("/users", usersPage)
    web.serve("html", 8080)
}

pages and html are deliberately different roots. Files in html are public static assets. Files in pages are executable source and are never served as files.

A route queries data and renders a named page:

let database = sql.open("data/application.db")

fn usersPage(request) {
    database.query(
        "SELECT id, name FROM users ORDER BY name LIMIT ?",
        [100],
    )

    return web.render("users.sipl.html", {
        "title": "Users",
        "users": database.rows,
    })
}

web.render(path, model) returns the normal web response Map with status 200, Content-Type: text/html; charset=utf-8, and the rendered body. The controller can return it directly or change its status and application-owned headers.

The first web.render argument must be a string literal and is resolved beneath the configured page root. Absolute paths, traversal, symlinks or junctions escaping the root, and a path derived from an HTTP request are rejected. Page files cannot be uploaded and executed at runtime. A controller that needs to choose a view uses ordinary branching with a literal render call in each branch.

Only page-safe values may cross the model boundary: None, Bool, Int, Float, String, and acyclic List and Map trees containing those values. Database handles, collections, functions, native objects, and request objects are rejected. Model values are read-only while rendering; mutating collection methods are unavailable. This makes the page's authority visible in the model.

Page syntax#

A page is HTML plus the following constructs. All constructs use <^ and ^>.

Escaped expressions#

An expression writes its value:

<title><^ model["title"] ^></title>
<p>Hello, <^ model["user"]["name"] ^>.</p>

none writes nothing. Booleans and numbers use their canonical SIPL text. Strings are never treated as markup.

Immutable local names#

let gives a repeated expression a short name and writes nothing:

<^ let users = model["users"] ^>
<p><^ users.length ^> users</p>

Conditions#

<^ if model["users"].length == 0 ^>
    <p>No users yet.</p>
<^ else ^>
    <p>Users are ready.</p>
<^ end ^>

else if is supported. Every block ends with <^ end ^> so mixed HTML and SIPL never depends on matching braces across languages.

Loops#

<tbody>
    <^ for user in model["users"] ^>
        <tr>
            <td><^ user[0] ^></td>
            <td><^ user[1] ^></td>
        </tr>
    <^ end ^>
</tbody>

Indexed iteration uses normal SIPL order:

<^ for index, item in model["items"] ^>
    <li data-index="<^ index ^>"><^ item ^></li>
<^ end ^>

The v1 page language intentionally has no var, assignment, while, function declaration, import, file access, database access, networking, or session access. That work belongs in a route or an ordinary SIPL function. Page expressions can read and transform page-safe values with their safe built-in operations.

HTML-aware output safety#

Escaping must happen where a value is rendered because HTML text, attributes, URLs, CSS, and JavaScript are different parsing contexts. A global string replacement is not sufficient.

The page compiler tokenizes the surrounding HTML and permits expressions only in supported contexts:

For example:

<a href="<^ html.url(model["userUrl"]) ^>"
   title="<^ model["userName"] ^>">
    <^ model["userName"] ^>
</a>

html.url(value) accepts relative application URLs and explicit http or https URLs. Dangerous schemes such as javascript: are rejected. A later API may offer narrower helpers for route paths and query components.

There is no raw-string tag in v1. html.include(...) is accepted only as a complete output expression and renders another compiled page directly; a database String cannot be cast to trusted markup. Rich user-authored markup requires a separately designed and maintained sanitizer; calling an escape hatch is not sanitization.

Client data may be embedded only as non-executable JSON:

<script id="page-data" type="application/json">
<^ html.json(model["clientData"]) ^>
</script>

html.json is accepted only in that exact context. It uses the SIPL JSON encoder and also protects the enclosing HTML script element, including <, >, &, and closing-script sequences. Inline JavaScript interpolation is not supported. Client code can read the element's text and call JSON.parse.

To write the literal opening delimiter in visible text, use <^.

These rules follow OWASP's guidance that output encoding must match the final browser context and that dangerous contexts should not accept dynamic values: Cross Site Scripting Prevention Cheat Sheet.

Partials#

html.include renders another page beneath the same private root:

<ul>
    <^ for user in model["users"] ^>
        <^ html.include("partials/user.sipl.html", {"user": user}) ^>
    <^ end ^>
</ul>

The include path must be a string literal so dependencies are known at compile time. It is relative to the current page, cannot escape the page root, and is compiled once. Include cycles and excessive nesting are compile errors.

Layouts and components can be added after the core renderer is measured. They should build on literal html.include dependencies and the same model boundary rather than add more delimiter forms.

Compilation, caching, and errors#

web.pages(root, options?) owns page compilation and caching.

• The default production mode scans and compiles every .sipl.html file once before web.serve. A failure reports the page path, line, and column and prevents startup.

• {"reload": true} is development-only. A changed page is compiled atomically; requests never observe a half-compiled page. The last good page may continue serving while the error is reported locally.

• Literal HTML is stored as immutable chunks. Dynamic values are escaped directly into one geometrically grown, bounded output buffer; the renderer does not build a chain of temporary concatenated Strings.

• The generated bytecode keeps source mappings back to the page. Runtime failures log the page location on the server and return the existing generic 500 response without source, model data, or stack details.

• Page source is limited to 4 MiB and a root to 1,024 page files. Rendering is bounded to 1 MiB of output, 1,000,000 work units, 2 seconds, and 32 nested includes. Models allow at most 100,000 values, 64 levels, and 1 MiB of String data; cycles and non-finite Floats are rejected.

The JIT may accelerate hot generated loops and expressions. HTML escaping, buffer growth, output limits, database latency, and request concurrency remain consistent regardless of the selected execution path.

AI-facing tools#

The CLI provides machine-readable validation rather than relying on a browser to reveal mistakes:

sipl page-check pages
sipl page-explain pages/users.sipl.html

page-check validates every page and returns a nonzero exit code on an error. Diagnostics include the source path, line and column when available, the rejected construct, and a focused repair message.

page-explain reports static byte count, dynamic slots, loops, conditions, include dependencies, and the checked execution model.

The Toolkit recognizes .sipl.html as SIPL Page, highlights HTML and SIPL tags in the same file, and uses the same page-check command for Scan. Page formatting is intentionally not automatic yet, so an HTML formatter cannot silently damage embedded SIPL tags.

Security invariants#

SIPL Pages enforce these rules:

• Request or database text is data; it is never reparsed as a page or SIPL.

• Every expression is context-escaped or rejected.

• Page source is never returned by static serving or an error response.

• Routes choose page paths; clients do not.

• Pages cannot acquire file, network, database, session, process, or other capabilities through globals or model values.

• Production requests perform no page parsing, compilation, or page-file I/O.

• Output, work, nesting, and time are bounded.

• Development reload cannot replace a valid cached page with a failed compile.

Treat any feature that weakens one of these invariants as a separate security design, not a convenient template option. User-controlled template source can become server-side template injection and must never reach this compiler; see the OWASP server-side template injection guidance.

Portable HTTP and TLS client#

SIPL includes an HTTP and HTTPS client on Windows and Linux. Requests return NetTask; they do not perform a blocking transfer at the call site. Drive tasks with net.pollBlocking() in your event loop, or call .waitBlocking() to block explicitly. No separate networking program is required.

let client: HttpClient = http.client({"maxConnections": 8})
let task: NetTask = client.get("https://example.com/api/items", {
    "headers": {"Accept": "application/json"},
    "connectTimeout": 3000,
    "readTimeout": 5000,
    "totalTimeout": 15000,
})
let response: HttpResponse = task.waitBlocking()
print(response.status, response.body.decodeUtf8())
client.close()

This prerelease replaces the old response Map and synchronous request return value. Use http.get(url).waitBlocking().body, rather than http.get(url)["body"]. There is no await syntax requirement. See shared networking for cancellation, streams, TCP/UDP, and WebSockets.

Requests and responses#

Module-level requests share a bounded default pool. Explicit clients isolate pools and provide deterministic cleanup. Client options are maxConnections (Int 1..64, default 8) and idleTimeout (Int milliseconds 1..600000, default 60000). At most 64 pools may exist. Idle pools are pruned when networking is polled; connection age/lifetime limits also prevent reuse of stale connections.

HttpResponse is a native, annotatable type with read-only properties:

Non-2xx status codes are ordinary responses. task.response exposes live headers/body for streaming; buffered body bytes are published on completion. task.result is none while pending. Inspect task.error when polling manually; task.waitBlocking() raises a catchable system Fail for network, policy, timeout, cancellation, certificate, or resource-limit failure. Invalid API arguments raise a runtime Fail before scheduling. Closed clients reject new requests.

Scheduling copies native options and retains request data: O(header and option bytes), without waiting for I/O. Transfers take O(transferred/decoded bytes), plus DNS/TLS work. Buffered requests use O(body limit + headers) space; streaming requests use bounded queues and headers. Closing a client takes O(pending requests + cached connections). Effects include DNS, network I/O, cryptography, native allocations, and reading explicitly named certificate files.

Request options#

Unknown HTTP option names are rejected.

Response headers are capped at 64 KiB across the transfer, including redirects, with at most 128 fields per response and 8192 bytes per value. Header names and values are validated; invalid UTF-8 values fail rather than becoming invalid SIPL Strings. The client owns Host, Connection, Content-Length, and Transfer-Encoding; application overrides are rejected. Proxy credentials use the dedicated options, rather than an injected Proxy-Authorization header.

Read timeout pauses while a stream is deliberately backpressured. Total timeout continues, so a producer or consumer that never resumes cannot hold a task forever. Cancellation is cooperative: call net.pollBlocking() to observe token changes and move pending operations forward.

Redirect policy#

No redirects are followed by default. same-origin requires the same scheme, host, and effective port at every hop. cross-origin explicitly permits an origin change, but still blocks TLS downgrade unless allowDowngrade is true. Custom application headers and client certificate/key credentials are removed on an origin change. They are never automatically restored at a later hop.

303 changes to GET; 301/302 change POST to GET. 307/308 preserve the method/body. Cross-origin resending of a body additionally requires redirectWithBody. A streaming upload cannot be replayed; a redirect requiring replay fails. URL-embedded credentials and non-HTTP(S) URLs are rejected at every hop.

Certificates and portability#

Peer verification and hostname verification are always enabled. TLS 1.2 is the minimum. The default uses the platform trust roots; caFile supplies an explicit trust bundle and pinnedPublicKey adds a public-key constraint.

PEM client certificates and keys are supported by default. Windows also supports PKCS#12 (P12) explicitly with clientCertType, using clientKeyPassword for encrypted input. Platform trust roots are used for peer verification. Certificate formats are selected explicitly with clientCertType; unsupported inputs fail. Certificate and password inputs stay out of command-line arguments and source-generated files. Application code chooses where to obtain them.

The built-in client exposes HTTP(S) only. HTTP/2, HTTP/3, gRPC, SMTP, and MQTT are not separate built-in protocols. Client private keys are not imported into the operating system certificate store.

Streaming examples#

let task = http.get(address, {"stream": true, "totalTimeout": 30000})
let body: NetStream = task.response.body
while not task.done or not body.eof {
    let chunk = body.read(16384)
    if chunk != none {
        consume(chunk)
    } else {
        net.pollBlocking(5)
    }
}
let response = task.waitBlocking() // reports any transfer failure

A response stream is read-only. body.close() cancels a pending response and releases queued data. Dropping a pending task aborts its transfer; a retained stream reports that failure. Keep the task reachable while using its stream.

let upload = net.stream(65536)
let task = http.post(address, upload, {"totalTimeout": 30000})
for chunk in sourceChunks {
    while not upload.write(chunk) {
        net.pollBlocking(5)
    }
}
upload.close() // signals end of request body
let response = task.waitBlocking()

Streams preserve Bytes; encode/decode text explicitly. JSON remains the one json.parse/json.stringify API. Drain streaming responses while polling, rather than waiting for the complete transfer with an unread full queue.

Verification#

tests/network_tests.py tests local HTTP/HTTPS, untrusted and explicit trust, redirect policies and credential stripping, decompression bounds, header/body limits, connection reuse, proxying, cancellation, separate read deadlines, request/response streaming backpressure, TCP framing, UDP, and packaged network types. All runtime scenarios run normally and with GC stress. The independent Python test servers do not contact external APIs or alter certificate stores.

Shared networking tasks, sockets, and WebSockets#

HTTP, TCP, UDP, and WebSocket operations use the same NetTask, CancelToken, and net.pollBlocking() scheduler. The scheduler runs on the calling SIPL thread; network transfer callbacks do not execute SIPL on background threads. DNS for outbound connections is resolved asynchronously. Pending operations are bounded at 256, open Socket objects at 128.

Task and stream contracts#

CancelToken.isCancelled reflects manual cancellation or its monotonic deadline. NetTask.done, .error, and .result allow polling without blocking; result is none while pending. HTTP tasks also provide .response before completion. NetStream.eof means closed and drained, .buffered reports queued bytes, and .capacity reports the queue bound. Closing a producer still lets a consumer drain its queued data. Transfer errors are observable through .waitBlocking() or stream reads; none means no data now, and .eof distinguishes completion.

A wait timeout leaves the task pending. Cancel explicitly if the caller no longer wants it. The underlying operation's total deadline still applies. Cancellation of partially consumed framing or partially sent data can close its socket to prevent reusing a stream with an uncertain frame boundary.

TCP and UDP#

let listener: Socket = tcp.listen("127.0.0.1", 9000)
let peer: Socket = listener.accept().waitBlocking()
let message = peer.readFrame(1048576).waitBlocking()
peer.send(framing.lengthPrefix(message)).waitBlocking()
peer.close()
listener.close()

TCP read limits are upper bounds, not an assumption that one send equals one read. Framing helpers make message boundaries explicit. Line reads buffer excess received bytes for the next read. A socket permits one pending read and one pending write concurrently; duplicate operations in a direction fail. UDP preserves datagram boundaries and never silently returns a truncated packet.

Operation options are timeout (10 seconds by default), connectTimeout, readTimeout, totalTimeout, and cancel, with the same millisecond ranges as HTTP. Listener options include backlog (1..128, default 32). tcp.connect additionally accepts tls: true, caFile, client certificate/key, format/password, public-key pin, and explicit proxy options as described in HTTP. Peer/hostname verification remain enabled. Bind/UDP destination addresses are numeric so those synchronous setup calls do not perform hidden DNS.

Socket.port is the assigned local Int port, .address is its numeric address Map, and .closed reports closure. Port zero lets the OS assign a free port. Scheduling costs O(outgoing bytes retained/copied); transfer/framing costs O(transferred bytes). Read buffers and frame allocations are bounded by their specified limits; sockets have a bounded 16 KiB read cache. Effects include DNS for connects, socket creation, network I/O, allocations, and optional TLS.

WebSocket client and server#

let task = websocket.connect("wss://example.com/updates", {
    "maxMessage": 1048576,
    "connectTimeout": 3000,
    "readTimeout": 10000,
    "totalTimeout": 15000,
})
let ws: WebSocket = task.waitBlocking()
ws.sendText("hello").waitBlocking()
let message = ws.receive().waitBlocking()
print(message["type"], message["body"])
ws.close(1000, "done").waitBlocking()

Server upgrades use the same Socket tasks:

let listener = tcp.listen("127.0.0.1", 9000)
let peer = listener.accept().waitBlocking()
let ws = websocket.accept(peer, {"origin": "https://my.example"}).waitBlocking()
let message = ws.receive().waitBlocking()
ws.sendBinary(message["body"]).waitBlocking()
ws.close().waitBlocking()
listener.close()

WebSocket options include the shared timeout/cancellation options, TLS/proxy client options, maxMessage (1..16 MiB, default 1 MiB), origin (up to 2048 bytes), and a single protocol token (up to 256 bytes). Upgrade headers are bounded at 64 KiB and 128 fields. Subprotocol mismatch fails. Extensions, including permessage-deflate, are not negotiated.

The server rejects a supplied browser Origin unless it matches its explicit origin option. Clients without an Origin header are permitted. Server-side listeners currently provide plain TCP/WS; use TLS termination in front of the listener for WSS service. WSS clients verify certificates on both platforms. The server does not silently accept every browser origin or enable compression.

Incoming ping frames automatically queue a matching pong while a receive/close task is being polled; pong frames are consumed internally. Keep a receive task pending to process control traffic. Close performs a bounded two-way handshake; timeout/cancellation aborts it. .closed, .path, and .closeCode expose status. One send and one receive may run concurrently; additional sends are rejected instead of accumulating an unbounded queue. Control output is capped at 512 bytes.

Frames enforce role-specific masking, reserved-bit/opcode rules, canonical lengths, control-frame limits, UTF-8 text/close reasons, and a total fragmented message bound. Handshake nonces and client masks use OS secure randomness. Wire processing is O(message bytes); fragmented data plus the current frame uses O(message limit) space. Effects include sockets, TLS for WSS clients, secure random generation, handshake hashing, and bounded native allocations.

DNS and discovery boundary#

Outbound HTTP/TCP/WSS connections already use cancellable, timeout-bounded DNS through the shared transport. A standalone DNS query API and service discovery are optional future modules; they should return NetTask and accept CancelToken, rather than introduce a second scheduler. Multicast/device discovery should also be tested on its target network, since WSL networking mode affects those paths. Ordinary local unicast TCP/UDP/TLS tests run on WSL 2.

Tests#

tests/network_tests.py runs independent local HTTP/TLS/proxy/socket servers. tests/websocket_tests.py supplies a raw RFC 6455 oracle, including TLS, fragmented UTF-8, ping/pong, close, origin policy, masking and size violations. Runtime cases run normally and with GC stress. Linux/WSL and native Windows use the same tests. Test certificates are temporary and are never installed in the system trust store.

Structured tasks and async I/O#

async fn, await, and spawn use one runtime-owned event loop on the SIPL thread. Each task has a suspended VM stack; SIPL values never move to another OS thread. Native DNS resolvers may use threads with native data only. Tasks can share SIPL values on the event-loop thread; this does not make multi-step updates transactional.

async fn fetchTitle(url: String) -> String {
    let response: HttpResponse = await http.get(url)
    return response.text()
}

async fn start() {
    let first = spawn fetchTitle("https://example.com")
    let second = spawn fetchTitle("https://example.org")
    print(await first, await second)
}

The runner starts an async start() automatically. Calling an async function schedules an owned Task; spawn makes this intent explicit and accepts a Task or NetTask. A function's return annotation describes its awaited result. Task itself is an annotatable handle. Async methods, nested functions, lambdas, and async test "name" { ... } use the same model. await and spawn are legal only in async functions. Synchronous callbacks cannot schedule or suspend tasks.

Ownership and scope exit#

The lexical block containing a task or I/O call owns it. The owner retains the operation even when its handle is discarded. At block exit, unfinished children are cancelled and joined before locals are released. This applies to normal fallthrough, return, break, continue, and failure. Cleanup can itself await I/O. Entering a failure handler waits for children of the abandoned block to finish cleanup first. An awaited task's failure is raised at the await expression; an unobserved child failure is raised in the owner and cancels its siblings.

async fn start() {
    {
        let child = spawn background()
        await task.sleep(20)
    } // cancels and joins child, including its next block
}

There is no detach or implicit ownership transfer. A handle stored outside its block refers to a completed or cancelled operation after that block closes. Loop bodies own their children for each iteration: returning a handle or adding it to an outer List does not extend that lifetime. Awaiting another owner's handle does not transfer ownership. Self-awaits, ancestor-awaits, and await cycles are rejected.

Tasks start with their owner's lifetime rather than a hidden wall-clock limit. setDeadline() adds an optional monotonic cancellation deadline. Cancellation is cooperative: it is delivered once on resumption, at an await, or after a VM fairness slice. A slice yields at a loop backedge after approximately 10,000 instructions; ordinary expressions are not interrupted mid-instruction. Callbacks run to completion. CPU-heavy or untrusted work belongs in an isolated worker process.

Typed cancellation and cleanup#

Cancellation raises Cancelled, a subtype of Fail, with kind == "cancelled". It uses ordinary try / fail / next. Catching cancellation can return a result; isCancelled records the request, and .failure is none after successful recovery. next may await bounded cleanup. A deadline requests cancellation, rather than forcibly terminating SIPL code.

async fn start() {
    let child = spawn fetchTitle("https://example.com")
    child.cancel()
    try {
        print(await child)
    } fail {
        if fail.kind == "cancelled" {
            print("Fetch cancelled")
        } else {
            fail(fail)
        }
    }
}

I/O timeouts raise a system Fail; explicit task/token cancellation raises Cancelled. NetTask.isCancelled exposes the latter. A successful Fail value can be returned as ordinary task data and is distinct from a task that failed.

Public task API#

Read-only Task properties are .done, .result, .failure, .isCancelled, and .owner. Result/failure are none while pending. Owner is the creating Task until the completed handle is released from its scope. Finished tasks release their VM stacks and captured execution state.

At most 128 structured tasks, 256 pending native I/O operations, and 512 owned operations may exist at once. Each task has 16,384 stack slots, 256 call frames, and up to 512 nested ownership scopes. Admission fails before unlimited work is queued. Observing a completed operation releases its ownership record. Scheduler work is bounded by runnable tasks, owned operations, and ready I/O; suspended contexts and queues count toward the stated memory bounds.

Awaitable I/O and blocking boundaries#

HTTP, TCP, UDP, WebSockets, timers, and bounded byte streams all feed the same event loop. See HTTP and networking. HTTP responses provide text() for strict UTF-8 decoding of a completed buffered body. Requests have connect/read/total timeout policies; sockets and WebSockets have operation deadlines. Stream operations default to a 30-second total timeout.

Synchronous callers use the explicit NetTask.waitBlocking(milliseconds?) and net.pollBlocking(milliseconds?) names. Blocking calls are rejected inside async tasks. The existing synchronous disk, database, process, stdin, UI frame, and AI ask APIs belong outside async tasks. Load configuration before entering async code. Blocking time.sleep(Duration) also belongs outside async tasks; use time.timer(Duration) for an awaitable delay. Calendar types, interval phases, cron syntax/DST behavior, and explicit locale profiles are documented in STDLIB.md.

Worker jobs retain their process-isolated lifecycle and do not become shared-memory async tasks. Outside async code, low-level native operations belong to the program lifetime and are cancelled when that lifetime ends.

Bounded byte-stream contract#

net.stream(capacity?) is a binary I/O queue with capacity 16 KiB..1 MiB, default 64 KiB. Bytes are copied into the queue; a producer and consumer may hold the same stream on the SIPL thread. This is not an actor or arbitrary-value channel. Ownership applies to each scheduled operation. Only one async reader and one async writer may be pending. Polling operations cannot bypass a pending operation in the same direction; uploads already have their stream consumer.

Options are timeout / totalTimeout (1..120000 ms, default 30000) and cancel (CancelToken). A full queue suspends the writer rather than growing. Closing a producer permits draining followed by EOF; closing a response stream cancels its HTTP transfer. Transfer failures propagate after queued bytes are drained. Cancellation does not retract bytes already consumed or sent.

select, race, arbitrary-value channels, actor APIs, and shared-memory threads are not introduced by this model. The lifetime and failure tests run in isolated processes with GC stress, and the async network oracle also tests packaged code.

SQL Tables#

SIPL provides a lightweight Table data type for raw SQL and parameterized queries. It is not an object mapper and does not add change tracking or an ORM. SQLite is built in and needs no database server. MySQL and MariaDB use the native MariaDB Connector/C driver and a running database server. Both drivers share the Table API and bind parameters through native prepared statements.

Creating a table#

SQLite supports two storage lifetimes. Existing sql.memory() and sql.open(path) calls continue to use SQLite.

PDO-style connections#

let local: Table = sql.connect("sqlite:application.db")
let temporary: Table = sql.connect("sqlite::memory:")
let remote: Table = sql.connect(
    "mysql:host=127.0.0.1;port=3306;dbname=application;charset=utf8mb4",
    "application_user",
    password,
)

sql.connect(dsn, username?, password?) accepts sqlite:, mysql:, and mariadb: prefixes. mysql: connects to either MySQL or MariaDB; mariadb: is an alias. MySQL options are host, port, dbname, charset, unix_socket, and sslmode. Defaults are localhost, port 3306, and utf8mb4. Omitted credentials are empty strings. Unknown options, invalid ports, and NUL bytes in DSNs or credentials fail. Connection establishment has a 10 second timeout.

For a Unix socket, use mysql:unix_socket=/run/mysqld/mysqld.sock;dbname=application. SQL dialects and schemas remain database specific; SQLite SQL is not rewritten. The API follows PDO's driver selection and binding model, with SIPL's existing query/execute result model. Parameters support positional ? placeholders with a List, or named :name placeholders with a Map. There are no user-visible reusable statement objects.

Building the MySQL/MariaDB driver#

Every standard build includes SQLite and MySQL/MariaDB. Pinned MariaDB Connector/C 3.4.11 source is bundled under third_party/mariadb and compiled into a static library during make, build.bat, and the Android CMake build. No separately installed Connector/C SDK, SIPL_MARIADB_ROOT, connector DLL, or connector shared library is required. TLS reuses Windows Schannel or the existing OpenSSL/LibreSSL build inputs. Authentication plugins are compiled into the client; they are not external DLLs.

Feature-off switches such as SIPL_MYSQL=0 are rejected. A missing build prerequisite stops compilation instead of silently omitting the driver. system.buildFeatures records the compiled capabilities and connector version; standard build reports record the runtime and Toolkit sizes as well.

The connector uses Connector/C authentication and transport support. TCP connections retain its certificate-verifying TLS defaults when sslmode is omitted. Explicit Unix sockets without sslmode use local transport and do not require server TLS. Set sslmode=preferred to request TLS without certificate verification and allow servers without TLS, sslmode=required to require encryption without certificate verification, or sslmode=verify_identity to require encryption and verify the server certificate and hostname. The latter two modes fail if TLS is unavailable. persistent describes remote database storage, not a pooled or reused PDO persistent connection.

Transactions and inserted IDs#

remote.begin()
try {
    remote.execute("INSERT INTO users(name) VALUES(?)", ["Ada"])
    let id = remote.lastInsertId()
    remote.commit()
} fail {
    remote.rollback()
}

begin(), commit(), and rollback() work with both drivers. MySQL tables must use a transactional engine such as InnoDB; server rules about implicit DDL commits still apply. lastInsertId() returns a decimal String, preserving unsigned MySQL BIGINT IDs. It refers to the connection's most recent insert. No transaction or query syntax is rewritten by SIPL.

Temporary, in-memory#

let table: Table = sql.memory()

The database exists only while the Table is open in the current process. Closing the table or ending the program destroys its contents.

Persistent, on disk#

let table: Table = sql.open("data/application.db")

SIPL writes this database to disk. It can be closed and reopened by the same program or a later SIPL process:

table.close()

let reopened = sql.open("data/application.db")

SIPL creates the file when it does not already exist.

Executing SQL#

Use execute for statements that do not need a result grid:

table.execute("""
    CREATE TABLE users(
        id INTEGER PRIMARY KEY,
        name TEXT NOT NULL,
        score REAL
    )
""")

let changed = table.execute(
    "INSERT INTO users(name, score) VALUES(?, ?)",
    ["Ada", 9.5],
)

execute returns the number of rows changed by the statement.

Parameters are safely bound. Use a List for positional ? placeholders or a Map for named :name placeholders. Both support None, Bool, Int, Float, and String values.

table.query(
    "SELECT `settingvalue` FROM `settings` WHERE `settingname` = :settingname LIMIT 1",
    {"settingname": "site_title"},
)
table.execute(
    "UPDATE settings SET settingvalue = :value WHERE settingname = :name",
    {"name": "site_title", "value": "My application"},
)

Map keys may include the leading colon (":settingname") or omit it. Do not supply both forms of the same name. Names are case sensitive and use ASCII letters, digits, and underscores, beginning with a letter or underscore. Repeated markers bind the same value to each occurrence. Missing, unused, or ambiguous keys fail, and named and positional placeholders cannot be mixed with a parameter map. Markers inside quoted strings, quoted identifiers, and comments are left untouched. MySQL backslash escaping follows the connection's NO_BACKSLASH_ESCAPES status. With MySQL backslash escapes enabled, use doubled double quotes rather than backslash-escaped double quotes in named statements; the latter are rejected because ANSI_QUOTES can change their meaning. Parameters inside executable MySQL comments are not supported.

Named markers are rewritten to native parameter slots; parameter values are never interpolated into SQL text. Both naming styles provide the same binding protection. Bind values only; identifiers such as table and column names cannot be supplied as parameters.

Querying rows#

query executes SQL, replaces the table's current rows and columns, and returns the same Table:

table.query(
    "SELECT id, name, score FROM users WHERE score >= ? ORDER BY score DESC",
    [5.0],
)

print(table.columns)
print(table.rows)

An optional third argument caps materialized rows: table.query(statement, params, maxRows), where maxRows is 1..10000. The statement is stopped after the cap, so use this for read-only browsing queries. The default is unlimited for compatibility with existing callers.

executeScript(script) runs a SQL script containing multiple statements and returns the number of rows changed. Scripts are limited to 16 MiB. Use this only for trusted imports or SQL entered by the user; statements already completed before an error are not automatically rolled back.

MySQL/MariaDB integer and floating types become Int and Float. Unsigned integers above INT64_MAX stay decimal strings. DECIMAL/NUMERIC, temporal, JSON, and other nonnumeric types remain strings, preserving exact decimal precision. Binary columns become strings containing exact bytes, including NUL.

SQLite query results use these conversions:

Table properties#

Closing#

table.close()

close is safe to call more than once. SIPL also closes an unreachable open table during garbage collection, but explicit closing releases the database immediately.

Calling query or execute after closing raises a sql failure.

Errors and limits#

• Each query or execute call accepts exactly one SQL statement.

• A placeholder count mismatch raises a sql failure.

• Preparation, execution, constraint, and file errors raise a sql failure with a useful database message.

• Table connections should be used by the SIPL execution context that created them.

This intentionally remains a thin SQL layer. Application code owns its SQL, schema, transactions, and migrations.

Verification#

Run SQLite regression tests with build/sipl run tests/test_sql.sipl and build/sipl run tests/test_sql_named.sipl. The named-parameter suite also tests MySQL/MariaDB when SIPL_MYSQL_DSN and optional credentials are set. The opt-in MySQL/MariaDB integration test creates only a temporary table in a disposable database:

SIPL_MYSQL_DSN='mysql:host=127.0.0.1;dbname=sipl_test' \
SIPL_MYSQL_USER=sipl_test SIPL_MYSQL_PASSWORD=secret \
build/sipl run tests/sql/mysql_connector.sipl

It checks native binding, Unicode and binary round trips, numeric conversion, transactions, inserted IDs, capped results, and connection reuse after errors. Repeat with SIPL_GC_STRESS=1 to check native resources alongside collection.

Connector implementation reference: MariaDB Connector/C prepared statements.

Document Collections#

SIPL's Collection and NoSqlDatabase use the same embedded BSON engine in src/nosql.c and src/nosql.h. Base opens these databases directly. SQLite provides transactional disk storage; documents are BSON bytes, not JSON text. No server is required. UnQLite and its Jx9 scripting engine have been removed; its unreleased file format and integer __id API are unsupported.

Opening a database or collection#

let notes: Collection = collection.memory("notes")
let id = notes.insert({"title": "Atlas", "score": 42})
print(notes.findById(id))
notes.close()

let saved = collection.open("documents.nsdb", "notes")
saved.insert({"title": "Persistent"})
saved.close()

collection.open(path, name) creates the database and named collection when absent. Each call owns a separate database connection. Collection names contain 1–255 UTF-8 bytes without NUL. The parent directory must already exist.

For multiple collections sharing one connection and transaction:

let db: NoSqlDatabase = nosql.open("documents.nsdb")
let notes: Collection = db.collection("notes")
let archive = db.collection("archive")
notes.begin()
let id = notes.insert({"title": "Shared", "score": 12})
archive.insert({"sourceId": id})
db.commit()
assert(db.findOne("notes", {"_id": id}) == notes.findById(id))
notes.close() // Closes this view; db and archive remain open.
db.close()    // Closes the connection and invalidates all its views.

collection.memory(name) and collection.open(path, name) own their connection: closing them closes the database immediately and rolls back an unfinished transaction. A view returned by db.collection(name) keeps the database alive through garbage collection; closing that view leaves the shared connection open. Unreachable databases release their resources and roll back unfinished writes. Closing is idempotent; reads/writes on a closed handle raise a failure.

Named collection interface#

The database interface exposes the same operations with a collection name as the first argument: db.find("notes", filter, options), db.insert("notes", doc), db.update("notes", filter, update, many, upsert), and so on. Use db.count(name, filter?) for counts. db.collections(), db.createCollection(name), db.renameCollection(name, newName), and db.dropCollection(name) manage collection names. db.begin/commit/rollback, db.backup(path) and db.close() apply to the whole connection. nosql.encode(document) and nosql.decode(bytes) convert raw BSON.

Identity, reads and writes#

The immutable _id field identifies each document. insert generates a BSON ObjectId when _id is absent and returns its {"$oid": "24 hex digits"} map. Explicit scalar IDs such as strings and integers are accepted. Duplicate IDs fail; replacement preserves the existing ID when omitted. There is no __id.

find defaults to 100 documents and supports sort, descending, skip, limit (1–1000) and projection. Results are limited to 32 MiB of BSON; page through larger collections. findOne returns one document or none. lastQueryIndexed reports the latest find candidate plan. Counts may scan.

let c = collection.memory("scores")
c.insertMany([{"name": "Ada", "score": 42}, {"name": "Grace", "score": 36}])
c.createIndex("score_idx", "score")
print(c.find({"score": {"$gte": 30}},
    {"sort": "score", "descending": true, "limit": 20, "skip": 0}))
c.update({"name": "Ada"}, {"$inc": {"score": 1}, "$set": {"profile.active": true}})
print(c.aggregate({:}, "score"))
c.close()

Filters support dotted paths, scalar/array equality, $eq, $ne, $gt, $gte, $lt, $lte, $in, $nin, $exists, $type, $size, $all, $elemMatch, field $not, and top-level $and, $or, $nor. Null equality also matches missing fields. Unsupported operators are errors even with no documents.

Updates support $set, $unset (a field map), $inc, $mul, $min, $max, $push, $addToSet, $pull, $pop, $rename, and $setOnInsert. $push/$addToSet accept {"$each": [...]}. Dotted document paths work; positional/array-index updates are unsupported. Updates return a map containing matched, modified, and upserted. Numeric aggregation returns exact Int64 integer sums or compensated floating sums; integer overflow raises an error.

Batches, multi-document updates/deletes, and BSON imports are atomic. A failed operation rolls back its changes while preserving other operations in an explicit transaction. insertMany accepts up to 10,000 documents and 64 MiB per batch. Documents are limited to 16 MiB and nesting to 64 levels. Cyclic values, non-string map keys, unsupported SIPL objects, invalid BSON, duplicate unique keys, and unsupported operations raise catchable failures. Database/encoding failures use kind collection; argument-shape mistakes can use kind runtime.

Indexes are ordered single-field indexes with sparse/multikey support and an optional uniqueness constraint. Compound indexes, text/geospatial/vector search, regex execution, collation, replication and network server protocols are not implemented. See src/nosql.h for exact engine limits.

BSON values and transfer#

BSON maps use type markers so JSON editing preserves values:

• ObjectId: {"$oid": "507f1f77bcf86cd799439011"}.

• Int64: {"$numberLong": "9223372036854775807"}.

• Double: {"$numberDouble": "1"} (including NaN and infinities).

• Date: {"$date": {"$numberLong": "1700000000000"}} (Unix milliseconds).

• Other BSON types: {"$bson": {"type": 19, "hex": "..."}}; hex is the exact little-endian BSON value payload. Binary subtypes and Decimal128 values survive a decode/JSON edit/encode cycle without losing their original bits.

Ordinary SIPL values encode naturally; small Int values become Int32, larger ones Int64, and Bytes become generic BSON binary. Recognized marker maps are reserved representations. This is a type-preserving BSON editing format, not a complete implementation of MongoDB Extended JSON (for example ISO date strings and $numberDecimal are not parsed). Decimal arithmetic is unsupported.

The database is a local embedded engine, not a MongoDB server or network driver. The C header documents supported query/update operators and index limitations. BSON transfer supports ordinary uncompressed per-collection streams, not MongoDB archives, compressed dumps or live server storage directories. Export requires a new filename to avoid overwriting an open database or existing export.

Base: tables and collections together#

Run examples/projects/Base/base.sipl. Its sidebar has Tables followed by Collections, each with a database filename and its own create/open actions. The Databases screen opens or creates either storage type. Global Open database detects Base BSON files by their SQLite application ID and routes them to Collections; other SQLite files open in Tables. A .nsdb file is a BSON document database, while .db / .sqlite files retain the relational workflow. Both connections remain open, and switching objects preserves the other view.

Selecting a table exposes Browse, Structure, SQL, Insert and Operations. Selecting a collection exposes Documents, Fields, Indexes, Aggregation and Collection tools:

• Documents: JSON filter, sort, page navigation, document list, expandable typed field inspection and JSON view. Insert/Edit opens a JSON editor with validation.

• Fields: presence/type analysis of up to 200 matching documents; this is a sample, not schema validation.

• Indexes: inspect, create or drop single-field indexes (including unique indexes).

• Aggregation: count, numeric count, sum, average, minimum and maximum, optionally grouped by a scalar field, using the current filter.

• Collection tools: rename/drop, atomic JSON/BSON imports, export the current JSON page or all matching BSON documents, and Save database as for a complete persistent snapshot, including a temporary in-memory database.

Destructive actions and discarding dirty JSON require an in-app confirmation; background controls are disabled while it is open. Failed filters leave the last successful result visible. Saving JSON preserves _id; importing duplicate IDs fails atomically.

Checks: sipl run tests/test_nosql.sipl, sipl run examples/projects/Base/base.sipl --test, --test-ui (headless input simulation), and --test-io=NEW_PATH_PREFIX. For document screenshots use --sample --collections, optionally followed by --tab=Fields, --tab=Indexes, --tab=Aggregation or --tab=Editor.

Mat: native linear algebra#

mat wraps reference Netlib BLAS and LAPACK in one namespace, with two native objects: Matrix for dense numeric data and LUFactor for reusable solves. This is a CPU linear algebra foundation for simulations. It does not provide ODE/PDE integrators, sparse matrices, FFTs, probability distributions, or a complete binding to every Netlib routine.

For an interactive desktop workbench, run sipl run examples/projects/Flux/flux.sipl. Flux combines worksheets, a command window, a matrix workspace, and figures using this API.

fn start() {
    let stiffness: Matrix = mat.matrix([[4, 1], [1, 3]])
    let load = mat.vector([1, 2])
    let displacement = stiffness.solve(load)
    print(displacement.toVector())
    print(stiffness.multiply(displacement).add(load, -1).norm())

    let factor: LUFactor = stiffness.factor()
    print(factor.solve(mat.vector([3, 5])).toVector())
    print(factor.rcond())
    factor.close()
}

Data and ownership#

Matrices use contiguous, column-major IEEE Float64 buffers. Constructors accept ordinary row-oriented SIPL lists and convert once. mat.vector creates an n×1 matrix; vectors can also have one row. Numeric inputs must be finite. Integers are converted to Float64 and may round beyond its exact integer range.

Calculations preserve their inputs and return independent objects. set, scaleInPlace, and functions ending in Into explicitly mutate their target. Matrices have no shared buffer views, so closing an input cannot invalidate a result. LU factors own an independent copy of the original matrix.

Each matrix/workspace is capped at 16,777,216 elements (128 MiB). Factorizations can require several buffers; 128 MiB is a per-buffer limit, not a total process budget. Dimensions must be positive. All indexes are zero-based. close() is idempotent; unreachable buffers are also freed and counted by the GC. Closing an LU factor releases its references to the private factor buffers.

Input validation and LAPACK INFO errors raise catchable runtime failures. Singular LU factors can be inspected (singular, zero rcond()), but cannot solve or invert. Ill-conditioned systems may produce a result with poor forward accuracy; inspect rcond() and the residual. Arithmetic may overflow to infinity or NaN; check isFinite. LAPACK solvers/decompositions reject nonfinite input.

Construction and basic operations#

BLAS operations#

Options are alpha (default 1), transposeA (default false), and, for GEMM, transposeB. Only the Into variants accept beta (default 0). Output buffers must have exactly the required shape and must not alias either multiplication input. Invalid shapes, flags, and options fail before the target changes.

let a = mat.matrix([[1, 2, 3], [4, 5, 6]])
let gram = mat.zeros(2, 2)
mat.gemmInto(gram, a, a, {"transposeB": true})
print(gram.toList()) // [[14, 32], [32, 77]]

Solvers and reusable LU factors#

Least squares returns {"solution": Matrix, "rank": Int, "singularValues": Matrix}. Its default rcond=-1 uses machine precision; explicit thresholds are in [0,1].

Decompositions and spectral problems#

Symmetric routines validate symmetry with relative tolerance 1e-12, then use the lower triangle. Symmetric eigenvalues are ascending; singular values descending. Eigenvectors are columns; their signs/phases and bases in repeated eigenspaces are not unique.

General eigen returns real, imag, vectorsReal, and vectorsImag matrices. Column j represents eigenvalue real[j] + i*imag[j] and its complex eigenvector. Real and imaginary parts are explicit because Matrix currently stores real Float64 data; arbitrary complex matrices and Float32 are not exposed yet.

Builds, provenance, and graphics isolation#

The pinned C distribution is Netlib LAPACK 3.12.0, as reported by mat.lapackVersion, from OpenBLAS 0.3.34's maintained translations. Reference BLAS sources from that same archive were translated with Netlib f2c 20240504. This does not link OpenBLAS's optimized kernels or threading runtime. Netlib's upstream Fortran release at integration (2026-10-16) is 3.12.1; we do not claim that the C translation is that patch release. Exact source URLs, checksums, licenses, and ABI handling are in the dependency pin.

Ordinary builds are offline and use only the existing C compiler and CMake. A separate static archive contains the dependency closure of the public API; other upstream C routines remain vendored for future wrappers. Only referenced objects are linked into SIPL. There are no runtime BLAS/Fortran DLL requirements. Source size, archive size, and executable growth are different measurements. Measured against the previous runtime with the same build profile, the current wrapper adds 407,040 bytes (0.39 MiB) on Windows and 449,416 bytes (0.43 MiB) on Linux. The Windows imported-DLL set is unchanged. These are measurements of this binding's linked routine set, not a promise for every future Netlib binding.

Math uses private CPU buffers and makes no graphics/GPU/compute API calls. Existing graphics backends, shader options, and type IDs are retained. Tests include numerical reconstruction/residuals, normal and GC-stress execution, packaging, existing Vulkan scenes, and math while rendering a scene.

This backend prioritizes a portable reference implementation. Large dense simulations can benefit from optimized BLAS in a future backend; reference BLAS should not be described as the fastest implementation. Simulation timestepping, boundary conditions and stability still belong to the application.

Generic 3D Graphics#

SIPL exposes a generic retained graphics API. Vulkan runs on Windows and Linux (X11/XWayland); the Apple implementation uses native Metal. Backend handles, flags, synchronization objects, and prefixed constants never appear in SIPL programs.

First Scene#

fn start() {
    let app = graphics.open("SIPL 3D", 960, 640)
    let scene = app.scene()
    let camera = scene.camera({
        "position": [3.2, 2.4, 4.2],
        "target": [0, 0, 0],
        "fov": 58,
    })
    let cube = scene.box({
        "size": 1.6,
        "color": "#4C9AFF",
    })

    app.background("#091426")
    while app.frame() {
        cube.turn(app.deltaTime * 0.7, app.deltaTime, 0)
        app.draw(scene, camera)
    }
}

Run it:

.\build.bat
.\build\sipl.exe run examples\graphics_cube.sipl

Escape or the window close button exits.

Application#

graphics.open(title, width, height) returns a GraphicsApp. One graphics application may be open at a time.

The module-level immediate drawing functions remain as a compatibility layer. New applications should use retained app, scene, object, and camera values.

Scenes And Standard Primitives#

let floor = scene.box({
    "position": [0, -1, 0],
    "rotation": [0, 0.2, 0],
    "size": [8, 0.25, 8],
    "color": "#25334D",
    "visible": true,
})

size may be one positive number or a three-number list.

Mutation methods return the object so calls can be chained.

The standard primitives all use shared indexed meshes:

let floor = scene.plane({"size": [10, 1, 10]})
let ball = scene.sphere({"position": [-2, 1, 0]})
let column = scene.cylinder({"position": [0, 1, 0]})
let marker = scene.cone({"position": [2, 1, 0]})
let player = scene.capsule({"position": [0, 1, 2]})

Available constructors are box, plane, sphere, cylinder, cone, and capsule. Each accepts position, rotation, size, color, material, and visible.

Custom Meshes#

Create reusable indexed or non-indexed triangle meshes:

let pyramidMesh = graphics.mesh({
    "positions": [
        [0, 0.8, 0],
        [-0.7, -0.5, 0.7],
        [0.7, -0.5, 0.7],
        [0, -0.5, -0.7],
    ],
    "indices": [0, 1, 2, 0, 2, 3, 0, 3, 1, 1, 3, 2],
})

let pyramid = scene.object({
    "mesh": pyramidMesh,
    "color": "#FF9500",
})

positions, normals, and textureCoordinates accept nested vectors or flat number lists. Normals are optional and generated from the triangles when omitted. Invalid indices, incomplete triangles, zero normals, degenerate geometry, and mismatched attribute counts fail explicitly.

Meshes expose vertexCount, indexCount, and remove(). A mesh cannot be removed while an object uses it. Use counterclockwise winding when looking at an outward-facing triangle. The Vulkan backend culls back faces; repeat a triangle with reversed winding when a surface must be visible from both sides.

Materials And Textures#

The standard material model uses glTF-compatible metallic/roughness PBR:

let tiles = graphics.texture("assets\\tiles.png")
let floorMaterial = graphics.material({
    "color": "#FFFFFF",
    "texture": tiles,
    "metallic": 0.05,
    "roughness": 0.82,
})
let floor = scene.plane({
    "size": [12, 1, 12],
    "material": floorMaterial,
})

Material options are:

• color

• texture

• metallic from 0 to 1

• roughness from 0 to 1

• emissive

• emissiveStrength

• unlit

object.material(material) changes an existing object. Textures decode PNG and JPEG files, upload into device-local image storage, use linear sampling, and expose width, height, and remove(). In-use materials and textures cannot be removed.

Scene Lighting#

Scenes retain ambient, directional, point, and spot lighting:

scene.ambient({"color": "#8AA8CC", "intensity": 0.12})

let sun = scene.sun({
    "direction": [-0.4, -0.8, -0.6],
    "color": "#FFF3DD",
    "intensity": 2.2,
})
let lamp = scene.pointLight({
    "position": [2, 3, 1],
    "color": "#FFB060",
    "intensity": 12,
    "range": 15,
})
let spot = scene.spotLight({
    "position": [-2, 4, 2],
    "direction": [0.3, -1, -0.2],
    "intensity": 10,
    "range": 14,
    "innerAngle": 0.35,
    "outerAngle": 0.55,
})

The renderer uses up to four active lights per scene. A scene with no explicit lights receives a useful default sun. Lights support position, direction, and remove().

GLB And glTF Models#

Load a complete static model directly into a scene:

let model = scene.model("assets\\robot.glb")
print(model.objectCount)

The loader validates GLB/glTF data, buffers, indexed triangle primitives, vertex positions, normals, texture coordinates, node transforms, embedded or external images, and metallic/roughness materials. Missing normals are generated. model.remove() removes its objects and releases the imported meshes, materials, and textures.

Toolkit GLB And glTF Preview#

The Toolkit opens .glb and .gltf files in model tabs alongside images and video. Drag inside the viewport to orbit; use the mouse wheel to zoom and Reset view to restore the camera. The inspector offers:

• Hierarchy: expand object branches and check or uncheck their visibility. Hiding a parent hides its descendants while preserving their own checkboxes.

• Textures: inspect image sources, dimensions, and a thumbnail.

• Materials: inspect base colors, texture assignments, metallic/roughness values, alpha mode, and sidedness.

• Geometries: inspect mesh vertex, triangle, primitive, and morph counts.

• Animations: select a named clip or return to the rest pose. The viewport also provides an animation dropdown, play/pause, a scrubber, loop, and speed.

• Info: inspect the format, generator, and model totals.

Small windows expose these sections through an Inspector menu. Animation pauses when leaving a model tab. Closing or reloading releases native resources.

The preview uses native C software rendering into the Toolkit canvas without a second graphics window or Vulkan. It supports triangle geometry with base-color materials and textures, node transforms, skeletal skinning, and position morphs. Animation supports STEP, LINEAR (including quaternion interpolation), and CUBICSPLINE channels. It does not provide the graphics module's full PBR lighting. Relative local buffers and PNG/JPEG/WebP textures, embedded resources, and data URIs work in .gltf files; network resource loading is disabled. Local assets must stay beneath the selected model's directory: absolute paths, parent traversal (including URI-encoded traversal), symbolic links and Windows reparse points are rejected. Keep related buffers and textures in that directory or its subfolders. The EXT_texture_webp extension resolves WebP sources for rendering and material inspection, including files without a standard PNG/JPEG texture source. WebP texture sources up to 16 megapixels decode directly to a preview within the 4-megapixel image budget; ordinary Image decode limits stay unchanged.

Limits include a 64 MiB input file, 256 MiB model memory budget, 4,096 nodes, 128 clips, 32 images, 1,024 primitives, and 500,000 vertices and triangles. Each software render is also limited to a one-megapixel cache and 16,777,216 triangle bounding-box pixel visits. Models that exceed the work budget produce a recoverable preview error instead of keeping the UI busy indefinitely.

Applications can use the same UI primitive directly:

let model = ui.loadModel("assets/robot.glb")
if model.animationCount > 0 { model.pose(0, 0.5) }
ui.open("Model", 960, 640)
while ui.frame() {
    ui.drawModel(16, 16, ui.width - 32, ui.height - 32, model,
        {"yaw": 0.65, "pitch": 0.35, "zoom": 1.0})
}
model.close()

ui.drawModel takes optional finite yaw (-1000..1000), pitch (-1.5..1.5), zoom (0.1..20), and background (#RGB, #RRGGBB, or #RRGGBBAA) values. model.pose(clipIndex, seconds) samples an animation; pose(-1, 0) restores the rest pose. Playback timing belongs to the caller. setVisible(nodeIndex, Bool) and resetVisibility() control hierarchy visibility. nodeTransform(nodeIndex) returns a column-major world matrix. texture(imageIndex) returns an independent Image that the caller closes.

Model exposes nodes, animations, materials, textures, and geometries as lists of metadata maps, along with format, generator, nodeCount, animationCount, materialCount, textureCount, skinCount, meshCount, vertexCount, triangleCount, visibleTriangleCount, and closed. close() is idempotent. ui.modelPreviewVersion >= 2 identifies this API.

Cameras#

let camera = scene.camera({
    "position": [0, 2, 6],
    "target": [0, 1, 0],
    "up": [0, 1, 0],
    "fov": 60,
    "type": "perspective",
})

Camera methods are position, target, lookAt, up, moveForward, moveRight, and moveUp. camera.up(x, y, z) sets the view's up direction and returns the camera. It accepts a finite, nonzero vector and normalizes it. The optional up creation field defaults to [0, 1, 0]; a banked aircraft can supply its body up vector. If up is parallel to the viewing direction, the renderer uses a perpendicular fallback to keep the view valid. moveRight uses world up; moveUp moves along world Y. The type option currently accepts "perspective" and defaults to it when omitted. A camera belongs to its scene and cannot draw another scene.

Orbit Camera#

For model viewers, editors, and scene inspection:

let camera = scene.orbitCamera({
    "target": [0, 0, 0],
    "yaw": 0.65,
    "pitch": 0.4,
    "distance": 6,
    "fov": 58,
    "type": "perspective",
})

while app.frame() {
    camera.orbitControls(app)
    app.draw(scene, camera)
}

orbitControls uses left-button dragging to orbit and the mouse wheel to zoom. Pitch is clamped before the camera can flip over, and zoom keeps a safe minimum distance from the target.

Scripted controls are also available:

Input#

Input state is updated by app.frame():

if app.keyDown("w") {
    camera.moveForward(app.deltaTime * 4)
}
if app.keyPressed("space") {
    print("Space was pressed this frame")
}

Available input:

• app.keyDown(key) and app.keyPressed(key);

• letters, digits, space, escape, enter, tab, arrows, shift, control, and backspace;

• app.mouseDown(button) and app.mousePressed(button) for left, right, and middle;

• app.mouseX, app.mouseY, app.mouseDeltaX, app.mouseDeltaY, and app.mouseWheel.

Down means currently held. Pressed means the transition happened during the current frame.

For native controller input, call gamepad.poll() once inside this frame loop. See gamepads for button presses/releases, sticks, triggers, and the standalone C interface.

Conventions#

• Positive Y is up.

• Positions and sizes are world units.

• Rotations are radians.

• Colors are #RRGGBB.

• Common creation uses named option maps.

• Nouns create resources; short verbs change or draw them.

• Queues, descriptors, command buffers, image views, render passes, and presentation chains are backend details and absent from SIPL.

Ownership And Safety#

Graphics resources are opaque typed values, not integer handles. Every resource records its owning application and slot generation. Removed resources, resources from another scene, and resources retained after close fail explicitly.

Closing the application waits for outstanding GPU work and releases all native resources. Current bounds are 32 scenes, 4096 objects, 128 cameras, 256 meshes, 256 materials, 256 textures, 128 lights, and 64 models per application.

Scene Inspection And Smoke Checks#

Run tests/graphics/scene_smoke.sipl normally to inspect its scene. Drag with the left mouse button to orbit, scroll to zoom, and close the window or press Escape to exit. Add --smoke to run its resource checks and close after four frames:

.\build\sipl.exe run tests\graphics\scene_smoke.sipl
.\build\sipl.exe run tests\graphics\scene_smoke.sipl --smoke

Keep calling app.frame() while a graphics window is open. A console input() pause before or inside that loop blocks window event processing and rendering. Use window input such as app.keyPressed("escape") to exit a scene viewer.

On Windows, run the desktop resize regression with:

python tests/graphics_windows_tests.py build/sipl.exe

It verifies rendered-frame progress across maximise/restore/minimise and closes while minimised, in normal and GC-stress modes. A test runtime linked with tests/graphics/resize_faults_win32.c can be supplied as --fault-runtime to force out-of-date swapchain creation and a temporary zero-sized surface. Resize recovery retries these transient conditions while still handling native close and restore events.

Automatic Batching#

Object submissions are accumulated in a dynamically sized mapped instance buffer and grouped by mesh and texture. The backend emits one indexed instanced draw per used mesh/texture pair instead of one GPU command per object. Retained draws also reject objects outside the camera frustum before submission.

Run the stress benchmark with an optional object count and frame limit:

.\build\sipl.exe run examples\graphics_benchmark.sipl
.\build\sipl.exe run examples\graphics_benchmark.sipl 200000 60

The default is 200,000 independently rotating, deterministically colored objects. It reports FPS and object transforms per second. This remains a deliberately severe CPU-side test: SIPL still executes the loop and computes every object's transform each frame, while automatic batching removes the GPU-command bottleneck.

Platform Requirements#

Windows/Linux graphics applications require a compatible Vulkan graphics driver. Windows users do not need the Vulkan SDK. Linux desktop applications also require an X11 or XWayland session. macOS builds use Apple Metal and a Cocoa window, enabled in every standard build; no Vulkan SDK or translation library is required. Native Mac execution still needs Mac acceptance. Check graphics.available before offering an optional 3D feature; device or display failures are reported when graphics.open() is called.

Keep the shader files distributed with SIPL beside sipl and packaged graphics executables when moving them to another folder. Programs locate these files relative to the executable rather than the working directory. This shader deployment requirement applies to Vulkan builds. The Mac executable embeds its Metal shader source and compiles it through the system Metal framework on open.

Next Layers#

The remaining production layers are multiple frames in flight, transparent object sorting and alpha modes, shadow maps, glTF animation and skinning, and generic shader, target, and pass APIs for advanced work.

The high-level API remains the normal path. Advanced controls will use generic resource and pass terminology rather than exposing the backend API.

Native audio playback#

SIPL's native audio module provides bounded WAV, Ogg/Vorbis and MP3 decoding, portable output selection, and one-shot/loop playback. Use audio.stream(path) for music, audiobooks and long local files. It reads and decodes chunks in a C worker into a fixed one-second PCM queue. The audio callback consumes that queue without file I/O, allocation, locks, or SIPL callbacks. audio.open() and AudioApp.load/decode remain available for predecoded short effects and mixing. Native microphone recording and bounded offline synthesis/effects/mixdown are available as described below. Network audio streams are not included. Video playback is described separately in VIDEO.md.

fn start() {
    let output: AudioApp = audio.open()
    let sound: AudioSound = output.load("notification.wav")
    sound.volume = 0.5
    sound.play()
    while sound.playing { time.sleep(time.seconds(0.01)) }
    sound.close()
    output.close()
}

Opening/enumerating hardware and reading/decoding files are synchronous control work. AudioApp.load predecodes a clip before an interactive/async playback loop; audio.stream provides chunked file playback. Playback itself runs independently of SIPL execution. No sound is played until play() is requested. Closing the application stops all owned sounds.

Streaming music and long files#

fn start() {
    let track: AudioStream = audio.stream("album/song.mp3")
    track.volume = 0.7
    track.play()
    while track.playing { time.sleep(time.seconds(0.05)) }
    track.close()
}

The sleep above keeps the console program alive. It does not feed the decoder: playback and native looping continue even when SIPL is busy. GUI applications use the same controls from their ordinary event loop. Prism is a complete single-file music/video player using this API.

audio.stream(path: String, options?) -> AudioStream opens a local, Unicode-aware regular file, validates its format and duration, primes a small block and starts a native decoder worker. Playback starts paused. Options accept headless: Bool, with the same SIPL_AUDIO default as audio.open. Hardware uses the system default device; custom backend/device selection is currently available through AudioApp only. Initialization and seeks are synchronous control operations; ongoing chunk reads and decoding run in C, never in a SIPL refill loop.

Supported formats are RIFF/WAVE, MP3 and Ogg/Vorbis. A file may contain up to 2 GiB and 24 hours of audio. These streaming limits are independent of the 16 MiB encoded / 64 MiB PCM clip limits. PCM queue and decode-block storage are always 416,768 bytes (about 407 KiB), with up to 32 MiB of separate decoder scratch, plus device/native bookkeeping. Encoded input is read from disk in chunks. At most eight streams may remain open. audio.info() exposes streaming, openStreams, maxStreams, maxStreamFileBytes, maxStreamSeconds and streamPCMBytes alongside the clip/mixer limits.

Streams expose play, pause, stop, seek(Duration) and idempotent close, with the same control meanings as sounds below. loop, volume (0..4), and pan (-1..1) are writable. Looping refills across the boundary in C; disabling it flushes any prefetched next-cycle audio. Read-only properties are playing, closed, headless, position, duration, frames, sampleRate (48000), channels (2), and pcmBytes (zero after close). Duration may be refined by a few frames at EOF because of resampler rounding. Positions represent consumed PCM, so buffering does not advance the playback clock. Seeking to the end stops; play() at the end restarts the track. A stopped/failed device reports a catchable error rather than silently switching output.

Explicit headless streams provide render(frames) and advance(Duration) for deterministic tests, using the same native decoder and PCM consumer. Their output format, request bounds and fractional-frame carry match the corresponding AudioApp APIs below. In headless mode, these explicit operations advance time; sleeping does not. Decoder errors can surface during a later control/property query or render, since streaming does not validate the entire file upfront.

Close joins the decoder worker and output device before freeing memory. Native state is cleaned up on GC and ordinary process exit. The full decoder budget is reserved in GC pressure on the control thread; workers never access the VM. Closing is recommended when changing tracks rather than waiting for collection.

Output and discovery#

• audio.info() -> Map: compiled codecs/backends, canonical PCM format and hard resource limits. Compiled support does not imply an installed output device.

• audio.devices(options?) -> List[Map]: enumerate outputs on the selected backend; each entry has id, name, backend and isDefault.

• audio.open(options?) -> AudioApp: start a stereo output app. Options are headless: Bool, sampleRate: Int (8000..192000, default 48000), maxSounds: Int (1..64, default 32), maxPCMBytes: Int (8..256 MiB, default 64 MiB), backend: String (default "auto") and device: String.

Device enumeration accepts only headless and backend. Select a returned ID with the same backend:

let outputs = audio.devices({"backend": "wasapi"})
let output = audio.open({"backend": "wasapi", "device": outputs[0]["id"]})

IDs are opaque and backend/session-specific, not persistent identifiers. Refresh enumeration after device changes. auto uses the platform's default output; an unavailable backend/device fails catchably and never becomes headless silently. Output stop/interruption notifications cause playback/position queries and controls to fail catchably instead of waiting forever on a lost callback; check the device and reopen the app. Automatic device recovery is not a beta feature. The null backend is not compiled. Hardware unplug/recovery, latency qualification, and device interoperability need testing on the actual release targets.

The compiled output backends are WASAPI and WinMM on Windows; ALSA and PulseAudio on Linux; Core Audio on macOS. Linux loads installed backend libraries at runtime. Headless mode needs no device libraries. Windows and Linux are the beta build/test targets; the Core Audio path is present but not hardware-qualified in this workspace.

SIPL_AUDIO=headless explicitly requests headless defaults; SIPL_AUDIO=device requests ordinary defaults. A supplied headless option overrides the environment. Any other nonempty environment value fails. Headless apps cannot also select a hardware backend/device. Enumeration in explicitly headless mode returns one synthetic PCM output.

Sound loading and controls#

app.load(path: String, options?) -> AudioSound uses the bounded, Unicode-aware binary filesystem reader. app.decode(data: Bytes, options?) -> AudioSound decodes immutable bytes without retaining the encoded input. Both accept maxSeconds: Int (1..600, default 120) and maxPCMBytes: Int (8..64 MiB, default 64 MiB). The duration, PCM, app and global limits are all enforced; increasing one does not bypass the others. Decoding converts to the app sample rate and interleaved float32 stereo off the real-time thread.

Mixing sums voices then clamps output to [-1,1]. Hardware control commands apply in order on an output callback; playback position snapshots can lag by one callback. playing reflects queued play/pause/stop requests immediately until the callback acknowledges them, then reflects actual completion. Control setters report the requested value immediately. A full 256-command queue fails catchably instead of growing or blocking the callback; retry after an output callback. Closing a sound does not require a queue slot.

App read-only properties are closed, headless, sampleRate, channels, backend, device, framesRendered, elapsed: Duration, soundCount and pcmBytes. app.close() is idempotent. Playback/decode/render/control calls on a closed owner fail; sound closed becomes true even if its own close() was not called. Immutable sound metadata remains readable after close; playing becomes false and position reads zero.

Deterministic headless PCM#

let output = audio.open({"headless": true, "sampleRate": 48000})
let sound = output.load("notification.wav")
sound.play()
let pcm: Bytes = output.render(480)
assert(pcm.length == 3840) // 480 frames * 2 channels * 4 bytes
output.advance(time.seconds(0.01))
sound.close()
output.close()

Headless is a real decoder/mixer with no hardware thread, not a playback stub. render(frames: Int) -> Bytes advances the mixer and returns canonical little-endian IEEE-754 float32 stereo samples. Each request accepts 0..2097152 frames, at most 16 MiB. advance(duration: Duration) -> Int advances without allocating PCM output, accepting 0..60 seconds per call. Fractional frames carry across advance calls so splitting a duration does not discard time. render adds whole frames without discarding that fractional carry. Both calls are headless-only. Controls apply immediately in headless mode.

elapsed and sound position use rendered sample counts, not a wall clock. Sleeping, async timers or test.advanceTime() do not automatically advance a headless app. This makes tests repeatable even when the wall clock is changed.

Bounds, codecs and lifetime#

For predecoded clips there may be at most 8 open apps, 64 loaded sounds per app, 16 MiB encoded input, 64 MiB decoded PCM per sound, 256 MiB PCM per app and 512 MiB decoded PCM across apps. Decoder scratch heap allocations are capped at 32 MiB. Decode temporarily holds the bounded encoded input, scratch and PCM allocation; render can use a bounded 16 MiB native buffer plus its 16 MiB returned Bytes. These are intentional predecode limits. Use audio.stream for long music playback.

Supported containers are ordinary little-endian RIFF/WAVE; one complete, CRC-validated Ogg/Vorbis logical stream (1..8 channels); and complete standard MPEG-1/2/2.5 Layer III MP3 frames, CBR or VBR, with ID3v2/ID3v1/APE metadata. RF64/RIFX, Ogg/Opus, chained/multiplexed Ogg, legacy Vorbis floor-0 files and free-format MP3 are not beta codecs. RIFF chunk boundaries, Ogg page sequence, checksums/EOS and MP3 frame/tag bounds are checked before decoding. Malformed, truncated, empty, oversized and non-finite-PCM inputs fail catchably. File access failures keep the normal filesystem failure kind; decoder/control failures use the runtime failure kind. Unknown option keys fail, including embedded NULs.

The callback owns mixer state; all callback-visible atomics must be lock-free. It does no allocation/free, locks, file I/O, user callbacks or SIPL/GC access. Immutable native PCM is published through a bounded single-producer/consumer command queue. Closing/GC requests generation-tagged retirement; only the control thread frees/reuses PCM after callback acknowledgement. Native refcounts retain owner state independently of GC sweep order. Closing an app or normal process exit stops/joins hardware callbacks before reclaiming PCM. Explicit close remains recommended: real-device retirement may wait for the next output callback, and GC is not an immediate resource-management policy. Persistent backend/PCM storage is charged to GC pressure without reallocating callback-visible pointers; accounting changes and reclamation stay on the control thread. Closing an app immediately releases heavy backend storage even when a closed wrapper or sound metadata is retained.

Tests and dependencies#

Run sipl test tests/test_audio_stream.sipl --jobs 2, python3 tests/audio_stream_tests.py build/sipl, sipl test tests/test_audio.sipl --jobs 2 and python3 tests/audio_tests.py build/sipl (use build/sipl.exe on Windows). Tests cover real WAV/OGG/MP3 PCM, an independent WAV numeric oracle, sine-wave spectral checks, malformed/truncated containers, bounds, controls, mock-clock independence, normal/GC ownership, slot reuse, recipes and standalone packaging. The Python driver also builds native concurrent callback/control/retirement stress tests if a C compiler is available.

On supported Unix hosts, SIPL_AUDIO_SANITIZE=1 enables ASan/UBSan for the native stress driver. Under ptrace-based tool sandboxes, LeakSanitizer may be unavailable; ASAN_OPTIONS=detect_leaks=0 preserves address/undefined-behavior checks, not a claim of leak-sanitizer coverage. Physical output has not been verified by these headless tests.

The runtime vendors miniaudio 0.11.25 and stb_vorbis 1.22, with upstream revision, source hashes and licenses in third_party/miniaudio/README.md and third_party/stb/README_AUDIO.md. File audio uses those native decoders; no codec subprocess is required. Video uses its separate static FFmpeg backend. Original regression fixture generation is documented in tests/fixtures/audio/README.md. See also sipl api audio --json, sipl api AudioApp --json, sipl api AudioSound --json, sipl api AudioStream --json, and examples/recipes/audio.sipl.

Native recording and offline production#

audio.record(path, options?) -> AudioRecording creates a new PCM16 WAV without overwriting an existing file and starts capture from the system default microphone. Options: sampleRate 8000–192000 (default 44100), channels 1–2 (default 1), maxSeconds 1–600 (default 120), and explicit headless (default false). Microphone capture does not inherit a headless playback environment. Two captures may be open concurrently. No input device selection API is provided in this first implementation; use system input settings.

Read active, closed, headless, sampleRate, channels, frames, position: Duration, and peak (latest captured block, 0–1). stop() / close() join the device and writer and finalize the WAV header. Duration limits stop capture automatically; call stop/close to finalize and release resources. GC and ordinary process exit also finalize captures. Abrupt process termination cannot guarantee a finalized header. Disk/input failures report an error at stop; a partial WAV is retained. Physical device timing and recovery need hardware qualification on each platform.

The capture callback copies finite/clamped PCM into a fixed 65,536-frame ring; only the writer thread performs WAV I/O. Producer overruns fail explicitly. Headless captures expose write(Bytes) for deterministic tests: interleaved float32 little-endian input, frame-aligned, at most 2 MiB per call. This method is unavailable on hardware captures. A record error is never replaced by silence or a synthetic input device.

audio.importFile(source, target) -> Map decodes a WAV/MP3/Ogg-Vorbis local file, at most 120 seconds, into a new stereo PCM16 WAV at 48 kHz. It returns file, seconds, peaks (128 amplitude maxima), peak and tracks (empty for imports). It uses the existing bounded file decoder. WAV input formats retain the decoder's existing supported format rules; malformed inputs fail.

audio.renderSession(session, target) -> Map creates a new stereo PCM16 WAV at 48 kHz. The synchronous API is intended for CLI jobs or a SIPL worker process, not an audio callback. Limits are 12 tracks, 16 bars, 40–240 BPM, 120 seconds, 32 clips per track, and 128 metering bins. Buffers are bounded to three stereo float32 session arrays (about 132 MiB at 120 seconds), plus existing decoder scratch. Native synth/mixer rendering does not open a hardware device.

A session Map has tempo, whole bars, master (dB, -48..6) and tracks. Each track has a unique positive Int id, an instrument kind from audio.instruments() or audio, gain (-48..12 dB), pan (-1..1 balance), mute, solo, note (MIDI 0..127), steps (16 velocities, 0..1), fx, and clips. Effects are low, mid, high (-18..18 dB) and compress, drive, delay, reverb (0..1). Missing numeric controls take their native defaults; required track/clip collections must still be supplied. A clip supplies a file path, start, source offset and length, measured in beats. Files are decoded only when rendering an audible audio track. Synth patterns repeat for every bar.

Tracks optionally supply mode (steps or notes, default steps) and notes (at most 512 Maps). Each note has start (0..64 beats), length (.0625..16 beats), whole MIDI pitch (0..127), and velocity (0..1). In notes mode the renderer plays the note list instead of the repeating steps. Notes may overlap for chords; releases can extend past the gate and are cropped at the session end. Notes outside the current arrangement are retained but not rendered. A track may accumulate at most 600 seconds of audible note gates plus a conservative 1.1 seconds of release per note, bounding offline voice work. Invalid pitches, velocities, modes and excess note counts fail before writing output.

audio.instruments() -> List[Map] enumerates 64 native presets with stable id, display name and category. These are additive/FM/noise synthesis definitions, not sample files. The original kick, snare, hat, bass and chord pad remain available. Acoustic names describe synthesized approximations. Definitions are in src/audio_instruments.def; voices are in src/audio_synth.inc.

audio.synthesize(kind, pitch, options?) -> Bytes produces a new in-memory stereo PCM16 WAV at 48 kHz. kind is a catalog ID; pitch is a whole MIDI 0..127. Options are seconds (.03..4 gate seconds, default .35) and velocity (0..1, default .8). The preset's release follows the gate, up to 1.1 additional seconds. Decode these Bytes with an AudioApp to audition them without disk files:

let app = audio.open()
let sound = app.decode(audio.synthesize("piano", 60))
sound.play()

Keep the app/sound alive while playing; close them after use. Synthesis is synchronous and bounded to less than 4 MiB of temporary native buffers. It does not open a device, and silent velocity produces silence. The per-note audition has no channel effects. audio.info().instrumentVersion is 2 for this API set.

EQ uses the bundled miniaudio low shelf at 220 Hz, peak at 1100 Hz (Q .7), and high shelf at 4 kHz. Compression blends a -18 dB / 4:1 envelope compressor (5 ms attack, 80 ms release); drive is tanh saturation. Delay adds four decaying eighth-note taps; room reverb adds seven short stereo diffusion taps. Effects are applied before per-track gain/pan and summing. The returned master peak is measured before [-1,1] output clamping; peaks and tracks contain actual rendered amplitude bins (track bins are after track gain/pan).

Output Maps include file, seconds, peak, peaks and a tracks Map keyed by decimal track ID. Render/import never overwrite an existing target. Render to a temporary file, then publish it after success when an application needs replacement semantics. File write failures may leave an identifiable partial target. No audio effect callbacks enter SIPL or the GC.

Copperline Studio demonstrates these APIs without Python or an external audio processor. It is an initial workstation, with rendered effects and unqualified hardware overdub timing, not a VST host or sample-accurate production DAW.

Gamepads#

SIPL's gamepad module reads controllers in native C, independently of UI, graphics, Vulkan, and SDL. Windows uses XInput plus native WinMM input for DualShock 4 controllers; Linux uses nonblocking evdev. Other platforms report gamepad.info()["supported"] == false and return empty input state. No new build dependencies or link libraries are required.

Call gamepad.poll() once per game tick, then read any controls you need:

while app.frame() {
    gamepad.poll()
    if gamepad.justConnected(0) {
        print(gamepad.name(0))
    }
    if gamepad.buttonPressed(0, "a") {
        print("Jump")
    }
    let horizontal = gamepad.axis(0, "leftX")
    let vertical = gamepad.axis(0, "leftY")
    camera.moveForward(-vertical * app.deltaTime * 4)
    app.draw(scene, camera)
}
gamepad.close()

app.frame() does not poll controllers automatically. The module also works with ui.frame() loops and programs without a window. Polling never waits for input. It uses fixed native storage; reading buttons and axes allocates no SIPL collections.

Indices are 0..3. Unknown controls and invalid indices raise catchable errors. State starts empty until the first poll. Edges remain available through repeated reads until the next poll. A controller connected with a held button reports that button as pressed. Disconnecting releases all held buttons and zeros all axes. Calling another input function after close() creates fresh empty state; native handles are also released at process exit.

Buttons use Xbox-style names for their physical positions:

Axes are leftX, leftY, rightX, rightY, leftTrigger, and rightTrigger. Sticks range from -1 to 1, positive right/down on both platforms. Triggers range from 0 to 1. Deadzones must be finite numbers greater than or equal to 0 and less than 1. Values inside the deadzone become zero; the remaining range is rescaled to reach 1 at full deflection. Pass 0 for raw normalized values. Deadzones are applied per axis; games can read raw axes to implement a circular stick deadzone.

Windows supports XInput-compatible controllers in the four XInput player slots. DualShock 4 controllers that report Sony's USB/Bluetooth v1, v2, or wireless adapter IDs (054c:05c4, 054c:09cc, 054c:0ba0) use the native WinMM joystick API and occupy available slots. This includes compatible PS4 controllers that report those IDs. No controller emulation software is required. Windows loads the system XInput and WinMM DLLs dynamically; backend is xinput+winmm. Other DirectInput layouts and rumble are not currently supported. Both Windows paths provide snapshots, so transitions entirely between polls cannot be observed. Poll at your game rate. PS4 connections are scanned once per second.

On PS4 controllers, a is Cross, b is Circle, x is Square, y is Triangle, back is Share, and start is Options. Shoulders are L1/R1, stick buttons are L3/R3, and trigger axes are L2/R2. The D-pad includes diagonal directions. If an XInput player reconnects into a PS4's slot, the old slot publishes a disconnect before replacement, and the PS4 moves to a free slot when available. Games should select a connected slot and reselect after a disconnection.

Linux discovers standard gamepads among /dev/input/event0 through event63, scanning for connections at most once per second. Slots stay assigned while a controller remains connected and may be reused after disconnection. The user running SIPL must have read access to the controller's evdev node; devices absent from the environment, including those not forwarded into WSL, cannot be read. The backend follows the Linux gamepad mapping, including digital or hat D-pads, standard stick axes, analog triggers on ABS_Z/ABS_RZ or ABS_HAT2Y/ABS_HAT2X, and digital trigger fallbacks. Legacy devices with nonstandard mappings need driver configuration. Linux preserves button edges from multiple input packets within a poll, limits reads to 1024 kernel events per slot per poll, and resynchronizes after queue overflow.

The macOS implementation uses Apple's system GameController framework for controllers with an extended gamepad profile. Up to four slots retain their assignments until disconnect; a retired slot publishes a disconnect before reuse. Buttons and axes use the same SIPL names and ranges. Call poll() at the game's update rate; transitions entirely between snapshots are not retained. Native device acceptance is described in MACOS.md.

For deterministic runs without hardware access, set SIPL_GAMEPAD=headless. The default is device. info().supported describes the compiled platform, not the presence of a controller or the current user's device permissions.

Native C interface#

src/gamepad_backend.h exposes the backend without depending on SIPL's VM:

SiplGamepads *pads = sipl_gamepad_open(true);
if (pads) {
    sipl_gamepad_poll(pads);
    const SiplGamepadState *pad = sipl_gamepad_state(pads, 0);
    if (pad->pressed & (1u << SIPL_PAD_A)) {
        /* Jump. */
    }
    sipl_gamepad_close(pads);
}

Call this interface on one thread. States are owned by the context and valid until it is closed. Native axis values have no deadzone; the SIPL binding applies the requested deadzone. sipl_gamepad_open(false) disables hardware access.

Run make test-gamepad for native input fixtures, SIPL validation, GC stress, packaging, and API discovery checks. Physical hardware qualification is separate.

Native video playback#

SIPL uses a statically linked, decoder-only FFmpeg 9.0.2 build with dav1d 1.5.3 for AV1. Windows and Linux use the same player and decoder allowlist. No FFmpeg executable, FFmpeg DLL, system codec pack, Vulkan device, or network access is needed for playback. Miniaudio remains the audio device backend: FFmpeg decodes and resamples audio; miniaudio sends stereo PCM to the operating system.

The Toolkit opens .mp4, .webm, and .mkv files, including uppercase suffixes, in reusable, read-only media tabs with controls enabled. Leaving a video tab pauses it. Reload and close release its decoder and audio device. SIPL chooses the tab from the extension; C inspects the bytes and selects the actual container and codecs. Renaming a WebM file to .mp4 does not change its decoder.

Supported content#

A container can hold codecs outside this list. For example, an MKV containing MPEG-4 Part 2, Vorbis, MP3, AC-3, or DTS is rejected with an unsupported-codec error. A present but unsupported audio track is an error, rather than silently discarded audio. FFmpeg's MOV/MP4 demuxer also contains QuickTime parsing internally; the Toolkit's accepted extensions remain exactly MP4, WebM, and MKV.

Encoding, muxing, CLI tools, networking, capture devices, filters, and hardware decoders are disabled. The player supports square/non-square pixels and 90-degree display rotation. HDR PQ/HLG and changing resolution are rejected. Subtitles, track selection, arbitrary display transforms, color-managed/HDR output, streaming, recording, and video editing are outside this initial playback implementation.

API#

fn start() {
    let clip: Video = ui.loadVideo("clip.mp4", {"controls": true})
    ui.open("Video", 960, 640)
    while ui.frame() {
        ui.drawVideo(16, 16, ui.width - 32, ui.height - 32, clip)
    }
    clip.close()
}

video.open(path, options?) and ui.loadVideo(path, options?) are equivalent. Loading decodes the first frame and leaves playback paused. Options are Bool:

ui.drawVideo(x, y, width, height, clip) draws in the active frame, preserves the display aspect ratio, and letterboxes in black. Coordinates and dimensions are Int logical pixels; coordinates must be within ±16,384 and sizes within 0..16,384. Controls occupy 40 pixels at the bottom. Small rectangles hide controls that do not fit: all controls below 100×64, mute below width 280, seek/time below width 360. Controls use the UI's clipping, DPI scaling, and pointer input. An optional sixth argument, {"enabled": false}, disables control input while keeping the controls visible. The Toolkit uses it behind menus and dialogs.

Video methods:

• play(), pause(), stop(); stop pauses and seeks to zero.

• seek(position: Duration); requires 0 <= position <= duration. Seeking is synchronous and decodes forward from a preceding keyframe. The first displayed frame is at or after the target, within the video's frame cadence.

• frame() -> Image; copies the current RGBA frame. It follows the Image limit of 4,194,304 pixels; larger videos can be drawn directly but cannot be copied into an Image through this method.

• close(); idempotent, stops and joins the decoder worker and audio callback.

Read-only properties: width, height (display dimensions), duration, position (Duration), playing, ended, hasAudio, and closed. Mutable properties: controls, muted, loop (Bool), and volume (finite Number from 0 to 1). video.info() reports the backend/version, decoder names, limits, and open count. Garbage collection and process exit also release players; explicitly close unused players to release devices and stay within the open-player limit.

Ownership and limits#

Only local, nonempty regular files are accepted. Limits are 2 GiB per file, 24 hours, 32 streams, 16,384 per display dimension, 8,294,400 display pixels shared across open players, and four open players. Individual decode contexts also cap decoded pixels. Initial open/probe and seek have a 10-second interrupt deadline; this is cooperative and cannot interrupt an individual codec instruction.

A worker owns the FFmpeg contexts, reads bounded frame queues, and performs decode/conversion. The UI consumes three queued RGBA frames and keeps one display frame. Rotation requires one additional scratch frame. Audio uses a one-second stereo ring and one bounded conversion block. The device callback performs no allocation, VM work, file I/O, or mutex locking. PCM consumption drives the video clock; underflow buffers playback, delayed audio tracks insert streamed silence, and decoder/resampler tails are drained. Pause and seek synchronize the worker and device before buffers are reset or released.

Persistent frame/PCM buffers contribute to GC pressure. FFmpeg/dav1d internal scratch allocations are additional memory and do not have a separate aggregate allocator cap; these bounds are not a sandbox for hostile files. Decode errors, unsupported media, and audio-device failures are reported as SIPL errors. Normal audio output does not silently switch to headless mode if a device cannot open.

Building#

Source pins and archive hashes are in third_party/ffmpeg/SIPL_PIN.md and third_party/dav1d/SIPL_PIN.md. Complete upstream sources are retained unchanged; only the selected features are compiled. The common allowlist is scripts/ffmpeg_video_options.txt. Dependency builds do not download anything. SIMD and CPU dispatch remain enabled for playback performance.

Linux development requirements: a C/C++ toolchain, GNU Make, pkg-config, Python 3, Meson, Ninja, and NASM. Install them with the host's development package manager, then run make (plus the project's existing network/X11 build dependencies). scripts/build_video_deps.sh supports MESON, SIPL_BUILD_JOBS, and SIPL_VIDEO_BUILD_DIR; the Makefile's corresponding directory is VIDEO_DIR.

macOS uses the same pinned codec sources and allowlist, Apple's C/C++ toolchain, CMake, pkg-config, Python 3, Meson and Ninja (plus NASM for Intel assembly). The build script supports the system Bash 3.2 and writes to build/video-macos by default. Core Audio/device playback requires native Mac acceptance.

Windows development requirements: x64 MSVC, Git Bash, Python 3/Meson, Ninja, NASM, and MSYS GNU Make. Native Windows GNU Make has incompatible shell quoting for FFmpeg's dependency rules. Run build.bat; the dependency script shares its initialized MSVC environment. SIPL_VIDEO_PYTHON, SIPL_VIDEO_BASH, SIPL_VIDEO_MAKE, and SIPL_BUILD_JOBS can select tools. Ninja bundled with Visual Studio is detected. Local tools under build/video-tools are recognized when present; they are development tools, not runtime dependencies.

Video is required in every desktop build. SIPL_VIDEO=0 is rejected; missing video build tools fail compilation. video.info()["available"] reports compiled support, and system.buildFeatures records it for build validation.

Distribution#

Packaged applications contain the selected decoder libraries in their executable. Ship the media files they open as application assets. The client still needs the ordinary operating-system UI/audio services; Vulkan is only needed by separate SIPL features that use it.

The enabled FFmpeg profile is LGPL 2.1-or-later, with GPL/nonfree options disabled; dav1d is BSD 2-Clause. Static linkage does not remove license obligations. Include FFmpeg/dav1d notices and license texts and provide corresponding source and a usable rebuild/relink route for the distributed executable, following the included upstream licenses. This repository contains the pinned library sources, SIPL source, and build scripts for rebuilding the runtime. A binary-only toolkit download by itself is not a complete release compliance bundle. Preserve the application's SIPL source or other relink material as needed for packaged apps; the sipl build command does not assemble that release bundle automatically.

Validation#

tests/test_video.sipl, test_ui_video.sipl, and test_toolkit_videos.sipl cover native decoding, content detection, seeking, lifecycle/limits, controls and tab routing in normal and GC-stress modes. tests/video_backend_tests.c exercises threaded playback, seek/restart/close, all selected decoders, and the compiled decoder/demuxer/protocol allowlist. Test media and regeneration commands are in tests/fixtures/video. Headless tests establish decoded pixels, timing, and resource behavior; audible quality and real desktop/device behavior still need release-target qualification.

Run python3 tests/video_tests.py build/sipl for the portable integration/package checks. On Linux, add --video-prefix build/video-linux/install --sanitize to compile the native player/profile and concurrent audio-callback tests with AddressSanitizer and UndefinedBehaviorSanitizer. These instrument SIPL's wrapper and audio code, not the prebuilt FFmpeg/dav1d archives. LeakSanitizer is disabled by the runner for hosts that execute tests under ptrace.

PDF generation and viewing#

The native pdf module creates PDF files on Windows, Linux and macOS without a browser or external process. It uses the vendored PDFGen C library, which is public domain. pdf.open() reads and renders pages through the pinned PDFium runtime installed beside SIPL by the build on Windows/Linux. The Mac implementation uses the system Core Graphics PDF engine instead. Toolkit opens a .pdf file in a read-only preview tab when selected, with page controls. Editing existing PDF files is outside this API.

let document: PdfDocument = pdf.create(595, 842, {
    "title": "Field notes", "author": "SIPL"
})
document.addPage()
document.setFont("Helvetica")
document.text("Field notes", 50, 780, 24, "#183153")
document.line(50, 765, 545, 765, 1, "#183153")
document.wrapText("This paragraph wraps within the page margins.",
                  50, 735, 12, 495, "left", "#333333")
document.rect(50, 615, 160, 60, "#dce9f7", true)
document.circle(400, 645, 25, "#de6b48", true)
document.save("field-notes.pdf")
document.close()

Coordinates and dimensions use PDF points (72 per inch). The origin is at the bottom left; text() positions the baseline. pdf.create(width, height, options?) sets the default page size. The optional Map accepts title, author, subject, and creator strings. addPage() appends a default-sized page; addPage(width, height) uses a custom size. Pages are numbered from 1, and selectPage(number) switches the target for later drawing. Drawing requires at least one page. width and height report the default size, and pageCount, currentPage, and closed report document state.

setFont(name) selects a built-in PDF font such as Helvetica, Times-Roman, or Courier; setFontFile(path) embeds and selects a TrueType (.ttf) font. text(value, x, y, size, color?) adds one line. wrapText(value, x, y, size, width, align?, color?) returns the height it used; align is left, center, right, or justify. Colors are #RGB or #RRGGBB; omitted text color is black. For shapes, line(x1, y1, x2, y2, width, color), rect(x, y, width, height, color, filled?, borderWidth?), and circle(x, y, radius, color, filled?, borderWidth?) use the same point system.

image(encodedBytes, x, y, width, height) embeds PNG or JPEG bytes. For an existing file, use fs.readBytes("photo.jpg"). For an Image value, encode RGB pixels first:

let picture = image.open("artwork.png").convert("RGB")
document.image(picture.encode("png"), 50, 400, 180, 120)
picture.close()

PNG alpha is not supported by PDFGen; convert to RGB if transparency is present. save(path) writes a PDF file and overwrites an existing path. toBytes() returns the complete PDF as Bytes for APIs that accept bytes. Both require at least one page. close() releases the native document and is idempotent; the garbage collector also releases documents that are no longer reachable.

Inputs are bounded: page dimensions are 1–14,400 points; documents may have up to 256 pages and 20,000 drawing calls. text() accepts 64 KiB per call, wrapText() accepts 511 bytes per call (PDFGen's fixed line buffer), and total text is limited to 4 MiB per document. Each encoded image is at most 8 MiB and 4 million pixels, with 16 MiB of encoded image data per document. Each TTF file is at most 8 MiB, with up to eight fonts and 16 MiB of font data per document. toBytes() follows the Bytes size limit of 16 MiB. Invalid arguments and PDFGen failures raise SIPL errors; check sipl api pdf and sipl api PdfDocument for exact signatures.

Run the PDF recipe with an output path to generate a complete sample.

Viewing PDFs#

pdf.open(path) returns an owned PdfFile for an existing PDF. Pages use 1-based numbers. pageCount reports the number of pages; pageWidth(number) and pageHeight(number) return the page dimensions in points. render(number, width, height) rasterizes a page at the requested pixel size and returns an independent Image suitable for ui.drawImage() or saving. Close each rendered image and then PdfFile when finished:

let file: PdfFile = pdf.open("field-notes.pdf")
let page: Image = file.render(1, 800, 1132)
page.save("first-page.png", "png")
page.close()
file.close()

pdf.open() reads at most 64 MiB and supports 1–4,096 pages. A rendered image may have at most 4,194,304 pixels and follows the Image dimension limits. Password-protected, invalid, or unsupported PDFs raise a SIPL error, which Toolkit shows in the preview tab. On Windows/Linux, rendering uses PDFium's C API; the DLL or shared library must remain beside SIPL and Toolkit. Mac page-content rendering needs no adjacent library; interactive PDF forms and annotation overlays are not implemented on that path. See Mac acceptance and PDFium dependency notes for the pinned runtime and redistributable notices.

AI in SIPL#

Toolkit OpenAI agent#

Toolkit has an expandable AI rail on its right edge. Open it by clicking the rail or choosing View → AI Agent. Configure the key and model in Tools → Settings → API keys, then write a message and press Ctrl+Enter or Send. Enter inserts a newline. Missing configuration offers Open Settings and keeps the current draft.

The default model is gpt-6-astra. The model field is editable for accounts with different model access.

The OpenAI and Google key fields are masked and can be remembered in an encrypted user or project config.enc file. Project settings override user defaults. The OpenAI field supplies the coding agent; the Google field supplies programs launched with Run through GEMINI_API_KEY and GOOGLE_API_KEY. Keys stay out of source files and command lines. See encrypted settings for locations and API contracts. When encrypted storage is unavailable, Settings offers Use for session.

An empty OpenAI field can use an existing OPENAI_API_KEY environment value. The agent can create and edit files under the opened project. Attach active buffer is off by default; enable it to include that buffer with a request. See the agent workflow.

Requests use the OpenAI Responses API with store: false. Escape cancels an active request. Request and response sizes are bounded, temporary key data is cleared after use, and captured output never includes the key.

Conversation API#

SIPL exposes a provider-neutral Ai conversation object:

fn start() {
    let assistant = ai.google("gemini-3.8-flash", {
        "system": "Answer clearly and concisely.",
        "temperature": 0.2,
        "maxOutputTokens": 1024,
    })
    print(assistant.ask("Explain a pointer in one sentence."))
}

ai.google(model, options?) and ai.load(provider, model, options?) create the same Ai value. Google options are apiKey, endpoint, system, temperature, maxOutputTokens, and timeout. The API key is read from GEMINI_API_KEY, then GOOGLE_API_KEY, unless explicitly supplied. Keep keys out of source control; cloud requests send prompts to the selected provider and may incur charges.

Conversations retain successful user and model turns. Call .reset() to clear that history. The endpoint option supports compatible gateways and local test servers.

Chat window example#

Set your Google key in the current shell and run the included chat example:

$env:GEMINI_API_KEY = "YOUR_API_KEY"
.\build\sipl.exe run examples\ai_chat.sipl

The key is read from the environment and is not stored in the source file. The chat waits for each response before accepting the next message.

Background Worker Jobs#

The worker module runs one isolated tool or program job at a time while a SIPL interface remains responsive. Jobs report their combined output, exit code, cancellation state, and output-limit state through worker.result().

API#

fn start() {
    let sourcePath = "src/main.sipl"
    var output = ""
    var status = "Scanning..."
    var received = false

    worker.start()
    worker.scan(sourcePath)
    ui.open("SIPL scan", 640, 360)
    while ui.frame() {
        if worker.done() and not received {
            let result = worker.result().unwrap()
            output = result["output"]
            if result["exitCode"] == 0 {
                status = "Scan passed"
            } else {
                status = "Scan failed"
            }
            received = true
        } else {
            ui.redraw()
        }
        ui.text(status)
        ui.text(output)
    }
    worker.stop()
}

• worker.start(executable?) creates the single background supervisor thread. By default child jobs use the current executable. A packaged development tool can provide a SIPL runtime path so its Run, Scan, Build, and Format jobs do not relaunch the packaged application. It is safe to call repeatedly; later calls are no-ops.

• worker.scan(path) runs sipl check path: parse, load modules, and compile without running top-level initializers or start().

• worker.pageCheck(path) runs sipl page-check path for one .sipl.html file or an entire private page directory.

• worker.run(path, apiKeyEnvironment?) runs the program in a child SIPL process. Sources with top-level named tests or use test use sipl test (normal plus GC-stress supervision), so Toolkit Run handles test files correctly. Ordinary programs retain their normal output and arguments. Optional overrides are limited to OPENAI_API_KEY, GEMINI_API_KEY, and GOOGLE_API_KEY and apply only to that Run job. They are not command-line arguments or changes to the parent environment. See encrypted Toolkit settings for validation and examples.

• worker.runPython(script, workingDirectory) runs a .py file with Python 3, captures standard output and standard error, reports the process exit code, and supports cancellation like other process jobs. It launches Python directly rather than through cmd, PowerShell, or a POSIX shell. Set SIPL_PYTHON to an explicit interpreter when needed. Otherwise Windows searches for py.exe, python3.exe, then python.exe; Linux and macOS search for python3, then python. Toolkit asks for confirmation before every Python run because the script has the user's normal OS permissions.

• worker.build(path, output, pages?) runs sipl build in a child process. Give the optional private Pages folder to compile templates into the output.

• worker.git(root, action, value?) runs a shell-free Git job in root. Supported actions are setup (initialize and add/update origin), status, pull (fast-forward only), commit (stage all and commit), push (publishes HEAD to origin and establishes its upstream), remote, publishPrivate, and publishPublic. The publish actions initialize and commit the project, then use the official GitHub CLI to create and push a new repository. Actions that need a URL, commit message, or repository name take value. Git credentials remain with Git Credential Manager, the SSH agent, or GitHub CLI; SIPL does not collect or store passwords or access tokens. The source-control sidebar also uses inspect, fetch, stage, unstage, stageAll, unstageAll, commitStaged, diff, diffStaged, and show. File actions require a literal repository-relative path; show requires a full commit hash. commitStaged commits the existing index without staging working-tree changes. inspect returns NUL-separated repository root and origin URL, then git status --porcelain=v1 --branch -z records, an empty record, and six fields per commit: full hash, short hash, full parent hashes, subject, author, and refs. History includes up to 100 commits from all refs. Missing origin and an unborn HEAD are valid empty fields.

• worker.openai(apiKey, requestBody) performs a fixed OpenAI Responses API request. An empty key reads OPENAI_API_KEY. The key is never passed through a process command line or temporary file and is cleared when the request finishes. This operation currently supports HTTPS on Windows. Request JSON is capped at 4 MiB.

• worker.format(path) runs sipl fmt path. Formatting mutates the file and

is deliberately not cancellable.

• Only one job may be pending or running. Submit another after the first job

completes. This bounded design avoids unbounded process/thread creation.

• worker.done() is a nonblocking completion check. worker.wait(milliseconds)

blocks the caller and is intended for command-line programs and tests, not UI frame loops.

• worker.result() requires a completed job and returns Result[Map]. The Map

contains exitCode, output, cancelled, and outputTruncated. The output capture is capped at 8 MiB. A child exit with nonzero status is still an Ok result; inspect exitCode. Worker setup/runtime failures raise a catchable Fail.

• worker.cancel() requests cancellation of the active scan, run, build, or Git job.

On Unix it terminates the child process group; on Windows it terminates the child Job Object when the OS permits assignment. A pending job is cancelled before launch. Cancellation is cooperative only at the process boundary; it does not interrupt worker.format().

• worker.stop() shuts down the supervisor, cancelling a pending/running scan

or run and waiting for it to exit. A queued format may be discarded before it starts; an in-progress format is allowed to finish and is never killed. It is called automatically at process exit.

• worker.isStarted and worker.isBusy are read-only status properties.

UI Pattern#

Never call worker.wait() from a window's frame loop. Submit once, keep rendering, and poll done() from the main thread as shown in the API example.

toolkit/main.sipl uses this pattern for Run, Build, Scan, Format, and AI requests: input and redrawing continue while a job works, and output appears after completion.

The toolkit's multi-step coding-agent loop and its separate project-confined workspace filesystem module are documented in AGENT.md.

Scope#

This is an asynchronous process-job API, not parallel execution of arbitrary SIPL functions. General-purpose worker.submit(fn, value) is not implemented. No async syntax is needed: worker.scan, run, runPython, build, format, and openai submit work, while done() and result() expose completion to normal SIPL code.

Toolkit Codex process bridge#

worker.codexInfo() -> Map reports available, the resolved native executable path, and setup error. It checks SIPL_CODEX, PATH and the Windows VS Code Codex extension. It does not start a model request.

worker.codex(root: String, prompt: String, options?: Map) starts a background Codex turn. Call worker.start first. Options are model: String, session: String (an explicit UUID, default new session), apiKey: String, network: Bool (default false), and timeoutSeconds: Int (1..3600, default 900). The key falls back to CODEX_API_KEY / OPENAI_API_KEY, then existing Codex login. It never enters argv. Prompt input is bounded to 512 KiB and supplied through a private temporary stdin handle removed on close. Project, model, session and key values are validated before a job is submitted.

worker.readOutput() -> String drains complete JSONL records while a Codex job runs; an incomplete UTF-8/JSON record stays buffered until its newline or job completion. It returns empty when no complete record is ready. worker.result contains any undrained tail and the normal exit/cancellation/truncation fields. Only Codex jobs support live draining. Keep polling to avoid the 8 MiB pending queue cap; after done, drain until empty before classifying completion.

Cancel and shutdown terminate the owned process group/job, including children. Windows creates the broker suspended and assigns its job before starting it; failure to establish process-tree ownership prevents launch. The broker passes workspace-write, no additional writable roots, and explicit command network and approval policies to Codex. Commands inherit no provider-key variables from the Toolkit-selected credential. See the agent reference for UI, security boundaries, engine installation, budgets and offline tests.

SIPL projects and packages#

SIPL's project commands use sipl.toml, a committed sipl.lock, and a local content-addressed cache. File-oriented commands continue to work for scripts outside a project. Inside a project, all compilation validates the lock and cached package contents first. Compilation never downloads dependencies.

sipl init my_app
cd my_app
sipl add geometry@^1.2.0
sipl fetch --locked
sipl check
sipl test --gc-stress
sipl doc
sipl build
sipl run -- firstArgument
sipl remove geometry
sipl publish

Commit sipl.toml, sipl.lock, source, tests and your license. Keep .sipl/ and build/ out of version control. init creates both configuration files, source, a smoke test, .gitignore, and .gitattributes; it refuses to overwrite existing source, tests or license files. The generated attributes preserve LF text bytes across checkouts so lock checksums survive platform changes. The initial license is LicenseRef-Proprietary with “All rights reserved.” Choose your actual license and replace LICENSE before distributing a package. Use sipl init my_library --lib to create src/lib.sipl without start(). sipl check validates libraries as well as applications.

A complete path-dependency example, including both lockfiles, is in the locked path dependency example. Its initial fetch works offline and restores the exact library snapshot from the repository.

Manifest and layout#

[package]
name = "my_app"
version = "0.1.0"
entry = "src/main.sipl"
sipl = "0.4.0"
license = "MIT"
license-file = "LICENSE"
api = "public"
kind = "sipl"

[dependencies]
geometry = "^1.2.0"
helpers = { version = "1.0.0", path = "../helpers" }

[capabilities]
requests = ["filesystem", "network"]

[build]
targets = ["host"]

Package names match [a-z][a-z0-9_]{0,63} and exclude language keywords, so they can be imported without new language syntax. Versions are stable MAJOR.MINOR.PATCH semantic versions. Requirements support exact versions (with optional =), caret and tilde: ^1.2.3 permits compatible 1.x versions, ^0.2.3 permits 0.2.x, ^0.0.3 is exact, and ~1.2.3 permits 1.2.x. All ranges exclude older versions. Prereleases, build metadata, wildcards and compound ranges are rejected in format version 1.

entry must point to a .sipl file under src/. sipl is the minimum runtime version. license declares an SPDX identifier/expression or your LicenseRef identifier, and license-file names a nonempty root-level file. The parser validates the identifier's shape; it does not resolve licensing terms. api = "public" uses the language's existing convention: module members are public except names starting with _. Packages expose their entry module. Consumers cannot import internal files through the cache.

The standard archive includes sipl.toml, the license file, optional README.md, and regular files beneath src/, tests/, docs/, pages/ and public/. It excludes .git/, .sipl/, build/ and the library's own lockfile. Paths, filenames, total bytes, file count and directory depth are bounded; absolute paths, traversal entries, symlinks and reparse points are rejected. Archives contain at most 4,096 files and 64 MiB. The graph contains at most 128 packages with one version/source per package name.

The supported TOML subset includes comments, basic/literal strings, string arrays, tables, inline dependency tables and integer format versions. Arrays may span lines and use trailing commas. Inline dependency tables occupy one line. Duplicate keys/tables, unknown fields, hooks and unsupported syntax fail with a diagnostic instead of being ignored.

Dependencies and lockfiles#

use geometry

fn start() {
    print(geometry.square(7))
}

A declared dependency name resolves to the cached package's entry module. Local use folder/module and use ../module declarations must stay inside that package's src/, tests/ or pages/ layout. A package can import only its direct dependencies; a transitive dependency must be declared explicitly. Dependency and import cycles fail.

Resolution chooses the alphabetically first unresolved name, tries stable versions newest first, and backtracks against all transitive requirements. The result does not depend on index ordering. Unsatisfiable versions or source conflicts leave the existing manifest and lock unchanged. Add/remove operations resolve the updated graph before committing either file.

sipl add geometry chooses a caret requirement for the newest registry release; an explicit requirement is preferable for review. sipl add helpers --path ../helpers reads the local package version and creates a checksummed snapshot. Path dependencies always need a version in the manifest. Nested path packages are independently snapshotted. Builds read those cached snapshots, even when the original directories change or disappear. Git dependencies are rejected.

The lock records format/runtime versions, the exact manifest SHA-256, exact package versions/sources, archive SHA-256, provenance/signatures and dependency edges in stable order. Review and commit it. The lock is a trusted project input: modifications to it are code changes and need the same review as source.

A missing locked registry archive is fetched by its checksum without consulting the current index. A missing path snapshot can be reconstructed from its original path only if its bytes still match the lock. Changed bytes fail; use --update deliberately to accept a new snapshot. Checksum failures in existing cache entries fail closed; remove the affected .sipl/cache/<checksum> entry and fetch again to restore it. To provision disconnected builds, copy the verified .sipl/cache directory from a machine that has fetched the same lock.

build, check, run, test and doc require the lock and complete cache. Their dependency handling ignores registry/network settings and never invokes a downloader or signature tool. With identical source, pages, lock, SIPL runtime binary and host target, executable output is byte-identical. SIPL's source-free executable contains a copy of the runtime, so changing the runtime binary or platform changes its output. The lock pins the SIPL version, not the host compiler/linker used to produce that runtime.

Capability requests and package limits#

Allowed capability names are filesystem, network, process, environment, ui, graphics and ai. Every dependency request must be explicitly present in the root project's requests, including requests from transitive packages. A new request stops resolution/building until the manifest declares it.

These declarations make dependency effects reviewable. They do not sandbox SIPL execution or dynamically restrict built-in APIs; runtime capability enforcement is a separate roadmap item. Tests and program execution run the project's SIPL code with the existing runtime permissions.

Only kind = "sipl" is accepted. Packages cannot load binary extensions or run build scripts, install scripts, or arbitrary package commands.

targets accepts host, windows-x64, linux-x64, linux-arm64, macos-x64 and macos-arm64. Builds select the current host and fail if the manifest does not allow it. Cross-compilation and additional runtime profiles are not implemented.

Tests and documentation#

sipl test discovers named test "name" { ... } blocks recursively under tests/. Legacy test_*.sipl and smoke.sipl programs with start() remain supported. Each case runs in fresh normal and GC-stress processes. Filtering, parallel jobs, deterministic seeds, snapshots, property and fuzz hooks, JSON/JUnit reports, and line coverage are built in. Production modules cannot import test sources or use test. See TESTING.md.

sipl doc checks the package entry and writes build/doc/api.md with public signatures and methods, source comments, canonical examples, package metadata, and dependency licenses. Each public contract states parameters, return type, failures, effects, complexity, and one example. Missing contracts fail creation without replacing the existing output. Use sipl doc --check to lint contracts or sipl doc -o docs/api.md to choose a destination.

The source debugger and structured crash reports are described in DEBUGGING.md.

Single official registry and publishing#

Format version 1 supports one logical official registry. Its HTTPS endpoint and trusted RSA public key must be provisioned by the operator:

$env:SIPL_REGISTRY = "https://<official-registry-origin>"
$env:SIPL_REGISTRY_KEY = "C:\registry\official-public-modulus.hex"

export SIPL_REGISTRY="https://<official-registry-origin>"
export SIPL_REGISTRY_KEY=/trusted/official-public-modulus.hex

SIPL_REGISTRY can instead name a trusted local mirror directory. No hosted registry or production signing key is created by this implementation. SIPL does not silently select an unprovisioned endpoint. There are no per-dependency registries. Mirrors must serve packages attested by the provisioned key. Distribute the key through a trusted channel separate from downloads. Never use the deliberately public test key in tests/fixtures/registry in production.

The endpoint/mirror contains:

index.toml
packages/<archive-sha256>.sipkg

index.toml starts with registry-version = 1 and contains [[package]] records with name, version, checksum, provenance and signature. Versions are immutable. signature is a 256-byte RSA PKCS#1 v1.5 SHA-256 signature, encoded as 512 lowercase hexadecimal characters, over this exact UTF-8 record, including the final newline:

SIPL-PROVENANCE-1
<name>
<version>
<archive-sha256>
<immutable-source/provenance-id>

The trusted key file contains the 2048-bit RSA public modulus as 512 lowercase hexadecimal characters, with optional trailing whitespace; the exponent is

65537. Windows verifies with CNG. POSIX uses the OpenSSL CLI for signature

verification and curl for HTTPS downloads; local mirrors need no curl. These tools are required only for fetching/attestation, not locked builds or deployed executables. HTTPS verifies normal TLS certificates, rejects redirects, and has bounded downloads/timeouts.

.sipkg is a canonical uncompressed source archive: SIPLSRC1, a little-endian u32 file count, then sorted records containing u32 UTF-8 path length, u32 content length, path bytes and content bytes. There are no timestamps or mode fields. The complete archive checksum and signature are checked before extraction; the extracted file tree is rehashed on every compilation. Cached source bytes are checked again immediately before parsing, so a concurrent cache edit cannot replace a verified module during the build.

sipl publish checks the package and prepares a deterministic archive, release.toml and provenance.txt under build/publish/. Use --output dir to choose a fresh directory. Submit these artifacts to the registry operator, who reviews the source/provenance, signs the attestation and installs the archive at its checksum path. Path dependencies must first become registry dependencies; publishing rejects local snapshots. The command performs no remote upload and needs no registry credentials.

An operator-signed release can be verified while preparing the submission:

sipl publish --output build/signed-release --provenance git:repo@commit --signature signature.hex

The archive's source files and manifest must remain unchanged between obtaining the signature and this command. Output directories are never overwritten.

Run the integration checks with python tests/project_tests.py build/sipl.exe on Windows or python3 tests/project_tests.py build/sipl on POSIX. Fixtures cover signed local-registry packages, deterministic resolution/backtracking, snapshot paths, cache corruption, stale locks, capability declarations, offline/identical builds, publication and failing test aggregation.

Packaged applications#

sipl build creates a directly executable SIPL application without copying the application's .sipl files into the output.

build\sipl.exe build app.sipl -o release\app.exe
release\app.exe firstArgument secondArgument

On Windows, sipl build selects the executable subsystem automatically. A program that directly calls ui.open(), ui.openFixed(), or graphics.open() in its controller or any compiled module is emitted with IMAGE_SUBSYSTEM_WINDOWS_GUI, so launching it from File Explorer does not create a console window. Other programs retain IMAGE_SUBSYSTEM_WINDOWS_CUI and behave as command-line applications.

Override detection when an application opens its window indirectly through a stored function or another dynamic call:

build\sipl.exe build app.sipl -o release\app.exe --subsystem windows
build\sipl.exe build tool.sipl -o release\tool.exe --subsystem console

--subsystem windows|console changes only the Windows PE subsystem. It does not replace the process entry point, alter system.args, change application exit codes, or hide a console after startup. For debugging a GUI application, build the same source with --subsystem console; stdout and stderr then remain available in the terminal and can be redirected to a log file normally.

Arguments passed to the packaged executable are available in system.args. A packaged executable runs its application directly; it does not expose the run, check, or other compiler commands.

Toolkit embeds its UI modules and the compiler/runtime in build/toolkit (build/toolkit.exe on Windows), so that executable can Run, Scan, Format, and Build when copied without a neighboring sipl executable. Updating source files or rebuilding only sipl does not update an older packaged application. make and build.bat now rebuild both artifacts; make package-toolkit refreshes Toolkit without launching it. The Linux toolkit.sh launcher checks the runtime's API/security revision, then refreshes the packaged Toolkit using that runtime before opening the app. Save and close existing windows, then reopen them to load a rebuilt version.

Toolkit uses toolkit/sipl.ico as its icon on Windows and Linux. The build extracts its largest PNG frame into toolkit/sipl.png and embeds the same image in Toolkit for its native window icon. Linux builds also create build/toolkit.desktop, a desktop launcher with the matching icon. Replacing sipl.ico and rebuilding updates all generated icon assets.

On Windows, package another application with its own executable icon using sipl build app.sipl -o app.exe --icon app.ico. The ICO's image sizes are preserved in Windows resources before the compiled application is appended.

PDF previews use the pinned PDFium shared library installed beside sipl and Toolkit by the build (pdfium.dll on Windows, libpdfium.so on x64 Linux). Carry that library and its notices when copying either executable to another folder. PDF creation with pdf.create() remains self-contained in the runtime. See PDF generation and viewing and the PDFium dependency notes.

macOS builds use the system Core Graphics PDF engine and native Metal, so neither PDFium nor Vulkan is deployed there. make also assembles build/SIPLToolkit.app. Signed Mach-O packaging, the Apple build tools, current coverage and native acceptance requirements are documented in MACOS.md.

In toolkit, open the .sipl controller and click Build, or choose Run → Build Executable…. The build runs in the background and reports its output in the existing Output panel. When the controller contains web.pages() and the workspace has a pages/ folder, toolkit embeds that folder automatically.

SIPL Pages#

Compile private .sipl.html pages into the executable with --pages:

build\sipl.exe build examples\web.sipl -o release\website.exe --pages pages

The package contains each page's compiled bytecode, literal HTML chunks, and include graph. It does not contain the page source or build-machine paths. A packaged application's existing web.pages("pages") call selects the embedded registry, so the pages directory is not needed in production. Development reload: true is automatically disabled because packaged pages are immutable; the controller does not need a production-only code path.

Static public files are deliberately separate. Deploy the directory passed to web.serve() beside the executable, or place those files in your normal static asset service.

What the package protects#

The output is a copy of the SIPL runtime with a versioned bytecode image. At startup SIPL:

1. verifies the image's SHA-256 digest before parsing it;

2. checks that its built-in ABI matches the runtime;

3. bounds-checks every record and object reference;

4. verifies every function's opcodes, operands, branches, stack depths, and exception-handler state;

5. runs the verified bytecode, with the JIT enabled normally.

Original source, comments, imported module files, page expressions, and absolute source paths are not stored. Values required by the program still exist: string literals, type/field/method names, and function names can be recovered by someone analyzing the executable. Machine code and bytecode can also be reverse engineered. Packaging raises the cost of copying the program; it is not a perfect secrecy boundary.

Keep API keys, database passwords, signing credentials, and uniquely valuable logic on a server or in an operating-system secret store. Do not treat an embedded key or bytecode encryption key as secret.

Bytecode checks and the embedded SHA-256 digest do not authenticate a publisher or restrict the program's host permissions. Only run applications you trust.

Preparing a SIPL beta release#

Use release.bat on Windows or make release on Linux to build the runtime and packaged Toolkit, run the release checks, and generate build/release-windows.json or build/release-linux.json with adjacent .SHA256SUMS files. These commands prepare files locally; they do not publish them. Build prerequisites and enabled features must be available for the selected platform.

The manifests record source/dependency content, component provenance and notices, the actual runtime/Toolkit/PDFium artifacts, static libraries, platform imports or system-library hashes, build features, and validation evidence. The gate rejects changed inputs and checks that the shipped PDFium library matches its pinned archive. After changing release inputs, prepare a new release manifest.

python scripts/release_manifest.py verify build/release-windows.json

On Linux, use python3 and build/release-linux.json. Verification checks local content consistency. Authenticate published artifacts through a trusted release channel or set SIPL_RELEASE_SIGNING_KEY to your own provisioned GPG key before the release command to sign the manifest. Supply both --signature and the expected --signer fingerprint when verifying a detached signature. Signing a manifest and Windows executable signing are separate steps. See the repository's release instructions for the complete commands.

Signing a Windows release#

Sign only after sipl build, because packaging and subsystem selection change the executable:

signtool sign /fd SHA256 /tr <your-timestamp-server> /td SHA256 /a release\app.exe
signtool verify /pa /v release\app.exe

Use a certificate from the Windows certificate store or a managed CI signing service. Do not commit a .pfx password or signing key. Signing authenticates the publisher and detects later modification; it does not hide the program.

Windows packages use the platform's stack protection, control-flow protection, ASLR, DEP, and high-entropy 64-bit ASLR features.

Compatibility and deployment#

A package is tied to the exact SIPL runtime ABI that created it. Rebuild the application after upgrading SIPL. The executable has no runtime dependency on the compiler source files or application modules.

Included networking#

HTTP, TCP/UDP, and WebSocket runtime support is included in packaged programs. HttpResponse, HttpClient, NetTask, CancelToken, NetStream, Socket, and WebSocket annotations survive packaging. Packages use the same shared task and cancellation API; there is no separate networking task runtime to deploy.

Netlib numerical library#

Normal Windows and Makefile builds compile the pinned Netlib C sources into a separate static archive using CMake. No Fortran compiler, downloaded dependency, BLAS DLL, or graphics backend change is required. The mat namespace, supported algorithms, memory bounds and exact versions are documented in MAT.md. The source pin is third_party/netlib/SIPL_PIN.md.

Windows candidate builds can set SIPL_OUTPUT and SIPL_TOOLKIT_OUTPUT to new filenames, preserving working runtime/Toolkit binaries while testing changes.

Compiled features and binary sizes#

Standard builds include every feature supported by the selected platform. Missing prerequisites and feature-off switches fail the build. MySQL/MariaDB is built from bundled Connector/C source and linked statically; it requires no separate connector SDK, DLL or shared library.

system.buildFeatures exposes the actual compiled capability inventory. build/build-report.json records compiledFeatures and exact artifactSizes; direct Make builds also write build/sipl.capabilities.json and build/toolkit.capabilities.json, with sizeBytes. Both runtime and packaged Toolkit are validated. Runtime headless settings retain compiled support. See full build policy and platform boundaries.

macOS implementation and acceptance#

The Apple implementation targets macOS 11 or later on Intel x64 and Apple Silicon ARM64. Build each architecture natively. It adds a Cocoa host for SIPL's existing software canvas and widgets, and a Metal backend for the shared graphics scene API. Windows continues to use Win32/Vulkan and Linux continues to use X11/Vulkan. No widget or application rewrite is required.

Release status: the Apple-specific Objective-C and GPU paths still require a native Mac build and the acceptance checks below. Source review or Windows/Linux regression results do not qualify a Mac release. The minimum OS version is a deployment target, not a claim that every Mac/driver combination has been tested.

Build and launch#

Install Apple's Xcode Command Line Tools and the existing project's build tools: CMake, Python 3, pkg-config, Meson and Ninja. Intel video assembly also needs NASM. SIPL does not install these tools or download dependencies during a build.

make -j4
./build/sipl run examples/fib.sipl
./toolkit.sh /path/to/project
open build/SIPLToolkit.app

make builds build/sipl, the packaged build/toolkit, and build/SIPLToolkit.app. The app contains its compiler/runtime and license notices. It needs no neighboring source tree or third-party dynamic libraries. make macos-app rebuilds the bundle alone after the executables are available.

The deployment target defaults to MACOSX_DEPLOYMENT_TARGET=11.0. The compiler and dependency builds inherit it. Apple-only sources are compiled as Objective-C with ARC; the shared runtime and UI engine remain C. Build separate native artifacts on Intel and Apple Silicon; combining packaged executables with lipo is not supported by the current bytecode-footer format.

Standard builds include Cocoa, Metal, video, WebP and MySQL/MariaDB. Feature-off switches are rejected. SIPL_UI=headless, SIPL_AUDIO=headless and SIPL_GAMEPAD=headless select runtime behavior while retaining every compiled capability. Opening a graphics window still requires a Metal-capable device.

Platform coverage#

Apple system frameworks used are AppKit, Foundation/Core Foundation, Core Graphics, Metal, QuartzCore, Core Audio/AudioToolbox, GameController, Security, CoreServices and SystemConfiguration. These ship with the OS/SDK. No new third-party library, MoltenVK, SDL, GTK or Qt is introduced. TLS, image, media, font and numerical dependencies reuse the repository's existing pinned sources.

Native CPU JIT compilation and automatic List.map GPU offloading remain qualified for Windows/Linux x64 only. Mac programs preserve language behavior through the interpreter. The Metal implementation is the graphics backend; a Darwin JIT allocator/ABI and a Metal compute offloader are separate future work. Toolkit's encrypted settings retain the existing Windows DPAPI limitation; Mac keys can be supplied for the session. A per-control macOS accessibility provider and on-screen IME preedit drawing are not part of this first host. Committed Unicode/IME text is delivered through NSTextInputClient.

Core Graphics renders PDF page content. Interactive PDF forms and annotation overlays are not implemented on the Apple path. PDFium remains the pinned renderer for Windows/Linux and is not redistributed in the Mac app.

Native acceptance#

Run from a Mac desktop login session:

make test
make macos-check

make macos-check checks executable discovery, signed packaging/repackaging, portable worker/gamepad/audio behavior, and Core Graphics PDF rendering. Its native smoke program opens a Cocoa window, checks key/Unicode event delivery, and submits indexed, textured Metal frames before and after resizing with Metal validation enabled. It requires a Metal-capable GPU. These checks have been added for execution on Mac hardware; they cannot run in the Windows/WSL workspace.

Before advertising support, also exercise Toolkit editing, Command shortcuts, clipboard exchange, file panels, Retina monitor changes, maximize/minimize, text input methods, audio/video devices, controllers, model textures and scene lighting on the native release architectures. Existing Windows/Linux desktop and Vulkan checks remain separate acceptance gates.

Signing and release#

sipl build stages a Mach-O executable privately, replaces the old package image/signature, includes the bytecode in __LINKEDIT, and uses Apple's /usr/bin/codesign to create and verify a local ad-hoc signature before committing the result. It uses direct process arguments and directory capabilities; neither output filenames nor signing commands pass through a shell. Existing outputs remain in place when packaging/signing fails.

An ad-hoc signature is for local execution. A public Mac release needs the maintainer's Developer ID signing identity and Apple notarization. Sign nested executables first and the app bundle last, after packaging, then notarize and staple using Apple's tools. Do not append more data after signing. See Apple's distribution guide and code-signature storage.

make release produces build/release-macos.json and adjacent checksums after the build/test gate. The manifest includes the app, static input archives and system-framework imports, and rejects accidental Homebrew/local dylib links. It does not sign with a publisher identity or submit anything to Apple.

Tests and API documentation#

Named tests#

Test files can declare named blocks without a start() function:

use test

fn total(values: List[Int]) -> Int {
    var sum = 0
    for value in values {
        sum += value
    }
    return sum
}

test "sum ignores empty list" {
    assertEqual(total([]), 0)
}

test "sum combines values" {
    assertEqual(total([2, 3]), 5)
}

Names must be nonempty literal Strings and unique within a file. Test blocks are top-level declarations. Their functions take no arguments. A returned value does not determine success: a test passes when execution completes without an uncaught failure. Use assertions to check its behavior.

Discovery, filtering, and isolated execution#

sipl test
sipl test tests
sipl test tests/totals.sipl --filter "empty list"
sipl test --jobs 4 --seed 42 --timeout 30000
sipl test --list

Without a path, the runner uses the project's tests/ directory, or tests/ in the current directory when no manifest is present. It discovers named tests recursively in .sipl files, skipping hidden directories, build output, and links. Existing test_*.sipl and smoke.sipl programs with start() remain supported as one legacy case per file. An explicit file can use either form.

--filter matches a substring of the file path or test name. --list lists matching cases without executing them. Finding no matching tests is an error. Discovery and report ordering are deterministic.

Every case runs in a fresh process, then runs again in another fresh process with SIPL_GC_STRESS=1. The stress run begins after its normal counterpart finishes. Globals, VM state, environment changes, and random state are isolated between cases. --jobs controls concurrent processes, from 1 to 64; the default is 1. Filesystem effects remain visible to other processes, so parallel tests must use separate paths for resources they mutate.

--timeout is a per-process timeout in milliseconds, default 30000 and maximum

3600000. Timed-out processes and their descendant processes are terminated.

Logs are capped at 8 MiB per process. --gc-stress remains accepted for compatibility; both execution modes always run.

Project tests validate their lockfile and dependency cache. Testing does not fetch dependencies. The child process's system.args is empty, preserving the contract of legacy test programs.

Test and production imports#

use test exposes test helpers inside test sources. Ordinary run and build commands reject named test declarations and the test-only module. Production modules cannot import a test source, even when a test process is compiling the production module. Test sources can import production modules.

Keep production code under src/ and test helpers under tests/ or test_support/. Files named test_*.sipl or *.test.sipl are also test sources. An explicitly selected test file is a test entry regardless of its name. Each test source that accesses test helpers declares its own use test. check, JSON diagnostics, and the language service can validate conventional test sources without executing them.

Assertions#

The three assertion functions are available as unqualified names and as members of test. They return None on success and raise a catchable Fail on a failed assertion. Their equality rule is SIPL's ordinary == rule.

Canonical examples:

test "equal values" { assertEqual([1, 2], [1, 2]) }
test "different values" { assertNotEqual(1, 2) }
test "expected failure" { assertThrows(fn() { fail("missing") }, "missing") }

Add use test at the top of the file containing these examples.

Deterministic seeds and property tests#

--seed accepts an integer from 0 to 9223372036854775807 and defaults to 1. Each case and its GC-stress replay start with the same seed. The runner seeds both the ordinary random/randomInt functions and the test module's random generator. Scheduling does not change a case's input sequence. Reproduce a failure using its reported seed and the same test code.

Canonical example for a property and its generator:

use test

test "addition preserves zero" {
    test.property(
        fn() => test.randomInt(-1000, 1000),
        fn(value) => value + 0 == value,
        500
    )
}

A predicate may return Bool or none after making assertions. Failure output includes the seed, zero-based iteration, input, and callback's failure reason. The runner reports the original failing input; it does not shrink inputs. --iterations sets the default property and fuzz iteration count, normally 100. An explicit callback API argument overrides that default.

Canonical seed and name inspection:

test "identify a run" {
    print(test.name, test.seed)
}

Deterministic clock fixtures#

These helpers are available only while executing a test. They never change the OS clock. Each isolated case starts with real clocks; a fixture freezes the runtime's wall and monotonic providers, including async deadlines.

All return None, mutate only the test process's clock fixture, and take O(1) time/space. Advancing/changing time without configuring a fixture fails. Create timer/interval/cron waits, explicitly advance time, then await the result:

use test

async test "one hour without a real wait" {
    test.clock(time.fromUnixNanoseconds(0))
    let began = time.monotonic()
    let alarm = time.timer(time.seconds(3600))
    test.advanceTime(time.seconds(3600))
    await alarm
    test.setWallClock(time.fromUnixNanoseconds(-1))
    assertEqual(time.elapsed(began), time.seconds(3600))
}

Time does not advance automatically. Awaiting a future frozen deadline without advancing it reaches the test supervisor's timeout. Configure/reset before creating waits; switching providers while waits exist changes their clock domain. Use seeded PRNGs and local HTTP/socket fixtures alongside these clocks for deterministic tests. See sipl api --task clock_tests --json.

Fuzz hooks#

test.fuzz(callback, iterations?, maxBytes?) invokes a one-argument Function with deterministic arbitrary Bytes values. They can contain zero bytes and invalid UTF-8. Binary parsers accept Bytes directly; validate and explicitly decode before calling a text parser.

Parameters are a Function, optional Int iterations from 1 to 100000, and optional Int maximum length from 0 to 1048576; maximum length defaults to 256. The API returns None. It raises Fail when the callback returns false or fails, and a runtime failure for invalid arguments or callback return types. It executes user code and advances deterministic random state. Complexity is O(iterations × (maxBytes + callback cost)) time and O(maxBytes) generation space, plus memory retained by the callback.

Canonical example:

use test

test "JSON validation tolerates arbitrary bytes" {
    test.fuzz(fn(bytes) {
        if unicode.isValid(bytes) {
            let valid = json.isValid(bytes.decodeUtf8())
            assertEqual(valid.type, "Bool")
        }
    }, 1000, 512)
}

Snapshots#

test.snapshot(name, value) compares the readable, quoted SIPL representation of a value with a checked-in snapshot. name is a String of 1 to 80 ASCII letters, digits, underscores, or hyphens. value can be any printable SIPL value. The return is None. Missing or changed snapshots raise Fail; invalid names raise a runtime failure; failed writes raise a file failure.

Normal execution reads snapshots. --update-snapshots explicitly permits creating or replacing them. Files live beside the test source under .snapshots/<source filename>/<test-name hash>-<snapshot name>.snap. Updates use an atomic per-process temporary file. Snapshots should be reviewed and committed with their tests. Map insertion order is preserved in snapshots. Printing custom values may call their toString methods.

Complexity is proportional to the formatted value and stored snapshot size, with the formatted representation retained while comparing or writing.

Canonical example:

use test

test "stable response shape" {
    test.snapshot("response", {"status": "ok", "count": 3})
}

sipl test tests/responses.sipl --update-snapshots
sipl test tests/responses.sipl

JSON, JUnit, and coverage#