Lichtspiel
A live audiovisual instrument for Ableton: session aware p5.js scenes generated at authoring time, validated by a five gate chain, and played from a monome grid and arc, with no model call ever on the render path.
A performer should not have to learn a second craft to have visuals. Lichtspiel derives musical features from the Ableton set itself through an MIR pipeline, then uses those features plus natural language prompts to generate code based p5.js animation, so the set drives set aware visuals with no node based tool like TouchDesigner to learn and no model call on the performance path.
01 · Demo
See it running
02 · At a glance
- Role
- Product & engineering lead on a hackathon team, then solo consolidation
- Timeframe
- May to June 2026
- Audience
- Live electronic performers and producers working inside Ableton Live.
- Collaborators
- A small hackathon team
- Stack
-
- TypeScript
- p5.js
- Vite
- Node (WebSocket bridge)
- Max for Live
- Python / FastAPI
- CLAP + librosa
- Claude (authoring time codegen)
- monome (serialosc)
Performers in Ableton have no way to drive expressive, code native visuals that understand the structure of their set (clips, scenes, sections) rather than just its loudness. Existing VJ tools map an audio envelope; they don't know the music.
Constraints that shaped it
- Built in a hackathon (a repeatable 3 to 4 minute demo had to work on stage)
- The performance runtime may never call a model or the network on the render path
- Degrade gracefully (run browser only with no Ableton, no bridge, no hardware)
- Adapt to whatever monome is plugged in (Grid 64 / Arc 2 up to Grid 128 / Arc 4)
- One person's judgment had to reconcile three diverging forks
03 · Outcomes
What changed
- Shipped a working live instrument for Ableton at the AIMS 2026 hackathon, cohosted by Music Hackspace with Berklee's AI Music Summit (Boston, June 2026)
- Consolidated three diverging forks into one coherent build across 43 commits in four days
- Generated visuals pass a five gate validation chain before they can play
- Reused Windchime's animation core (one lineage, two products)
04 · Origin
Made at Berklee's AI Music Summit
AIMS 2026 · AI Music Summit
Berklee College of Music, Boston · June 3–5, 2026
Musicians, educators, researchers and industry on how AI is changing music creation, production, performance and learning.
AIMS 2026 at Berklee Boston Hackathon 26 · Music Hackspace
Berklee College of Music · June 6–7, 2026
The hackathon cohosted with the summit: challenges set by industry partners, each team keeping full ownership of its work. Lichtspiel was built there, for Ableton Live.
Watch the hackathon film05 · Architecture
How it fits together
06 · The hardware
Tactile control, mirrored on screen
Digital twin: grid 128 (16×8, varibright) + arc 2 · ♪ LISTEN
grid ● hardware: 16×8 (128) · 2 quads · varibright 0–15 · tilt ✓ arc ● hardware: 2 enc × 64 LED (varibright) · push per enc
07 · The storyHow it came together
The opportunity: not another VJ plugin
VJ tools map an audio envelope; they do not know this is the B section. Lichtspiel reads the Ableton set itself (clips, scenes, locators, transport) and shapes code native visuals from that structure, with the monome as a latent space instrument. Built at the AIMS 2026 hackathon, cohosted by Music Hackspace with Berklee College of Music’s AI Music Summit (June 2026), under one constraint: a few minute performance that could not fail on stage.
How it works, at the boundary
A thin Max for Live shell feeds a Node bridge, which routes set state to a p5.js runtime for rendering and a Python service for authoring time generation. One rule governs the design (runtime purity): the performance path never calls a model or the network, so the visuals keep running browser only if the bridge or service disappears.
Scenes are single file sketches built from reusable idioms: fader banks, arc macros, step sequencers. The hardware layer is capability adaptive, folding a four encoder sketch onto a two encoder Arc, and a digital twin mirrors every LED so it plays with no hardware at all. Rehearsal produced two standing rules: nothing on the monome navigates, everything expresses; and a generated scene stays disposable until a deliberate Keep.
The judgment call, and how it is validated
The most instructive part is a judgment call, not a feature. It forked three ways: the team’s base, a rigorous solo generator, and a newer tree with better UX whose validation had been stubbed out. I consolidated onto the newest and restored the dropped rigor, each restoration its own commit. Nothing generated is trusted: every scene runs a five gate chain (typechecking, allowlist lint, monome playability check, headless render smoke test) inside a bounded self repair loop. The first scene through the rebuilt pipeline passed every gate at 60 fps.
08 · RoadmapNow, next, later
Confirm open gate mitigations on a live rig
Confirm the concurrent generation and source switching mitigations on real hardware.
Make the playability gate assert coverage
Require scenes to map all grid lanes and encoders, not just present idioms.
Hardware verification pass on the consolidated build
Run the consolidated build on a real rig to confirm mapping and mitigations.
Package as a Max for Live device
Package it as a distributable Max for Live device.
Fold Windchime's newer visual work back across the lineage
Update the shared parameter vector contract both ways so scenes travel between projects.
Ship at the Ableton Hackathon
A working live instrument demonstrated at the hackathon, consolidated from three forks.
09 · TimelineWhat happened, in order
-
Release The consolidated build
One build reconciling three forks, the rigorous fork's validation restored.
Customer value. A single instrument that is both pleasant to use and safe to play live, instead of three partial versions each missing something the others had.
- Rebase onto the newest tree
- Restore validation gates and the generated → promoted curation tier
- Fold the restored rigor into the Python authoring pipeline
Risks
- Restoring rigor reexposed a stubbed validation path (fixed, see incident)
-
Incident · Sev2 Resolved Feeder poll loop wedged the live trigger path
The feeder stalled, so live state stopped reaching the bridge.
Impact. With the trigger path stalled, the visuals stopped following the set even though the music played on. For a live instrument this is a showstopping failure of the core loop.
Detection. Seen in the bridge log: only the OSC sourced live state arrived and never the feeder source, and no scene launched or locator crossed events fired despite active playback.
Root cause. A poll loop wedge. A read against the control socket collided with the bridge's own snapshot query, the interleaved bytes never parsed, the inactivity timeout never fired, and the loop stayed stuck in a busy state.
Fix. The feeder's Ableton read now has an absolute settle timeout so it heals itself from a wedged read. If it ever recurs, stopping and restarting the feeder process clears it.
Followups
- Watch for recurrence now that the read has an absolute settle timeout (open)
- Keep Session scene launches as the reliable trigger where locator crossings are suppressed (done)
Blameless note. Two independent readers sharing one control socket is a subtle race that only shows up under live timing. Adding a self healing timeout and a documented manual recovery is a proportionate, honest response to an intermittent hang.
- Milestone Ship at the Ableton Hackathon
-
Release Constrained p5 code generation: Sync, Dream, Fuse
Three authoring modes turn the live set into scenes: Sync, Dream, Fuse.
Customer value. A performer can conjure a fresh, playable visual scene conditioned on what they're actually playing, without writing code and without risking a broken scene on stage.
- Audio "vibe" extraction (CLAP + librosa features) feeding generation
- Text prompt generation conditioned on the live set
- Five gate validation chain with a bounded self repair loop
- Keep / Promote curation tiers for generated scenes
Risks
- The playability gate checks that controls exist, not that they cover the surface
-
Incident · Minor Monitoring Generated scenes can pass while leaving the controller half dead
The gate checks idioms exist, not that they cover the surface.
Impact. A generated scene can be technically valid but underwhelming to play. Part of the instrument sits unused. A quality gap, not a crash.
Detection. Found in live testing while playing generated scenes and noticing unmapped hardware.
Root cause. The gate was written to confirm idioms exist, which is necessary but not sufficient for a satisfying instrument mapping.
Fix. Proposed: extend the gate to require coverage of all grid lanes and encoders. Not yet implemented, tracked as the next roadmap item.
Followups
- Extend the playability gate to assert coverage (open)
- Confirm the related concurrent generation mitigations on a live rig (open)
Blameless note. A good gate that turned out to be too permissive is a normal iteration of a quality bar, not a defect in judgment.
-
Incident · Sev3 Monitoring Scene then arrangement generated the same template
Session then Arrangement capture produced the same template, not a new one.
Impact. A performer expecting fresh imagery from the newest capture got a repeat, which undermines trust in the generate on capture flow during a play session.
Detection. Found during a live play session while exercising the capture then generate path back to back, comparing the scene result against the arrangement result.
Root cause. Not yet confirmed. Three hypotheses are open: the capture source not switching to the newest file, two generations colliding with no in flight guard, and near identical audio yielding a near identical brief. A missing context set call was also found on the auto path.
Fix. Request source instrumentation logs the path each generation actually used, a latest wins queue serializes concurrent generations so a stale run cannot shadow a newer one, and the missing context set call was folded into the autogenerate path.
Followups
- Run the live reproduction and confirm the second generation names the newest capture (open)
- If sources are correct but results still match, compare the two provenance banners and vibe logs (open)
Blameless note. Concurrent long running generations without a guard is an easy gap to leave under time pressure. Instrumenting first and confirming with a live repro, rather than assuming a fix, is the right way to close an intermittent issue like this.
-
Incident · Sev3 Resolved Consolidation reexposed a stubbed validation path
The consolidation base had its validation gates stubbed out with TODOs.
Impact. Without real gates, a generated scene could reach the stage without passing type, lint, playability, or render checks, the exact failure the rigor was meant to prevent.
Detection. Found during the consolidation itself, by comparing the two forks' generation paths rather than assuming the newer one was complete.
Root cause. The fork had prioritised UX and a new pipeline and left validation as a stub; the gap only mattered once that fork became the base everything else built on.
Fix. A real validation script (strict typecheck, an allowlist lint, a playability marker check, and a headless render smoke test) shelled from the generator with up to three self repair passes before failing.
Followups
- Prove the rebuilt path end to end with a first generated scene (done)
- Strengthen the playability gate to check coverage, not just presence (open)
Blameless note. Stubbing validation to move fast in a hackathon fork is a reasonable local choice; the risk only appeared at integration. Catching it there is the system working as intended.
-
Research · Discovery The fallback ladder that keeps a live demo alive
A ladder where every dependency has a safe substitute below it.
Insights
- Each fallback must be reachable without operator intervention midperformance
- A substitute that is never exercised is not a fallback
-
Research · Market scan Why not "another VJ plugin": finding the wedge
A scan of live visual tools: none understand the set's structure.
Insights
- Session aware mapping is the wedge competitors do not occupy
- Treating the controller as an instrument, not a remote, is the second half of it
-
Release Scene launch and locator autoretrieval, live in Ableton
Launching a scene or crossing a locator autoloads a playable variant.
Customer value. Visuals hot swap per song section as the set plays, so the imagery follows the arrangement on its own while the performer keeps playing music.
- Max outlets emitting scene launch and locator crossing events, forward only with a seek guard
- Bridge decoding of both events into wire messages broadcast to the runtime
- Runtime template picking that respects the onscreen lock, with mapped or random modes
- A live and simulated event source toggle so the flow demos without Ableton
Risks
- On a heavy 24 track set, detection can lag one to two seconds and Live can get sluggish
-
Incident · Sev3 Resolved Encoder presses switched scenes midperformance
Encoder clicks switched the active scene, via a legacy navigation mapping.
Impact. Expressive presses doubled as navigation, so a performer leaning into an encoder could yank the whole visual world out from under their set.
Detection. Reproduced during rehearsal playthroughs on the physical Arc.
Root cause. An early fallback mapping survived into the instrument era: encoder presses and a grid region were still bound to template switching from before the idiom layer existed.
Fix. Removed all template switching from the monome mapping. Hardware drives parameters only; navigation stays on the keyboard and in Ableton.
Blameless note. This established a rule the idiom layer inherited: the instrument surface is for expression, never for navigation, so no gesture can destroy the context it is played in.
-
Release Animation corpus and the monome idiom layer
A control layer: declare intent once, adapt to any monome.
Customer value. Crafted sketches keep their hand tuned control feel on any hardware, so the performer never loses reach of a control when moving between a small and a large device.
- Four capability aware idioms plus a compose function, as a pure control and LED layer
- Faithful ports of nine sketch families plus a handbuilt hero scene
- Folding so a large hardware sketch couples down onto a smaller device, nothing dropped
- Adapt up so a small hardware sketch lights bonus controls on a larger device
- A gestural panel and variant browser, with a headless idiom smoke suite
Risks
- Closing the full round trip (folding the extended set back down) is left as future work
-
Research · Field notes The monome as a latent space instrument
Making a grid and arc feel like an instrument, not a remote.
Insights
- Expressive gestures need immediate LED acknowledgement to feel played rather than sent
- Navigation bound to the same surface as expression makes a performer hesitant
-
Release Monome integration with capability adaptive folding
Grid and arc control over serialosc, adapting to the connected device.
Customer value. A performer plays the visuals on whatever monome hardware they own, and the LEDs mirror the performance so the controller reads as an instrument, not a remote.
- A pure Node serialosc layer with device discovery and hotplug recovery
- A capability matrix per device (cells, quads, varibright, tilt, encoders, push)
- Profile aware column fader mapping that adapts to grid width and encoder count
- A digital twin dashboard that mirrors LEDs and runs diagnostic sweeps
- Rate limiting so a fast encoder spin cannot flood or freeze the browser
Risks
- Self healing recovery restarts the daemon, which briefly blips every attached device
-
Release Max for Live Live API probe
A Max for Live device streaming transport, track, scene, and clip state.
Customer value. The visuals start to reflect what is actually happening in the Live Set, and a performer can nudge parameters from the Ableton device without leaving Live.
- A Live API helper that emits a stable session state snapshot, guarded to degrade to defaults
- An OSC receiver in the bridge for state, scene, and parameter addresses
- Device dials mapped to visual parameters and buttons mapped to scenes
- Read paths for the playing clip, clip color, and selected track device names
Risks
- Arrangement property names are best effort and need in set verification
-
Release The Node bridge between Max and p5
A Node hub carrying validated messages between Max for Live and the runtime.
Customer value. The visual runtime only ever receives well formed control messages, so a bad input upstream cannot corrupt or crash the performance.
- Loopback WebSocket server with a p5 client and reconnect with backoff
- JSON validation against shared schemas, with readable rejection errors
- Message logging and an HTTP status route
- A CLI sender for scenes, parameters, state, and retrieval, for testing without Ableton
Risks
- A silently dropped invalid message could hide an upstream formatting bug
-
Release The p5 visual runtime (browser only engine)
A standalone p5.js engine rendering scenes with no Ableton or bridge.
Customer value. The performer can open a page and immediately see and play visuals, so the system is usable and demonstrable even when nothing else in the stack is running.
- Template registry, message bus, and smoothed parameter interpolation
- Seeded RNG so any visual is reproducible from a seed
- Keyboard fallback for scene switching, distance, mutation, and lock
- Five initial scenes ported from the Processing corpus, verified at 60 fps
- A diagnostics panel (frame rate, active template, live parameter readout)
Risks
- A template that throws midframe must not kill the host loop
10 · DecisionsDecision records
Accepted Consolidate three forks onto the newest tree, and restore the dropped rigor
Context. Collaboration had forked the project into three lines: the team's base from before the AI work, a rigorous solo generator with real validation and curation, and a newer tree with a much better UX and a new generative pipeline, but whose validation had been stubbed out.
Decision. Base the consolidation on the newest tree and restore the validation and curation from the rigorous fork, each restoration as its own reviewable commit.
Why. The best UX and the best safety came from different forks. Merging judgment, not just code, was the only way to keep both, and doing it in discrete commits kept it auditable.
Consequences. A single coherent build across 43 commits in four days. It immediately surfaced a regression (stubbed validation gates), which was then fixed. See the linked incident.
Accepted Keep and Promote curation tiers for generated scenes
Context. Generated visual scenes vary in quality and should not silently join the trusted corpus. The team needed a way to keep a good generation for the current session and, separately, to graduate a proven one into the committed set of scenes.
Decision. Generated scenes surface behind a banner with two actions. Keep holds a scene in local session state that survives reload, and Promote moves the file into the committed tier through an explicit, human curated step.
Why. Validation proves a scene runs; it does not prove a scene is worth keeping. A human taste step is the right gate for the corpus, and separating a session keep from a permanent promote matches how a performer actually works.
Consequences. The trusted corpus only grows on a deliberate human action, and a session can still hold onto promising scenes without polluting it. The first user approved graduate came from a live session, which validated the flow.
Accepted Runtime purity: no model or network calls on the performance path
Context. Lichtspiel generates visuals with a language model, but it is played live on stage. A model call midperformance means unpredictable latency and a hard dependency on a network that hackathon venues rarely provide reliably.
Decision. Draw a hard line: the performance runtime never calls a model or the network. Every generative step happens at authoring time and produces a validated artifact the runtime can play deterministically.
Why. On stage, predictability beats cleverness. A visual that renders every frame with no external dependency is worth more than one that occasionally stutters waiting on a model.
Consequences. The runtime degrades gracefully to browser only with no Ableton, bridge, or model service. It also forced a clean split between an authoring pipeline and a play pipeline, which made validation and curation natural rather than bolted on.
Accepted Capability adaptive monome mapping
Context. The performer owns two classes of monome hardware (a Grid 64 with an Arc 2 and a Grid 128 with an Arc 4) that differ in size and capabilities. Sketches were authored against specific hardware, and hardcoding any one layout would strand the others.
Decision. A sketch declares its control intent as logical idioms. The idiom layer maps that intent one to one when the hardware matches, folds (couples or pages) controls onto smaller hardware so nothing becomes unreachable, and lights bonus controls on larger hardware.
Why. Adaptation belongs in one shared layer, not in every sketch. Coupling keeps every logical control reachable on a smaller device, which matters more for playability than a perfect one to one layout.
Consequences. New sketches get hardware adaptation for free and never carry per device branches. The cost is a real abstraction to maintain, verified by a headless smoke suite that exercises both a small and a large profile.
Accepted Graceful degradation to a browser only runtime
Context. A live audiovisual system stacks several fragile dependencies (Ableton, a Max device, a Node bridge, a Python service, and monome hardware). Any of them can be missing or drop midshow, and a hard dependency on all of them would make the instrument undemonstrable.
Decision. The p5 runtime runs fully browser only with no Ableton, bridge, or model service. Every experimental layer reduces to a safe control message (a scene id, a parameter vector, or a morph target). If the bridge appears the runtime autoconnects, and if it drops it reconnects.
Why. For a live instrument, staying up is worth more than any single feature. Reducing every layer to the same small control vocabulary means an absent or failed layer degrades the experience rather than ending it.
Consequences. The demo can start from nothing but a browser and gain capability as layers come online. This shaped the whole architecture toward optional, reconnecting layers rather than a monolith, and made testing each layer in isolation straightforward.
Accepted The VisualParamVector shared control contract
Context. Many layers (Max, the bridge, the monome mapping, keyboard input, and every visual scene) needed a common language for control. Without a fixed contract, each new input source or scene would invent its own parameter shape and the layers would drift apart.
Decision. Adopt a single VisualParamVector of sixteen normalized parameters plus a scene id as the one control surface every template understands, defined in the shared schemas package.
Why. A narrow, stable contract lets any input source (hardware, keyboard, generation, or automation) drive any scene interchangeably. Normalizing to a fixed range keeps smoothing, mapping, and generation simple across the whole system.
Consequences. Inputs and scenes became interchangeable, and generated scenes were constrained to the same sixteen keys with no new fields. The ceiling is sixteen parameters, which is a deliberate trade of raw expressivity for a dependable shared interface.