How
In-app agent loop (starter kit) calling typed AgentActionUtils on a live editor; externally via the tldraw MCP App (create/edit/delete shape tools, hosted at an HTTP MCP endpoint) or community MCP servers.
tldraw SDK + Agent starter kit
Infinite-canvas SDK whose agent kit reads and edits shapes live.
tldraw.dev ↗repo ↗docs ↗Agent template repo ↗History (undo/redo, marks) docs ↗tldraw MCP App announcement ↗License ↗
- language
- TypeScript (React)
- backing
- tldraw, Inc. (company; venture-funded, details not verified here)
- stars
- 50,825 (as of 2026-10-09)
- downloads
- tldraw npm: 430,719/week (api.npmjs.org, 2026-10-01..07); site claims 512.6K weekly
- latest release
- v5.5.2, 2026-10-02
- first release
- 2021-05
borrow
Different medium and license, but its agent context design and history-mark semantics are the best reference for scene's readback and planned time travel.
Borrow from it: take a specific idea, API or format.
the steelman: the best honest case for it
tldraw is what an agent-drivable visual surface looks like when a well-funded team builds it for years: the whole document is JSON records with snapshots, diffs and multiplayer sync, the history system has named marks you can bail or squash to, and the agent kit is a thoughtful, production-shaped architecture rather than a demo. Its context system is genuinely clever: the model gets a screenshot and three levels of structured detail (focused shapes in full, viewport shapes 'blurry', off-screen shapes as clusters), plus lints and the user's recent actions, and its actions stream onto the canvas as they are generated. Modes, prompt parts and action utils make 'what the agent can see' and 'what it can do' explicit data. With 430k weekly downloads and logos like Google and Replit, it is the reference implementation for agents collaborating on a shared visual canvas.
scores
| UI as data | 4 scene 5 | canvas document is fully JSON records; app chrome and custom shapes are React code |
|---|---|---|
| Agent can drive it | 4 scene 5 | first-class in-app agent and MCP App, but external agents get only three coarse tools |
| Agent can see it | 5 scene 5 | screenshot plus structured shape formats |
| Small, targeted edits | 5 scene 5 | id-addressed shape ops streamed incrementally |
| Time travel | 3 scene 0 | marks/undo/redo/bail/squash and snapshots exposed as APIs; no agent seek/fork |
| Live data rate | 3 scene 5 | real-time multiplayer sync, no published rates |
| Terminal native | 0 scene 5 | |
| Pixel graphics | 5 scene 4 | |
| Teaching | 2 scene 3 | |
| Maturity | 4 scene 1 | |
| Openness | 2 scene 4 | source-available, license key required in production |
tldraw SDK + Agent starter kit scene now scene planned
how agents use it
Wiring it in
Starter kit: `npm create tldraw@latest -- --template agent` plus provider API keys; production use needs a tldraw license key. MCP App: install in Cursor/other MCP Apps hosts.
Seeing the result
Yes: viewport screenshot plus structured shape data in three detail levels, selection, user's recent actions, lints.
Small edits
Yes: shape-id-addressed create/update/delete, applied incrementally while streaming.
History
Editor undo/redo with named marks and bail/squash; store snapshots and diffs; agent kit keeps chat/action history. No agent-facing seek/fork of document history out of the box.
In short
The strongest example of an agent working on a live, structured, visual document with both screenshot and data readback and id-addressed edits, but it is a web canvas under a commercial license.
architecture
tldraw is a React canvas over a reactive record store: every shape, page, binding and camera is a JSON record, with getSnapshot/loadSnapshot, store listeners emitting diffs, and multiplayer sync. The Editor API (createShapes, updateShape, deleteShapes, run with history modes) mutates the store; a history manager stores diffs and marks for undo/redo/bailToMark/squashToMark. The agent starter kit puts an agent loop inside the app: PromptPartUtil classes gather context (user message, selection, viewport screenshot, simplified shape formats BlurryShape/FocusedShape/PeripheralShapeCluster, lints, session history) and AgentActionUtil classes define what it can do, with streaming application of each action, modes that swap parts/actions, and multi-provider models. A separate tldraw MCP App (launched with Cursor 2.6, March 2026) exposes create/edit/delete shape tools and returns a live canvas to MCP Apps hosts.
- Document is data: JSON records in a store with snapshots and diffs
- Agent actions: create/update/delete shapes, freehand pen strokes, align/distribute/stack/reorder, viewport moves, todo list, scheduled follow-ups, external API calls
- Dual readback: screenshot of agent viewport + structured simplified shapes
- Actions stream and apply incrementally as the model emits them
- History marks (markHistoryStoppingPoint), bailToMark, squashToMark, history modes record/record-preserveRedoStack/ignore; remote changes never recorded
- Programmatic agent API: agent.prompt(), agent.request(), agent.cancel(), agent.reset(); modes with onEnter/onExit hooks
- MCP App: three tools (create, edit, delete shapes) with canvas state passed back into the chat
performance
No published throughput numbers for agent edits or sync found in the pages read.
adoption and upkeep
Adoption
The dominant whiteboard SDK; widely embedded in commercial products. The agent template repo itself is small (37 stars) but ships via `npm create tldraw -- --template agent`.
Used by: Google, Replit, ClickUp, Autodesk, Luma, Runway, BlackRock, Padlet (company logos on tldraw.dev)
Maintainability
Professionally maintained with a fast release train.
Releases: Frequent: v5.4.1 2026-09-08, v5.4.2 2026-09-10, v5.5.0 2026-09-30, v5.5.1 and v5.5.2 2026-10-02 · Contributors: ~223 (GitHub contributors API with anon, last page index) · Recent: 689 commits on main since 2026-07-11 · Bus factor: high: funded company team
weaknesses
- Not open source: production use requires a license key; hobby license requires a watermark
- Web/React only; no terminal surface
- The agent loop lives inside the browser app; external agents get only the MCP App's create/edit/delete tools
- No agent-facing history seek/fork/replay beyond undo marks
- A drawing canvas, not a general app UI toolkit; data binding to live streams is up to the integrator
- Agent template repo has modest traction (37 stars)
and thc-scene
Overlap
A live structured surface an agent edits by id, with dual readback (picture + structure), history marks, and an agent that explains and annotates.
What scene would be reinventing
Multi-resolution context for the model (full detail for focus, summary for viewport, clusters off-screen) and history marks with bail/squash; scene's planned marks/time travel should study tldraw's mark semantics rather than invent new ones.
The gap it leaves
tldraw is a browser whiteboard under a commercial license; it has no terminal renderer, no data sources bound to commands/streams, no keyboard-first app UI, and no binary hot-swap. thc-scene is a terminal app UI with live data.
What to borrow
- Focused/Blurry/Peripheral context levels for scene's `state`/`screen` readback to keep tokens low
- History marks with names plus bailToMark/squashToMark and record/preserveRedo/ignore modes (agent cursor moves = ignore)
- Modes as data: per-mode lists of what the agent sees (parts) and can do (actions)
- Streamed incremental application of agent actions so the human watches the agent work
- Lints passed to the model (overlaps, truncated labels) as a readback channel
sources
- tldraw Agent starter kit docs tldraw.dev
- tldraw History docs tldraw.dev
- tldraw License tldraw.dev
- tldraw MCP App blog tldraw.dev
- Cursor 2.6 MCP Apps (date context) forum.cursor.com
- GitHub API repos/tldraw/tldraw api.github.com
- tldraw/agent-template github.com
- npm downloads tldraw api.npmjs.org