Time-indexed multimodal data store and viewer, now agent-drivable via MCP.
rerun.io ↗repo ↗docs ↗Viewer MCP docs ↗0.34.0 release notes (Viewer MCP) ↗viewer_control.proto (ViewerControlService) ↗Blueprints ↗Timelines ↗Column chunks blog (0.18 perf) ↗Real-time kHz time series blog (0.13 perf) ↗Blueprint undo (undo.rs) ↗
Different medium and domain (GPU viewer for robotics data), but the best shipped design for agent-driven, time-indexed viewers; scene should copy its MCP and blueprint-undo patterns.
Borrow from it: take a specific idea, API or format.
Rerun is the closest shipping relative of thc-scene's whole thesis, just in pixels instead of cells. It separates data from presentation cleanly: data is an append-only, multi-timeline, columnar store you query at a cursor (so time travel over data is free and scrubbing millions of points is a query), and presentation is a blueprint that is itself data — stored as a recording, so layout undo is literally moving a time cursor back. Since 0.34 an agent drives a live or headless Viewer over MCP: it reads structured state including every view's own error reports, gets and sets the whole layout as JSON, seeks any timeline, runs any command-palette command, screenshots a single view, and draws a pulsing highlight to point a human at something. It has real, published performance (60k scalars/s at 60 fps with <30 ms latency; 100x ingest gains), $17M of funding, 195 contributors and adoption by Meta, Google, Hugging Face and Unitree. A fan would say scene should study how Rerun split 'high-level generated tools first, low-level widget pokes as fallback'.
| UI as data | 4 scene 5 | Blueprint (layout, views, properties, overrides) is data readable/writable as JSON; view kinds themselves are fixed Rust code. |
|---|---|---|
| Agent can drive it | 4 scene 5 | Official MCP controls a live Viewer (open, seek, layout, commands, clicks); data creation still through SDK code. |
| Agent can see it | 5 scene 5 | Structured viewer state + blueprint JSON + schema + screenshots + accessibility tree. |
| Small, targeted edits | 3 scene 5 | set_blueprint is whole-document; per-view edits need commands or clicks. |
| Time travel | 4 scene 0 | Agent can read timelines and seek the cursor; blueprint undo exists; no fork/branch of state. |
| Live data rate | 5 scene 5 | Published 60k scalars/s at 60 fps, <30 ms latency; data streams from SDK bypassing any model. |
| Terminal native | 0 scene 5 | |
| Pixel graphics | 5 scene 4 | |
| Teaching | 3 scene 3 | highlight_rect to point a user at something, learning course, 'visual walkthroughs of papers' examples; no narrated tours. |
| Maturity | 4 scene 1 | |
| Openness | 4 scene 4 | MIT/Apache dual, but company-controlled with a commercial data platform. |
Rerun scene now scene planned
MCP server built into the CLI: `claude mcp add rerun -- rerun viewer-mcp` (stdio), connecting to a running or `rerun --headless` Viewer over gRPC (`rerun_connect` or `--endpoint`). Two families: high-level rerun_* tools generated from viewer_control.proto, and low-level egui widget tools. Data goes in via the Python/Rust/C++ SDKs, which agents write and run. Repo also ships agent skills (`npx skills add rerun-io/rerun`).
Low: one MCP registration plus a running Viewer.
Strong: rerun_get_viewer_state (active recording, open recordings with timelines, time ranges and current cursor, every view with its warnings/errors), get_blueprint (JSON), get_recording_schema (entities, components, Arrow types), viewer logs appended to every tool result, screenshots of the whole Viewer or one view, an accessibility tree; values via the catalog server or RrdReader.
Blueprint: set_blueprint replaces the whole blueprint ('anything it leaves out is cleared'); finer changes via run_command or widget clicks. Data: the SDK logs per entity path at a time point (append-only).
Agent can seek (rerun_set_time_cursor: recording, timeline, time; optional play), read timelines and ranges, query data at any time through the catalog, and run command-palette commands including playback and blueprint undo. No fork of a recording; the data log is append-only and the app producing it is outside Rerun.
A shipping example of exactly the pattern scene wants — an agent seeking a time cursor, reading structured viewer state and screenshots, rewriting the layout as JSON and highlighting things for a human — on a GPU desktop viewer.
Data is logged by SDKs as entities (paths) with components (Apache Arrow columns), each stamped on one or more timelines (log_time, log_tick, plus user timelines such as frame or sensor time). The store keeps data in column chunks and answers latest-at and range queries for whatever time cursor each view has, so scrubbing is a query, not a replay. How data is shown is a separate 'blueprint' (layout, views, per-view properties and overrides), itself stored as a recording with its own timeline: blueprint undo/redo moves that timeline's cursor back and forth, and a new edit drops the redo tail. Since 0.34, `rerun viewer-mcp` lets an agent drive a running (or headless) Viewer over gRPC.
Vendor-published but concrete, reproducible benchmarks (the 2.25M-point dataset is downloadable). Numbers are for data ingest/plotting, not UI control; no published latency for MCP calls.
The most popular open-source logger/visualizer for robotics and multimodal ML data (per the company's funding post), with real PyPI volume.
Used by: Meta (Project Aria tools), Google Research (brush_splat), Hugging Face LeRobot, Unitree
Very actively developed by a funded team; fast cadence with breaking changes between 0.x minors.
Releases: Minor every ~2–4 weeks with patch releases: 0.34.0 (2026-07-06), 0.35.0 (2026-07-23), 0.36.0 (2026-08-10), 0.37.0 (2026-09-01), 0.38.1 (2026-09-17). Each minor ships a migration guide (pre-1.0 API churn). · Contributors: 195 (GitHub contributors API, incl. anonymous) · Recent: ~791 commits in the last 90 days; repo pushed 2026-10-09; 1,237 open issues+PRs. · Bus factor: high: funded company team, many active committers
Agent drives a live viewer over a local protocol; layout is data (blueprint JSON); time cursor seek; structured state + screenshot readback; highlight to point the human; data streams bypass the model; Rust.
Time-indexed state with a cursor that every view reads, layout-as-data with undo-by-time, and an agent tool surface with state/screenshot/seek/highlight. Rerun has already shipped the agent ergonomics scene is planning (generated high-level tools, view error reports, logs piggybacked on every result).
Terminal-native rendering, arbitrary composable UIs (not just data views), id-addressed patches of a live UI, fork/replay of UI state, and narrated teaching tours with learner takeover.
| 1–7 | home · features · use cases · docs · changelog · under the hood · blog |
| t | light or dark |
| j k | scroll |
| g G | top · bottom |
| ? Esc | this · close |