How
MCP servers (tui-mcp, mcp-tui-driver, mcp-tuikit, mcp-tui-test) or a CLI the agent shells out to (both agent-tuis), with JSON output.
PTY/TUI screen drivers for agents (agent-tui and peers)
Run any terminal app in a PTY; agent reads screen, sends keys.
repo ↗docs ↗ConductorOne/agent-tui (refs + outline) ↗michaellee8/mcp-tui-driver (a.k.a. mcp-tui-server, Show HN) ↗nvms/tui-mcp ↗dragoscirjan/mcp-tuikit ↗GeorgePearse/mcp-tui-test ↗Show HN: MCP for controlling bubbletea/ratatui apps ↗
- license
- MIT · pproenca/agent-tui MIT; ConductorOne/agent-tui Apache-2.0 per README badge (GitHub API reports NOASSERTION); mcp-tui-driver MIT; tui-mcp MIT; mcp-tuikit MIT per README badge (GitHub reports NOASSERTION); mcp-tui-test MIT.
- language
- Rust (pproenca/agent-tui CLI crates, ConductorOne, mcp-tui-driver); TypeScript/Node (tui-mcp via node-pty + xterm-headless, mcp-tuikit via tmux); Python + Go (mcp-tui-test)
- backing
- Individuals, except ConductorOne/agent-tui (company ConductorOne)
- stars
- 118 (as of 2026-10-09)
- downloads
- agent-tui: 42/week npm (api.npmjs.org, 2026-10-01..07), 420 all-time crates.io; tui-mcp 19/week npm; @dragoscirjan/mcp-tuikit 6/week npm
- latest release
- v1.1.0, 2026-07-08
- first release
- 2026-01
complement
They drive any TUI from outside; scene is a TUI built to be driven from inside, and one of them is a good black-box test rig for it.
Complement it: it does a neighbouring job; interoperate.
the steelman: the best honest case for it
Screen drivers are the most universal answer to 'let an agent use a terminal UI': they need nothing from the app, so they work today on vim, htop, psql, k9s, lazygit, gdb, installers and every TUI ever written, including thc's own TUI and thc-scene. They hold long-lived interactive sessions that a bash tool cannot (a REPL in a transaction, an SSH login with 2FA), they return readable screens, and the best of them (ConductorOne's refs, mcp-tui-driver's ref-addressed clicks) are converging on a DOM-like accessibility layer for terminals, with selectors and wait-for-ref semantics that avoid flaky regexes. If the goal is merely 'an agent can see and drive a TUI', this category already solves it for free.
scores
| UI as data | 1 scene 5 | UI is whatever the driven app draws; ref outlines are derived read-only data |
|---|---|---|
| Agent can drive it | 4 scene 5 | full live operation (launch, keys, mouse, wait) but no creation or change of UI |
| Agent can see it | 4 scene 5 | text + PNG + semantic outline, but never the app's own state |
| Small, targeted edits | 1 scene 5 | input only; no edits to UI |
| Time travel | 2 scene 0 | asciicast recording and scrollback, no state seek |
| Live data rate | 1 scene 5 | |
| Terminal native | 5 scene 5 | |
| Pixel graphics | 0 scene 4 | PNG capture is readback, not a graphics layer |
| Teaching | 1 scene 3 | |
| Maturity | 2 scene 1 | all under a year old with tiny usage |
| Openness | 4 scene 4 |
PTY/TUI screen drivers for agents (agent-tui and peers) scene now scene planned
how agents use it
Wiring it in
Minimal: one `claude mcp add` line or a binary install; the target app needs no changes.
Seeing the result
Screen text, cursor, scrollback, raw bytes, PNG screenshots; ConductorOne and mcp-tui-driver add a semantic outline with refs. App state is not visible, only what it draws.
Small edits
None to the UI; the agent can only send input. Addressing is by screen position, text match or heuristic refs.
History
mcp-tui-driver records asciicast (.cast) with optional input; tui-mcp exposes scrollback; no seek/fork of application state.
In short
The universal fallback: an agent can operate any terminal program like a human, but it is scraping a picture of an app it cannot change or introspect.
architecture
All of them spawn the target program on a pseudo-terminal, feed its output into a headless terminal emulator (vt100/xterm-headless/pyte/vt10x/tmux), and expose verbs to the agent: launch, send keys/text/mouse, wait for text or idle, snapshot the screen as text (some also PNG), resize, kill. pproenca/agent-tui runs a daemon on a Unix socket managing sessions, with a CLI and a JSON-RPC WebSocket live preview. ConductorOne/agent-tui adds 'refs': a semantic outline of the screen with durable app-specific refs (@vim.mode, @vim.buffer) and generic positional refs (@e1), queried with a CSS-subset selector and waited on with `wait --ref`. mcp-tui-driver offers 23 MCP tools including ref-addressed clicks from a snapshot, asciicast recording, and a tui_run_code JavaScript scripting tool.
- Works on any existing TUI with no cooperation from the app
- Readback is the rendered screen (text, sometimes PNG), not the app's state
- ConductorOne refs bring a DOM-like outline to terminal screens (agent-browser pattern for PTYs)
- mcp-tui-driver tools: tui_launch, tui_close, tui_list_sessions, tui_get_session, tui_resize, tui_send_signal, tui_text, tui_snapshot, tui_screenshot, tui_press_key(s), tui_send_text, tui_click, tui_click_at, tui_double_click, tui_right_click, tui_wait_for_text, tui_wait_for_idle, tui_get_code_interface, tui_run_code, tui_get_input, tui_get_output
- tui-mcp tools: launch, kill, list_sessions, status, resize, screenshot (PNG), snapshot (text), scrollback, output (raw bytes), read_region, cursor, send_keys, send_text, send_mouse, wait_for_text/idle/exit; plus a monitor TUI
- mcp-tuikit: tmux-isolated sessions, txt/png snapshots, YAML flow runner
- mcp-tui-test: stream (pexpect) and buffer (pyte) modes, pytest plugin, tui://{session}/screen resource
performance
No published numbers; interaction is paced by wait-for-idle/wait-for-text polling.
adoption and upkeep
Adoption
A crowded, young niche with no breakout: pproenca/agent-tui leads with 118 stars; mcp-tui-driver 34; mcp-tui-test 19; ConductorOne/agent-tui 13; tui-mcp 13; mcp-tuikit 0. The Show HN for mcp-tui-driver scored 8 points.
Maintainability
Many near-identical tools, each maintained by one person; activity has slowed for most after launch.
Releases: pproenca/agent-tui: v1.1.0 2026-07-08 and v1.0.1/v0.3.x on 2026-07-09; multi-channel distribution (brew, npm, crates, install script) · Contributors: pproenca/agent-tui: 1 · Recent: pproenca/agent-tui: 2 commits on default branch since 2026-07-11; mcp-tui-test pushed 2026-10-06; ConductorOne last push 2026-07-29; mcp-tui-driver last push 2026-01-12 · Bus factor: low: each tool is a single maintainer or a small company side project
weaknesses
- The agent sees pixels-as-text, not state; meaning is inferred from layout
- Cannot change the UI, only press keys on it
- Refs are heuristic or need per-app adapters (ConductorOne has adapters for vim, shell prompt)
- Timing relies on wait-for-idle/text polling; flaky on animated or streaming screens
- Fragmented: six-plus near-duplicates, each single-maintainer, low adoption (tens of npm downloads/week)
- Unix-only for several (pproenca/agent-tui explicitly not Windows)
and thc-scene
Overlap
Agent sends keys/mouse to a live terminal UI and reads the screen back as text.
What scene would be reinventing
scene's `key`, `mouse`, `screen` socket ops duplicate what these drivers already do for any TUI, including scene itself. For black-box testing of scene, one of these is enough.
The gap it leaves
These tools cannot read app state, push or patch UI, bind data streams, add callouts, hot-swap the binary or rewind; scene is the cooperating app side that exposes structure instead of forcing scraping.
What to borrow
- wait_for_idle / wait_for_text / wait --ref primitives for agents (scene can implement them exactly from its own state instead of polling pixels)
- ConductorOne's ref grammar and CSS-subset selectors (@app.role[value=x]) for addressing scene nodes
- read_region and scrollback ops on scene's screen
- asciicast export of a scene session for sharing
- A monitor TUI listing live sessions with attach (tui-mcp monitor)
sources
- pproenca/agent-tui README github.com
- ConductorOne/agent-tui README github.com
- michaellee8/mcp-tui-driver README github.com
- nvms/tui-mcp README github.com
- dragoscirjan/mcp-tuikit README github.com
- GeorgePearse/mcp-tui-test README github.com
- Show HN item 46579303 news.ycombinator.com
- npm downloads agent-tui api.npmjs.org
- crates.io agent-tui crates.io