Drawgent: Coding agent on a live Excalidraw canvas

drawgent: your coding agent on a live Excalidraw canvas#

drawgent connects your own Claude Code, Codex or opencode (your install, login, config and repo) to an Excalidraw whiteboard. Ask for a diagram in the chat panel, or write AGENT: … next to the part of a drawing you want changed. The agent looks at the canvas (screenshot + scene), edits it live, checks the result, and marks the note DONE.

Quick start#

Prerequisite: one of claude, codex or opencode installed and logged in. The Claude and Codex bridges also need Node.js ≥ 18 (npm).

drawgent setup claude        # once per agent: claude | codex | opencode
cd ~/my-repo
drawgent up                  # new agent session in this repo + canvas in your browser
drawgent up --attach         # or: pick one of your running sessions and connect the canvas to it

`drawgent setup <agent>`#

Checks everything once, fails with the exact fix when something is missing, and writes ~/.config/drawgent/config.toml:

The agent CLI. Your claude / codex / opencode on PATH.
Login. claude auth status, codex login status or opencode auth list.
ACP bridge.
- opencode speaks ACP itself (opencode acp).
- Claude Code and Codex use the official ACP adapters. They are installed once into ~/.cache/drawgent/adapters (~60 MB), without their bundled agent binaries, and pointed at your CLI (CLAUDE_CODE_EXECUTABLE, CODEX_PATH).
- Setup then verifies the ACP handshake.
Canvas tools for attached sessions. Only Codex needs a change: codex mcp add drawgent -- drawgent mcp. Claude and opencode get the tools at attach time.
Headless Chrome for the renderer. Setup uses your Chrome/Chromium if you have one. Otherwise it proposes:
- downloading Chrome Headless Shell (Chrome for Testing, ~120 MB, no sudo) into ~/.cache/drawgent/chrome, and telling you exactly which system libraries are missing, if any;
- installing Chromium with your package manager (apt, snap, dnf, pacman, zypper, apk, brew, or nix without sudo).
Non-interactive: --chrome download | system | /path/to/chrome.

drawgent up refuses to start until setup succeeded for that agent, or if something setup recorded disappeared.

`drawgent up`#

Runs in the current directory (the workspace):

starts the editor + API on 127.0.0.1:7300 (next free port if taken);
starts a new session of your agent over ACP, working in the workspace;
opens your browser. On a headless box it prints the ssh -L command instead.

The scene is kept in .drawgent/scene.json, which is git-ignored automatically.

`drawgent up --attach [id]`#

Connects the canvas to a session you already run. Without an id it lists the sessions it finds (current directory first) and lets you pick one:

agent	discovery	how the canvas reaches it
Claude Code	`claude agents --json` (interactive and background sessions)	fork: a new session carrying the full conversation, driven by drawgent over ACP. Your terminal session is left untouched
opencode	opencode servers listening locally: start the TUI with `opencode --port 4096` (or `opencode serve`)	live: messages go into your running session (you see them in your TUI); drawgent adds its MCP tools to that server at runtime; replies, tool calls and prompts you type in the TUI are mirrored into the chat panel
Codex	sessions in `~/.codex/sessions`	live: `codex queue --thread <id>`; replies are mirrored from the session's rollout file. The session needs the drawgent MCP (registered by setup, loaded when Codex starts)

Using the canvas#

Chat panel (right side): send a request, watch replies and tool calls stream in, approve permission prompts, press Stop to cancel a turn.
On the canvas: write a text starting with AGENT: next to or inside a shape, or draw an arrow from the note to a shape. It fires about 2.5 s after you stop typing, with its position, what it points at and what is nearby. The agent resolves it into a green DONE: … note. Edit it back to AGENT: to send it again.
Prompts are queued and run one at a time. A queued note that was already handled is skipped.

excalidraw.com rooms#

drawgent up --room 'https://excalidraw.com/#room=<id>,<key>'

drawgent joins the room as a collaborator (🤖 Agent, with a cursor that follows its edits). Humans can stay on excalidraw.com: their AGENT: notes reach your agent, and its edits appear there live.
Traffic is end-to-end encrypted with the room key. Empty rooms are loaded from and saved to excalidraw's Firestore storage.
The local editor mirrors the room and still has the chat panel.

Other commands#

drawgent mcp: stdio MCP server with the canvas tools. Agents launch it; it finds the running drawgent up by itself.
drawgent serve: low-level server with explicit agents (--agent claude,codex,opencode as set up, or name=command for any ACP agent); for scripts and containers.
Options for up / serve: --port, --room, --token (API bearer + ?token= in the URL), --permissions canvas|ask|all (default canvas: drawing tools auto-approved, anything else asked in the chat panel), --data, --settle-ms.

Optional: Docker#

docker compose up --build runs a canvas server (drawgent + Chromium, no agents), e.g. to host a shared canvas or a room bridge on a server. Agents are never bundled: they always run with your own setup.

Build from source#

npm ci && npm run build      # editor + renderer pages -> dist/ (embedded into the binary)
cargo install --path .       # or: cargo build --release

Agent tools (MCP)#

get_scene, get_screenshot (vision; zoom with element_ids), add_elements (Excalidraw skeletons; arrows bind by id and are routed edge-to-edge), add_mermaid (auto-layout), update_elements (labels follow shapes, bound arrows re-route), delete_elements, clear_canvas, list_instructions, resolve_instruction, set_status.

API#

GET /api/health · GET /api/scene · GET /api/screenshot?ids=&padding=&max= · POST|PATCH|DELETE /api/elements · POST /api/mermaid · POST /api/clear · GET /api/instructions · POST /api/instructions/{id}/resolve · POST /api/status · POST /api/chat {agent?, text} · WS /ws (browser sync + chat events)

Tests#

cargo test                          # fractional indices, room crypto/framing
node scripts/smoke.mjs [url]        # chat turn + AGENT: note against a running drawgent
node scripts/e2e-browser.mjs [url]  # real browser: chat panel + note typed on the canvas
node scripts/room-e2e.mjs           # fresh excalidraw.com room ↔ drawgent, both directions

Layout#

src/ (Rust):

main.rs: CLI.
setup.rs, config.rs, chrome.rs: setup, config, renderer install.
attach.rs: session discovery and picker.
agents.rs + acp.rs: ACP driver (new / fork sessions).
live.rs: live opencode / Codex drivers.
hub.rs: routing, notes, chat log.
scene.rs: store and edit operations.
renderer.rs: Chrome over CDP.
mcp.rs: MCP server.
room.rs: excalidraw.com client.
fractional.rs, geometry.rs, el.rs: helpers.

web/: editor (main.jsx, chat.jsx) and renderer page (render.jsx).

Limits#

The renderer needs Chrome (a native renderer is planned).
Claude "attach" is a fork, because Claude Code has no public way to inject into a running terminal session.
Codex live attach is implemented, but was not yet tested against a logged-in Codex.
One scene per workspace. Images/files are not synced.