Durable coding agents

Run any ACP coding agent (Claude Code, Codex, Hermes, ...) so a crash resumes the same session, tool calls are journaled before they run, and a newer driver cuts off an older one.

cua agent run --durable runs a coding agent under a re-driver. When the process that drives the agent dies, or the machine it runs on does, the session is started again, the agent's own session is reloaded, and the agent is told what had already happened. It works with any harness that speaks the Agent Client Protocol (ACP); the same runner drives all of them.

Run one#

cua agent run --durable --on auto claude-code "Fix the failing tests in /work"

The session keeps a journal in its state directory (CUA_SESSION_STATE_DIR, or $CUA_JOB_DATA/session when that is empty). It holds:

What happens when it crashes#

1. The re-driver issues a higher epoch and starts the driver again, on any machine that can reach the state directory.
2. The new driver raises the fence first. An older driver that is still alive sees the higher floor on its next write, stops, and exits with code 75 without writing anything.
3. The driver reloads the agent's session (session/resume, else session/load). A harness with neither gets a new session and the run is marked degraded: the recovery note then carries a summary of the journal in place of the lost conversation.
4. The first prompt of the resumed turn is a recovery note. It names the interrupted request, the tool calls that completed (with a short result), and the calls that may have partially run, with the instruction to check the current state before repeating them. A turn that already ended is never replayed.
5. If the agent process itself dies, the driver restarts it in place (up to three times) and resumes the same way before it gives up and exits non-zero for the re-driver.

Exit codes: 0 finished, 78 cannot succeed (authentication, agent not found, bad configuration), 75 fenced by a newer epoch; anything else is a crash and the session is resumed with backoff.

Working in a durable Space#

A session can be bound to a durable Space (spec.space, a Space the coordinator places on one of your hosts and keeps). Each drive takes the Space's lease and the driver gets the Space's name, address, token and lease token (CUA_SESSION_SPACE, _ADDR, _TOKEN, _LEASE); the epoch is the larger of the session's counter plus one and the lease token, so both only grow. One session holds a Space at a time: a second one stays pending with the reason held by session:NAME until the first ends, and a superseded lease token is refused. If the Space's host loses it, the coordinator places it again as the next generation, restored: "from-scratch", and the next drive receives the new address and token.

This was run for real with the Pi Durable driver (40 runs on Linux with Docker: driver kills, the Space removed after a kill and under a live driver, a coordinator restart, two sessions racing for one Space; every run consistent, see Pi Durable). It was not run with the ACP driver of this page against a coordinator-placed Space: the contract is the same, the measurements are not. A Space placed again from scratch is empty. What the agent had written there survives only if the driver can rebuild it (Pi Durable's transactional mode keeps the workspace image in its log); the ACP driver's journal holds the agent's conversation and tool ledger, not the Space's files, so the agent is told which calls completed but must recreate the files itself.

What is and is not guaranteed#

For every permission request the harness makes, the driver writes an intent record (call id, name, input digest, epoch) and syncs it to disk before it answers. It approves each call id once. If the same id is requested again while its outcome is unknown, for example after a crash between approval and result, the driver does not approve it: it records may_have_run and refuses, and the recovery note tells the model to verify first. With policy: "rerun-safe", calls the harness classifies as read-only (read, search, fetch) are approved again instead.

Outcomes come from the harness's tool_call updates and are written to the ledger. Tools that never ask permission only get started and outcome records, marked unguarded. After a crash they are listed as "may have partially run", but the driver cannot stop the harness from running them again. Exactly-once for those needs a hook inside the harness.

Guard mode (on by default) keeps the harness asking. The driver does not switch it to its full-access mode, approves with "allow once" instead of "allow always" (which would make the harness stop asking), puts Codex in its "Ask for approval" mode, and gives OpenCode a permission config that asks for bash, edit and webfetch. Both were measured: Codex's default mode never asks the client at all, and setting approval_policy has no effect (see Per-harness coverage). The Space is still the security boundary: the driver approves everything.

A driver that freezes while its process is alive (a stopped process, a deadlocked runtime) is replaced too. The driver prints a CUA-HEARTBEAT line every stall_secs / 4 seconds from a timer of its own. When a running driver's output has not grown for stall_secs (default 120; --stall-secs, 0 turns it off) the coordinator cancels it, records a stalled history entry, and resumes after the usual backoff. A stall counts as a drive toward max_attempts. A driver that has written nothing yet gets at least 600 s, for installs. A host that cannot be reached is not a stall; the lost-host path handles it. A driver that heartbeats but makes no progress (a model call that never returns while the timer runs) is not caught: heartbeats prove the process is alive, not that the agent is working.

Other limits:

• The fence is checked before every write, not atomically with it. The Space epoch (older-epoch processes are killed) is the backstop.
• A refusal cannot carry a message through ACP, so the model learns why from the recovery note, not from the refusal.
• Tool call ids are matched as the harness sends them. A harness that reuses an id for a different call while the first has no outcome sees the second refused.
• Side effects of a half-run tool (a partly written file, a process left running) are not undone. Use Pi Durable checkpoints for that.

Per-harness coverage#

The four measured rows come from real harness binaries (installed by the cua installer) driven against a scripted offline model, on a Linux host with Docker, with the faults described under "Crash campaign" (runner SIGKILL, harness SIGKILL, container kill, a second driver started while the old one lives, kills before and after the intent is journaled). No model key was involved. The other rows are from the run captured in tests/fixtures/recorded, made in each harness's full-access mode, so a harness that asked nothing there may ask in another mode. Resume support is read from each adapter's ACP initialize answer.

What the campaign showed, per kind of tool:

• Guarded tools were never run twice by the driver. The runner never saw a permission request for a call id it had already approved (0 in 200 runs), so its refusal path was exercised only by the runner's own tests. A guarded step ran twice only when the model itself issued it again under a new id (2 of 160 Claude Code effects, with a scripted model that does not read the recovery note).
• Unguarded tools are not exactly-once. Hermes ran 10 of 160 effects twice after a kill (6%): it reports a call only once it finished, so the ledger has no record of a call that was running when the process died and the recovery note cannot list it. OpenCode with its default permissions lost 11 of 160 effects (flagged in the note) and ran none twice in this run; that is not a guarantee.
• A killed call is listed, not repeated blindly. Every recovery note matched the ledger (180 of 180 notes across the five sets). A call killed before its intent was journaled is simply absent: the harness never got an answer and the scripted model moved on.

Closing the unguarded gap per harness:

• Claude Code: every state-changing tool asked in the measured version, so no hook is needed today. A PreToolUse hook would only add the read-only Read inside the working directory; it is not built. A future built-in that changes state without asking would need one (the hook would be a second writer to the ledger, and the call id the hook sees has not been compared with the ACP toolCallId).
• Codex: guard mode puts it in its "Ask for approval" mode. Measured on Codex 0.156.1: -c approval_policy=... on the command line has no effect, approval_policy = "untrusted" in config.toml is refused, and the default mode ("Approve for me") never asks the client. A command that stays inside Codex's own sandbox is not asked about in any mode but the full-access one, so a command that Codex runs sandboxed has only an outcome record.
• Harnesses with their own permission setting (OpenCode, Goose, Gemini): set that harness's permission to ask in its own config. Guard mode does it for OpenCode (bash, edit, webfetch).
• Hermes has no setting that makes every tool ask; there is no guard.

Driving it from a program#

let spec = cua_agents::durable_driver_spec("claude-code", "Fix the tests", &RunOptions {
    env: keys, // goes to the job's sealed env, never into argv or the bundle
    durable: Some(DurableOptions { session: Some("nightly".into()), ..Default::default() }),
    ..Default::default()
})?;
// spec.argv, spec.env, spec.bundle (gzip tar, under 1 MiB), spec.state_dir

DurableOptions has policy (ask-model or rerun-safe), max_agent_restarts (default 3) and guard (default true). For a run inside a Space, Agents::start with durable set and Agents::redrive(run_id, epoch) do the same job without a coordinator.