Keep an agent running through crashes

Run a coding agent so a crash of the process or computer driving it resumes the same session, with tool calls journaled before they run and a newer driver cutting off an older one.

A durable agent runs under a re-driver. When the process that drives the agent dies, or the computer it runs on does, the session is started again, the agent's own session is reloaded, and the agent is told what had already happened. It works with any harness that speaks the Agent Client Protocol (ACP); the same runner drives all of them.

A durable run is at-most-once for tool calls the harness asks permission for, and at-least-once-with-a-warning for the rest. Read What is and is not guaranteed before you rely on it.

Run one#

cua agent add nightly --harness claude-code --durable --on auto "Fix the failing tests in /work"
cua agent run --durable --on auto auto claude-code "Fix the failing tests in /work" --name nightly

The two commands do the same: cua agent add --durable is cua agent run --durable with the agent's name as the session's name. --on picks the computer the driver runs on (auto, or one of your computers by name; the default is this one), --in a durable Space to work in (default none), --max-attempts caps the drives (default 5) and --stall-secs sets when a silent driver counts as wedged (default 120).

Follow and manage it as session:NAME:

cua agent sessions
cua agent logs session:nightly -f
cua agent status session:nightly
cua agent stop session:nightly

A durable session has no MCP tool yet, no home in the Cua Volume and no schedule; those are on named agents today.

The session keeps a journal in its state directory (CUA_SESSION_STATE_DIR, or $CUA_JOB_DATA/session when that is empty). cua agent add --durable leaves it empty, so the journal sits in the run's data folder on the computer that drives it: it outlives the driver's Space, and a driver that dies or stalls is started again on the same computer with the same conversation. A new drive on another computer does not find it and starts the task from the beginning. It holds:

FileWhat
events.jsonlEvery ACP update, turn start and end, restart and notice
state.jsonTurn, ACP session id, whether the session was degraded
tools.jsonlThe tool ledger
epochThe fence floor

What to expect, by kind of failure#

Measured on real computers (Linux, Docker with gVisor, three computers on a private account, a scripted model so no model keys or spend; three runs per cell, so read the times as rough).

What failsWhat happensSame conversation?
The agent process dies (kill -9)The runner restarts it in place and resumes the session, in secondsYes, the recovery note lists what ran and what may have run halfway
The runner (the driver process) diesNoticed at once; a new driver starts a new Space on the same computer, 40 to 80 s with Claude Code and CodexYes by the journal: the harness's own session files are gone with the old Space, so the recovery note carries the journal's summary instead of the conversation
The driver's Space is killed or runs out of memoryThe computer sees the Space stop within a few seconds and ends the driver; a new driver starts a new Space on the same computer, about 55 s from the kill to the resumed agent with Claude CodeSame as above
The driver's Space is removed, or the driver freezesNoticed only by the stall limit (--stall-secs, default 120 s): about 140 to 170 s from the fault to the resumed agentSame as above
The computer running the agent stops answeringAfter about 65 to 75 s the session records the lost computer and starts the next drive on another computer (see the cases below)No. The task starts from the beginning
The daemon on the computer running the agent restarts (an update)The driver is lost 2 to 11 s later (the job runner lives in the daemon) and the next drive starts on the same computer about a second after the 2 s backoff; it finishes 35 to 55 s after the restart. The old driver's Space is removed by the computer within 30 sYes, by the journal, on the same computer
The daemon on the computer that coordinates restartsNothing visible; a run on another computer is not disturbed (down 0.7 to 14 s, five runs)Not applicable
The disk of the computer is fullEach drive crashes at its first write (exit 1, 38 to 61 s) and uses a try; the status line names ENOSPC. With the space freed in time the next drive finishes: two runs ended done on drive 4 of 5Yes, by the journal
The computer's clock is 10 minutes ahead or behindThe computer is shown down and a session pinned to it waits; the message says the clock is off. When the coordinator computer's clock is 10 minutes ahead the first drive starts about 86 s late; 10 minutes behind made no differenceNot applicable
The network is slow (300 ms added, 5% loss, three runs)About 2.5 times slower (340 to 369 s against 138 s for a 20 step task); no restart and no moveNot applicable

The lost-computer case in numbers (three computers, Claude Code and Codex, a scripted model; the computer was cut off with a firewall rule):

  • Cut off from the account but still running (traffic to the relay dropped, the model server still reachable), eight runs: the other computers declare it lost after 64 to 66 s and the next drive is running 67 s after the cut. Its first tool step comes 107 to 121 s after the cut (median 114 s, the six runs where it was timed). The old driver keeps working meanwhile: it did 32 to 36 steps on its own, and in every run 2 to 4 of them while the new driver was already running (two drivers at once for 8 to 17 s), and the new driver repeats every step the old one had done. When the cut ends, the coordinator's cancel stops the old driver within 2 to 11 s; the epoch fence does not, because it is a file on each computer. A task whose side effects leave the computer (a git push, an API call, a message) can run twice at the same time.
  • Cut off completely, four runs: lost after 65.6 to 65.9 s. The old driver cannot reach the model, so it stops by itself within a step; no overlap.
  • When the cut-off computer also held the coordinator, six runs: the next drive starts 94 to 186 s after the cut, because the new coordinator starts counting the silence only when it takes over.
  • Three computer losses in a row on one session: it moved between the two computers each time, finished on drive 4 of 5, and each drive started the task from the beginning. A drive that comes back to a computer that ran an earlier drive resumes that computer's old journal (and sees only its old working folder), not the progress made elsewhere.
  • Every computer unreachable: the session waits (waiting for a computer that can take work) with no deadline and uses no try; when the computers answer again it carries on, or ends failed at the limit.

Other measurements: signing out of the account on the computer you type on leaves a running session alone, but cua agent sessions then lists only the sessions of this computer without a warning and other commands wait about a minute before saying you are not signed in. Signing the computer that runs the agent out lets the driver finish, but the session stays running until the computer signs in again. Starting the same name again from another computer used to start a second driver; since #6236 it does not. Twenty agents pinned to one 4 CPU, 7.7 GB computer took 1643 s one at a time when each driver reserved 4 GiB; the driver now reserves 1.5 GiB and four run at once (12 of 12 sessions finished on drive 1 with no out-of-memory kill). A killed agent waits for a free slot when the computer is full: 158 s from the kill to the resumed request with six running.

By harness (a runner kill, from the kill to the first model request that carries the recovery note): Claude Code 50 to 80 s, Codex 38 s, OpenCode 47 s (one run), Hermes 55 to 62 s since the fix that gives it a new session when it lost its own (three runs), Pi 49 s (one run, since Pi runs through the same driver as the others). Three runs per cell elsewhere, so read them as rough.

When a drive moves to another computer, the new one does not have the journal, the conversation or the working folder of the old one. Plan for it:

  • Put the files the task needs, and the results you want to keep, in Cua Volume. The default storage, This Mac, is a folder on one computer, so another computer does not see it. Use your own S3-compatible bucket (AWS S3, R2, MinIO) so every computer reaches the same files. cua agent add --durable --on auto says so when more than one computer is signed in and Cua Volume is still on This Mac. The same goes for a durable Space (durable=true): it is created again on another computer with an empty disk, and a Volume on This Mac is not visible from there.
  • Write the task so running it again from the start is safe.
  • A session makes at most 5 drives by default (--max-attempts), counting the first one and every restart for any reason. When they are used up it ends failed and says it gave up after 5 of 5 drives, with the last failure.
  • A drive that no computer has room for waits and says what it waits for, for example waiting for a machine with 1 CPU and 1.5 GB memory free. A driver that runs a harness asks for 1 CPU and 1.5 GiB, so a computer with 8 GB runs four of them at once.
  • A durable Space whose computer is lost is created again on another computer within about a minute, with the same name and an empty disk; the old record stays listed as offline until you delete it.

cua agent logs session:NAME shows one short line for each event: the driver and the agent's session starting, each turn starting and ending, the kind and outcome of each tool call (tool execute completed, never the command or its output), restarts and recovery notes, and errors. It never prints your prompt, a tool's input or output, or the agent's text. It shows the latest drive's output; cua agent status has the history of every drive.

What happens when it crashes#

  1. The re-driver issues a higher epoch and starts the driver again, on the computer that ran the last drive when it has room, else on another one (which starts the task from the beginning, see above).
  2. The new driver raises the fence first. An older driver that is still alive on the same computer sees the higher floor on its next write, stops, and exits with code 75 without writing anything. The fence is a file in the computer's own run folder, so it cannot reach an old driver on a computer that is cut off from the account; the coordinator's cancel stops that one when the computer is reachable again.
  3. The driver reloads the agent's session (session/resume, else session/load). A harness with neither gets a new session and the run is marked degraded: the recovery note then carries a summary of the journal in place of the lost conversation.
  4. The first prompt of the resumed turn is a recovery note. It names the interrupted request, the tool calls that completed (with a short result), and the calls that may have partially run, with the instruction to check the current state before repeating them. A turn that already ended is never replayed.
  5. If the agent process itself dies, the driver restarts it in place (up to three times) and resumes the same way before it gives up and exits non-zero for the re-driver.

Exit codes: 0 finished, 78 cannot succeed (authentication, agent not found, bad configuration), 75 fenced by a newer epoch; anything else is a crash and the session is resumed with backoff.

Working in a durable Space#

A session can be bound to a durable Space (spec.space, a Space the coordinator places on one of your hosts and keeps). Each drive takes the Space's lease and the driver gets the Space's name, address, token and lease token (CUA_SESSION_SPACE, _ADDR, _TOKEN, _LEASE); the epoch is the larger of the session's counter plus one and the lease token, so both only grow. One session holds a Space at a time: a second one stays pending with the reason held by session:NAME until the first ends, and a superseded lease token is refused. If the Space's host loses it, the coordinator places it again as the next generation, restored: "from-scratch", and the next drive receives the new address and token.

This was run for real with the Pi Durable driver (40 runs on Linux with Docker: driver kills, the Space removed after a kill and under a live driver, a coordinator restart, two sessions racing for one Space; every run consistent, see Build a durable agent in code). It was not run with the ACP driver of this page against a coordinator-placed Space: the contract is the same, the measurements are not. A Space placed again from scratch is empty. What the agent had written there survives only if the driver can rebuild it (Pi Durable's transactional mode keeps the workspace image in its log); the ACP driver's journal holds the agent's conversation and tool ledger, not the Space's files, so the agent is told which calls completed but must recreate the files itself.

Files on all your computers#

With one computer signed in, the Volume lives on that computer only. Once you are signed in on a second computer and no bucket is set as the Volume's storage, the Cua Spaces daemon keeps a copy of the Volume on each of your computers, with no cloud involved. One computer owns writes at a time and the others receive them. A write counts as durable when two computers hold it, and it never waits for a computer that is away. When the owner goes away, the most reliable computer with a copy takes over, and if the old owner wakes up later it is fenced: it loses nothing and writes nothing over the new owner.

Durable Spaces are placed on the most reliable computer that has room: an always-on desktop before a laptop on power, a laptop on power before a laptop on battery, and never the laptop you are using unless it is the only one that fits. Closing a laptop's lid still puts it to sleep unless it is plugged in with a display attached, and Cua says so plainly instead of promising otherwise.

Replication starts by itself and needs no setting. To turn it off, start the daemon with CUA_REPLICA=0. Turning it on never costs data: the Volume's current heads are backed up first, a Volume prepared by a newer Cua Spaces is refused with a message instead of being rewritten, and a build that does not know replication can still read the same folder. History from before a second computer joined stays on the computer that wrote it, and two copies are kept.

What is and is not guaranteed#

For every permission request the harness makes, the driver writes an intent record (call id, name, input digest, epoch) and syncs it to disk before it answers. It approves each call id once. If the same id is requested again while its outcome is unknown, for example after a crash between approval and result, the driver does not approve it: it records may_have_run and refuses, and the recovery note tells the model to verify first. With policy: "rerun-safe", calls the harness classifies as read-only (read, search, fetch) are approved again instead.

Outcomes come from the harness's tool_call updates and are written to the ledger. Tools that never ask permission only get started and outcome records, marked unguarded. After a crash they are listed as "may have partially run", but the driver cannot stop the harness from running them again. Exactly-once for those needs a hook inside the harness.

Guard mode (on by default) keeps the harness asking. The driver does not switch it to its full-access mode, approves with "allow once" instead of "allow always" (which would make the harness stop asking), puts Codex in its "Ask for approval" mode, and gives OpenCode a permission config that asks for bash, edit and webfetch. Both were measured: Codex's default mode never asks the client at all, and setting approval_policy has no effect (see Per-harness coverage). The Space is still the security boundary: the driver approves everything.

A driver that freezes while its process is alive (a stopped process, a deadlocked runtime) is replaced too. The driver prints a CUA-HEARTBEAT line every stall_secs / 4 seconds from a timer of its own. When a running driver's output has not grown for stall_secs (default 120; --stall-secs, 0 turns it off) the coordinator cancels it, records a stalled history entry, and resumes after the usual backoff. A stall counts as a drive toward max_attempts. A driver that has written nothing yet gets at least 600 s, for installs. A host that cannot be reached is not a stall; the lost-host path handles it. A driver that heartbeats but makes no progress (a model call that never returns while the timer runs) is not caught: heartbeats prove the process is alive, not that the agent is working.

Other limits:

  • The fence is checked before every write, not atomically with it. The Space epoch (older-epoch processes are killed) is the backstop.
  • A refusal cannot carry a message through ACP, so the model learns why from the recovery note, not from the refusal.
  • Tool call ids are matched as the harness sends them. A harness that reuses an id for a different call while the first has no outcome sees the second refused.
  • Side effects of a half-run tool (a partly written file, a process left running) are not undone. Use Pi Durable checkpoints (Build a durable agent in code) for that.

Per-harness coverage#

HarnessResumeAsks permission (guarded, exactly once)Never asks (unguarded, outcome only)Measured
Claude Code 2.1.281session/resume and session/loadBash, Write, Edit, NotebookEdit, WebFetch, WebSearch, and Read outside the working directoryRead inside the working directory40/40 runs, 37 kills; every state-changing tool guarded
OpenAI Codex 0.156.1session/resume and session/loadCommands that ask to leave Codex's sandbox, in the "Ask for approval" mode that guard mode selectsA command that stays inside Codex's own sandbox40/40 runs, 40 kills; guarded only for commands that escalate
Hermes 2026.9.21session/resume and session/loadDangerous commands only (rm -rf asked in the probe)All other tools; a call is reported only after it finished40/40 runs, 32 kills; 10 of 160 effects ran twice (no exactly-once)
OpenCode 1.18.32session/resume and session/loadbash, edit, webfetch (guard mode sets OPENCODE_PERMISSION)Other tools; every tool when guard is off40/40 runs with guard (40 kills), 40/40 without (32 kills, no duplicate seen)
Gemini CLIsession/loadTools in the default approval mode (a permission was asked in the recorded run)Read-only tools, and everything in yolo modeNot run durably
Google Antigravitysession/resume and session/loadCommands (asked in the recorded run)Not knownNot run durably
Goosesession/loadNone seen in the recorded runAll tools seenNot run durably
Pisession/loadNone seenAll tools seenRuns through the same driver as the others (pi-acp adapter): 6 steps done, and one runner kill resumed with the recovery note in 49 s (one run each, so not a campaign)
OpenClawsession/resume and session/loadNone seenAll tools seenThe driver spec is refused (needs a launch wrapper)

The four measured rows come from real harness binaries (installed by the cua installer) driven against a scripted offline model, on a Linux host with Docker, with the faults described under "Crash campaign" (runner SIGKILL, harness SIGKILL, container kill, a second driver started while the old one lives, kills before and after the intent is journaled). No model key was involved. The other rows are from the run captured in tests/fixtures/recorded, made in each harness's full-access mode, so a harness that asked nothing there may ask in another mode. Resume support is read from each adapter's ACP initialize answer.

What the campaign showed, per kind of tool:

  • Guarded tools were never run twice by the driver. The runner never saw a permission request for a call id it had already approved (0 in 200 runs), so its refusal path was exercised only by the runner's own tests. A guarded step ran twice only when the model itself issued it again under a new id (2 of 160 Claude Code effects, with a scripted model that does not read the recovery note).
  • Unguarded tools are not exactly-once. Hermes ran 10 of 160 effects twice after a kill (6%): it reports a call only once it finished, so the ledger has no record of a call that was running when the process died and the recovery note cannot list it. OpenCode with its default permissions lost 11 of 160 effects (flagged in the note) and ran none twice in this run; that is not a guarantee.
  • A killed call is listed, not repeated blindly. Every recovery note matched the ledger (180 of 180 notes across the five sets). A call killed before its intent was journaled is simply absent: the harness never got an answer and the scripted model moved on.

Closing the unguarded gap per harness:

  • Claude Code: every state-changing tool asked in the measured version, so no hook is needed today. A PreToolUse hook would only add the read-only Read inside the working directory; it is not built. A future built-in that changes state without asking would need one (the hook would be a second writer to the ledger, and the call id the hook sees has not been compared with the ACP toolCallId).
  • Codex: guard mode puts it in its "Ask for approval" mode. Measured on Codex 0.156.1: -c approval_policy=... on the command line has no effect, approval_policy = "untrusted" in config.toml is refused, and the default mode ("Approve for me") never asks the client. A command that stays inside Codex's own sandbox is not asked about in any mode but the full-access one, so a command that Codex runs sandboxed has only an outcome record.
  • Harnesses with their own permission setting (OpenCode, Goose, Gemini): set that harness's permission to ask in its own config. Guard mode does it for OpenCode (bash, edit, webfetch).
  • Hermes has no setting that makes every tool ask; there is no guard.

Driving it from a program#

let spec = cua_agents::durable_driver_spec("claude-code", "Fix the tests", &RunOptions {
    env: keys, // goes to the job's sealed env, never into argv or the bundle
    durable: Some(DurableOptions { session: Some("nightly".into()), ..Default::default() }),
    ..Default::default()
})?;
// spec.argv, spec.env, spec.bundle (gzip tar, under 1 MiB), spec.state_dir

DurableOptions has policy (ask-model or rerun-safe), max_agent_restarts (default 3) and guard (default true). For a run inside a Space, Agents::start with durable set and Agents::redrive(run_id, epoch) do the same job without a coordinator.