Keep an agent running through crashes
Run a coding agent so a crash of the process or computer driving it resumes the same session, with tool calls journaled before they run and a newer driver cutting off an older one.
A durable agent runs under a re-driver. When the process that drives the agent dies, or the computer it runs on does, the session is started again, the agent's own session is reloaded, and the agent is told what had already happened. It works with any harness that speaks the Agent Client Protocol (ACP); the same runner drives all of them.
A durable run is at-most-once for tool calls the harness asks permission for, and at-least-once-with-a-warning for the rest. Read What is and is not guaranteed before you rely on it.
Run one#
cua agent add nightly --harness claude-code --durable --on auto "Fix the failing tests in /work"
cua agent run --durable --on auto auto claude-code "Fix the failing tests in /work" --name nightlyThe two commands do the same: cua agent add --durable is cua agent run --durable with the agent's name as the session's name. --on picks the
computer the driver runs on (auto, or one of your computers by name; the
default is this one), --in a durable Space to work in (default none),
--max-attempts caps the drives (default 5) and --stall-secs sets when a
silent driver counts as wedged (default 120).
Follow and manage it as session:NAME:
cua agent sessions
cua agent logs session:nightly -f
cua agent status session:nightly
cua agent stop session:nightlyA durable session has no MCP tool yet, no home in the Cua Volume and no schedule; those are on named agents today.
The session keeps a journal in its state directory (CUA_SESSION_STATE_DIR,
or $CUA_JOB_DATA/session when that is empty). cua agent add --durable
leaves it empty, so the journal sits in the run's data folder on the computer
that drives it: it outlives the driver's Space, and a driver that dies or stalls
is started again on the same computer with the same conversation. A new drive on
another computer does not find it and starts the task from the beginning. It holds:
| File | What |
|---|---|
events.jsonl | Every ACP update, turn start and end, restart and notice |
state.json | Turn, ACP session id, whether the session was degraded |
tools.jsonl | The tool ledger |
epoch | The fence floor |
What to expect, by kind of failure#
Measured on real computers (Linux, Docker with gVisor, three computers on a private account, a scripted model so no model keys or spend; three runs per cell, so read the times as rough).
| What fails | What happens | Same conversation? |
|---|---|---|
The agent process dies (kill -9) | The runner restarts it in place and resumes the session, in seconds | Yes, the recovery note lists what ran and what may have run halfway |
| The runner (the driver process) dies | Noticed at once; a new driver starts a new Space on the same computer, 40 to 80 s with Claude Code and Codex | Yes by the journal: the harness's own session files are gone with the old Space, so the recovery note carries the journal's summary instead of the conversation |
| The driver's Space is killed or runs out of memory | The computer sees the Space stop within a few seconds and ends the driver; a new driver starts a new Space on the same computer, about 55 s from the kill to the resumed agent with Claude Code | Same as above |
| The driver's Space is removed, or the driver freezes | Noticed only by the stall limit (--stall-secs, default 120 s): about 140 to 170 s from the fault to the resumed agent | Same as above |
| The computer running the agent stops answering | After about 65 to 75 s the session records the lost computer and starts the next drive on another computer (see the cases below) | No. The task starts from the beginning |
| The daemon on the computer running the agent restarts (an update) | The driver is lost 2 to 11 s later (the job runner lives in the daemon) and the next drive starts on the same computer about a second after the 2 s backoff; it finishes 35 to 55 s after the restart. The old driver's Space is removed by the computer within 30 s | Yes, by the journal, on the same computer |
| The daemon on the computer that coordinates restarts | Nothing visible; a run on another computer is not disturbed (down 0.7 to 14 s, five runs) | Not applicable |
| The disk of the computer is full | Each drive crashes at its first write (exit 1, 38 to 61 s) and uses a try; the status line names ENOSPC. With the space freed in time the next drive finishes: two runs ended done on drive 4 of 5 | Yes, by the journal |
| The computer's clock is 10 minutes ahead or behind | The computer is shown down and a session pinned to it waits; the message says the clock is off. When the coordinator computer's clock is 10 minutes ahead the first drive starts about 86 s late; 10 minutes behind made no difference | Not applicable |
| The network is slow (300 ms added, 5% loss, three runs) | About 2.5 times slower (340 to 369 s against 138 s for a 20 step task); no restart and no move | Not applicable |
The lost-computer case in numbers (three computers, Claude Code and Codex, a scripted model; the computer was cut off with a firewall rule):
- Cut off from the account but still running (traffic to the relay dropped,
the model server still reachable), eight runs: the other computers declare it
lost after 64 to 66 s and the next drive is running 67 s after the cut. Its
first tool step comes 107 to 121 s after the cut (median 114 s, the six runs
where it was timed). The old
driver keeps working meanwhile: it did 32 to 36 steps on its own, and in
every run 2 to 4 of them while the new driver was already running (two
drivers at once for 8 to 17 s), and the new driver repeats every step the old
one had done. When the cut ends, the coordinator's cancel stops the old
driver within 2 to 11 s; the epoch fence does not, because it is a file on
each computer. A task whose side effects leave the computer (a
git push, an API call, a message) can run twice at the same time. - Cut off completely, four runs: lost after 65.6 to 65.9 s. The old driver cannot reach the model, so it stops by itself within a step; no overlap.
- When the cut-off computer also held the coordinator, six runs: the next drive starts 94 to 186 s after the cut, because the new coordinator starts counting the silence only when it takes over.
- Three computer losses in a row on one session: it moved between the two computers each time, finished on drive 4 of 5, and each drive started the task from the beginning. A drive that comes back to a computer that ran an earlier drive resumes that computer's old journal (and sees only its old working folder), not the progress made elsewhere.
- Every computer unreachable: the session waits (
waiting for a computer that can take work) with no deadline and uses no try; when the computers answer again it carries on, or endsfailedat the limit.
Other measurements: signing out of the account on the computer you type on leaves a
running session alone, but cua agent sessions then lists only the sessions of this
computer without a warning and other commands wait about a minute before saying you
are not signed in. Signing the computer that runs the agent out lets the driver
finish, but the session stays running until the computer signs in again. Starting
the same name again from another computer used to start a second driver; since
#6236 it does not. Twenty agents pinned to one 4 CPU, 7.7 GB computer took 1643 s one
at a time when each driver reserved 4 GiB; the driver now reserves 1.5 GiB and four
run at once (12 of 12 sessions finished on drive 1 with no out-of-memory kill). A
killed agent waits for a free slot when the computer is full: 158 s from the kill to
the resumed request with six running.
By harness (a runner kill, from the kill to the first model request that carries the recovery note): Claude Code 50 to 80 s, Codex 38 s, OpenCode 47 s (one run), Hermes 55 to 62 s since the fix that gives it a new session when it lost its own (three runs), Pi 49 s (one run, since Pi runs through the same driver as the others). Three runs per cell elsewhere, so read them as rough.
When a drive moves to another computer, the new one does not have the journal, the conversation or the working folder of the old one. Plan for it:
- Put the files the task needs, and the results you want to keep, in
Cua Volume. The default storage, This Mac, is a
folder on one computer, so another computer does not see it. Use your own
S3-compatible bucket (AWS S3, R2, MinIO) so every computer reaches the same
files.
cua agent add --durable --on autosays so when more than one computer is signed in and Cua Volume is still on This Mac. The same goes for a durable Space (durable=true): it is created again on another computer with an empty disk, and a Volume on This Mac is not visible from there. - Write the task so running it again from the start is safe.
- A session makes at most 5 drives by default (
--max-attempts), counting the first one and every restart for any reason. When they are used up it endsfailedand says it gave up after 5 of 5 drives, with the last failure. - A drive that no computer has room for waits and says what it waits for, for
example
waiting for a machine with 1 CPU and 1.5 GB memory free. A driver that runs a harness asks for 1 CPU and 1.5 GiB, so a computer with 8 GB runs four of them at once. - A durable Space whose computer is lost is created again on another computer within about a minute, with the same name and an empty disk; the old record stays listed as offline until you delete it.
cua agent logs session:NAME shows one short line for each event: the driver
and the agent's session starting, each turn starting and ending, the kind and
outcome of each tool call (tool execute completed, never the command or its
output), restarts and recovery notes, and errors. It never prints your prompt,
a tool's input or output, or the agent's text. It shows the latest drive's
output; cua agent status has the history of every drive.
What happens when it crashes#
- The re-driver issues a higher epoch and starts the driver again, on the computer that ran the last drive when it has room, else on another one (which starts the task from the beginning, see above).
- The new driver raises the fence first. An older driver that is still alive on the same computer sees the higher floor on its next write, stops, and exits with code 75 without writing anything. The fence is a file in the computer's own run folder, so it cannot reach an old driver on a computer that is cut off from the account; the coordinator's cancel stops that one when the computer is reachable again.
- The driver reloads the agent's session (
session/resume, elsesession/load). A harness with neither gets a new session and the run is markeddegraded: the recovery note then carries a summary of the journal in place of the lost conversation. - The first prompt of the resumed turn is a recovery note. It names the interrupted request, the tool calls that completed (with a short result), and the calls that may have partially run, with the instruction to check the current state before repeating them. A turn that already ended is never replayed.
- If the agent process itself dies, the driver restarts it in place (up to three times) and resumes the same way before it gives up and exits non-zero for the re-driver.
Exit codes: 0 finished, 78 cannot succeed (authentication, agent not
found, bad configuration), 75 fenced by a newer epoch; anything else is a
crash and the session is resumed with backoff.
Working in a durable Space#
A session can be bound to a durable Space (spec.space, a Space the coordinator
places on one of your hosts and keeps). Each drive takes the Space's lease and the
driver gets the Space's name, address, token and lease token
(CUA_SESSION_SPACE, _ADDR, _TOKEN, _LEASE); the epoch is the larger of the
session's counter plus one and the lease token, so both only grow. One session
holds a Space at a time: a second one stays pending with the reason
held by session:NAME until the first ends, and a superseded lease token is refused. If
the Space's host loses it, the coordinator places it again as the next generation,
restored: "from-scratch", and the next drive receives the new address and token.
This was run for real with the Pi Durable driver (40 runs on Linux with Docker: driver kills, the Space removed after a kill and under a live driver, a coordinator restart, two sessions racing for one Space; every run consistent, see Build a durable agent in code). It was not run with the ACP driver of this page against a coordinator-placed Space: the contract is the same, the measurements are not. A Space placed again from scratch is empty. What the agent had written there survives only if the driver can rebuild it (Pi Durable's transactional mode keeps the workspace image in its log); the ACP driver's journal holds the agent's conversation and tool ledger, not the Space's files, so the agent is told which calls completed but must recreate the files itself.
Files on all your computers#
With one computer signed in, the Volume lives on that computer only. Once you are signed in on a second computer and no bucket is set as the Volume's storage, the Cua Spaces daemon keeps a copy of the Volume on each of your computers, with no cloud involved. One computer owns writes at a time and the others receive them. A write counts as durable when two computers hold it, and it never waits for a computer that is away. When the owner goes away, the most reliable computer with a copy takes over, and if the old owner wakes up later it is fenced: it loses nothing and writes nothing over the new owner.
Durable Spaces are placed on the most reliable computer that has room: an always-on desktop before a laptop on power, a laptop on power before a laptop on battery, and never the laptop you are using unless it is the only one that fits. Closing a laptop's lid still puts it to sleep unless it is plugged in with a display attached, and Cua says so plainly instead of promising otherwise.
Replication starts by itself and needs no setting. To turn it off, start the
daemon with CUA_REPLICA=0. Turning it on never costs data: the Volume's
current heads are backed up first, a Volume prepared by a newer Cua Spaces is
refused with a message instead of being rewritten, and a build that does not
know replication can still read the same folder. History from before a second
computer joined stays on the computer that wrote it, and two copies are kept.
What is and is not guaranteed#
For every permission request the harness makes, the driver writes an intent
record (call id, name, input digest, epoch) and syncs it to disk before it
answers. It approves each call id once. If the same id is requested again while
its outcome is unknown, for example after a crash between approval and result,
the driver does not approve it: it records may_have_run and refuses, and the
recovery note tells the model to verify first. With policy: "rerun-safe",
calls the harness classifies as read-only (read, search, fetch) are approved
again instead.
Outcomes come from the harness's tool_call updates and are written to the
ledger. Tools that never ask permission only get started and outcome
records, marked unguarded. After a crash they are listed as "may have
partially run", but the driver cannot stop the harness from running them
again. Exactly-once for those needs a hook inside the harness.
Guard mode (on by default) keeps the harness asking. The driver does not
switch it to its full-access mode, approves with "allow once" instead of
"allow always" (which would make the harness stop asking), puts Codex in its
"Ask for approval" mode, and gives OpenCode a permission config that asks for
bash, edit and webfetch. Both were measured: Codex's default mode never asks
the client at all, and setting approval_policy has no effect (see
Per-harness coverage). The Space is still the security
boundary: the driver approves everything.
A driver that freezes while its process is alive (a stopped process, a
deadlocked runtime) is replaced too. The driver prints a CUA-HEARTBEAT line
every stall_secs / 4 seconds from a timer of its own. When a running
driver's output has not grown for stall_secs (default 120; --stall-secs,
0 turns it off) the coordinator cancels it, records a stalled history entry,
and resumes after the usual backoff. A stall counts as a drive toward
max_attempts. A driver that has written nothing yet gets at least 600 s, for
installs. A host that cannot be reached is not a stall; the lost-host path
handles it. A driver that heartbeats but makes no progress (a model call that
never returns while the timer runs) is not caught: heartbeats prove the process
is alive, not that the agent is working.
Other limits:
- The fence is checked before every write, not atomically with it. The Space epoch (older-epoch processes are killed) is the backstop.
- A refusal cannot carry a message through ACP, so the model learns why from the recovery note, not from the refusal.
- Tool call ids are matched as the harness sends them. A harness that reuses an id for a different call while the first has no outcome sees the second refused.
- Side effects of a half-run tool (a partly written file, a process left running) are not undone. Use Pi Durable checkpoints (Build a durable agent in code) for that.
Per-harness coverage#
| Harness | Resume | Asks permission (guarded, exactly once) | Never asks (unguarded, outcome only) | Measured |
|---|---|---|---|---|
| Claude Code 2.1.281 | session/resume and session/load | Bash, Write, Edit, NotebookEdit, WebFetch, WebSearch, and Read outside the working directory | Read inside the working directory | 40/40 runs, 37 kills; every state-changing tool guarded |
| OpenAI Codex 0.156.1 | session/resume and session/load | Commands that ask to leave Codex's sandbox, in the "Ask for approval" mode that guard mode selects | A command that stays inside Codex's own sandbox | 40/40 runs, 40 kills; guarded only for commands that escalate |
| Hermes 2026.9.21 | session/resume and session/load | Dangerous commands only (rm -rf asked in the probe) | All other tools; a call is reported only after it finished | 40/40 runs, 32 kills; 10 of 160 effects ran twice (no exactly-once) |
| OpenCode 1.18.32 | session/resume and session/load | bash, edit, webfetch (guard mode sets OPENCODE_PERMISSION) | Other tools; every tool when guard is off | 40/40 runs with guard (40 kills), 40/40 without (32 kills, no duplicate seen) |
| Gemini CLI | session/load | Tools in the default approval mode (a permission was asked in the recorded run) | Read-only tools, and everything in yolo mode | Not run durably |
| Google Antigravity | session/resume and session/load | Commands (asked in the recorded run) | Not known | Not run durably |
| Goose | session/load | None seen in the recorded run | All tools seen | Not run durably |
| Pi | session/load | None seen | All tools seen | Runs through the same driver as the others (pi-acp adapter): 6 steps done, and one runner kill resumed with the recovery note in 49 s (one run each, so not a campaign) |
| OpenClaw | session/resume and session/load | None seen | All tools seen | The driver spec is refused (needs a launch wrapper) |
The four measured rows come from real harness binaries (installed by the cua
installer) driven against a scripted offline model, on a Linux host with
Docker, with the faults described under "Crash campaign" (runner SIGKILL,
harness SIGKILL, container kill, a second driver started while the old one
lives, kills before and after the intent is journaled). No model key was
involved. The other rows are from the run captured in tests/fixtures/recorded,
made in each harness's full-access mode, so a harness that asked nothing there
may ask in another mode. Resume support is read from each adapter's ACP
initialize answer.
What the campaign showed, per kind of tool:
- Guarded tools were never run twice by the driver. The runner never saw a permission request for a call id it had already approved (0 in 200 runs), so its refusal path was exercised only by the runner's own tests. A guarded step ran twice only when the model itself issued it again under a new id (2 of 160 Claude Code effects, with a scripted model that does not read the recovery note).
- Unguarded tools are not exactly-once. Hermes ran 10 of 160 effects twice after a kill (6%): it reports a call only once it finished, so the ledger has no record of a call that was running when the process died and the recovery note cannot list it. OpenCode with its default permissions lost 11 of 160 effects (flagged in the note) and ran none twice in this run; that is not a guarantee.
- A killed call is listed, not repeated blindly. Every recovery note matched the ledger (180 of 180 notes across the five sets). A call killed before its intent was journaled is simply absent: the harness never got an answer and the scripted model moved on.
Closing the unguarded gap per harness:
- Claude Code: every state-changing tool asked in the measured version, so
no hook is needed today. A
PreToolUsehook would only add the read-only Read inside the working directory; it is not built. A future built-in that changes state without asking would need one (the hook would be a second writer to the ledger, and the call id the hook sees has not been compared with the ACPtoolCallId). - Codex: guard mode puts it in its "Ask for approval" mode. Measured on
Codex 0.156.1:
-c approval_policy=...on the command line has no effect,approval_policy = "untrusted"inconfig.tomlis refused, and the default mode ("Approve for me") never asks the client. A command that stays inside Codex's own sandbox is not asked about in any mode but the full-access one, so a command that Codex runs sandboxed has only an outcome record. - Harnesses with their own permission setting (OpenCode, Goose, Gemini):
set that harness's permission to
askin its own config. Guard mode does it for OpenCode (bash, edit, webfetch). - Hermes has no setting that makes every tool ask; there is no guard.
Driving it from a program#
let spec = cua_agents::durable_driver_spec("claude-code", "Fix the tests", &RunOptions {
env: keys, // goes to the job's sealed env, never into argv or the bundle
durable: Some(DurableOptions { session: Some("nightly".into()), ..Default::default() }),
..Default::default()
})?;
// spec.argv, spec.env, spec.bundle (gzip tar, under 1 MiB), spec.state_dirDurableOptions has policy (ask-model or rerun-safe),
max_agent_restarts (default 3) and guard (default true). For a run inside
a Space, Agents::start with durable set and Agents::redrive(run_id, epoch)
do the same job without a coordinator.