Build a durable agent in code

Run Pi Durable tools in a Cua Space and keep the sandbox consistent with the conversation log across crashes: idempotent commands, fencing, and checkpoint-based recovery.

Pi Durable records every step of an agent in a durable log, so a crashed worker resumes where it stopped. The log is only half of the state, though: the tools change files and processes in a sandbox, and a crash between a tool's start and its committed result leaves the sandbox in an unknown state. On its own, Pi then reports the call as interrupted and the sandbox keeps whatever the crash left behind.

@trycua/pi-durable (MIT, in libs/cua/typescript/integrations/pi-durable) runs Pi Durable's tools in a Space and closes that gap.

Set it up#

import { embedded, SandboxCreateOptions } from "@trycua/cua"
import { createRegistry, Harness } from "@earendil-works/pi-durable"
import { CodingTools } from "@earendil-works/pi-durable/tools"
import {
  CuaSandboxSpaces,
  DirectoryCheckpointStore,
  durableSpaces,
  WorkspaceSnapshots,
} from "@trycua/pi-durable"
 
const spaces = new CuaSandboxSpaces({
  sandboxes: embedded().sandboxes(),
  createOptions: (name) => SandboxCreateOptions.create({ image: "linux", name }),
})
const durable = durableSpaces({
  spaces,
  snapshots: new WorkspaceSnapshots({
    spaces,
    root: "/home/cua/work",
    // Optional: a copy of every checkpoint outside the Space, so a lost Space
    // is rebuilt from its newest checkpoint, on this machine or another.
    store: new DirectoryCheckpointStore("/srv/agent-checkpoints"),
  }),
  cwd: "/home/cua/work",
})
 
const registry = createRegistry()
registry.install(CodingTools)
registry.install(durable.extension) // wraps bash, write and edit
const harness = await Harness.open(storage, { models, registry, env: durable.env }, context)

Each conversation gets its own Space, named from the conversation id. durable.env is the Harness's env factory; durable.extension wraps the mutating tools. The Space needs a cua-spacesd that reports the process.exec_ledger capability.

Whole-machine checkpoints#

WorkspaceSnapshots captures one directory. On a microVM Space, checkpoint the whole machine instead: SandboxSnapshots uses the cua daemon's checkpoints (cua sb checkpoint / fork), which hold memory and disk, so a restore brings back running processes (servers, browsers, shells) exactly as they were, and the checkpoints live with the daemon, outside the Space.

import { connect, SandboxCreateOptions } from "@trycua/cua"
import { CuaSandboxSpaces, durableSpaces, SandboxSnapshots } from "@trycua/pi-durable"
 
const sandboxes = connect().sandboxes() // the cua daemon owns the microVMs
const spaces = new CuaSandboxSpaces({
  sandboxes,
  createOptions: (name) =>
    SandboxCreateOptions.create({ image: "linux", name, kind: "microvm" }),
})
const durable = durableSpaces({
  spaces,
  snapshots: new SandboxSnapshots({ sandboxes }),
  cwd: "/home/cua/work",
})

A restore replaces the Space with a fork of the checkpoint under the same name. Use the daemon (connect()), not an embedded runtime, so the Spaces outlive the process that runs the Harness.

Modes: transactional and checkpoint#

durableSpaces({ mode }) chooses how a Space is kept consistent with the log. The default is "auto".

• checkpoint: a checkpoint of the Space (or its workspace) after every call, as described under "How it works".

• transactional: no per-call checkpoint. The log holds a content-addressed image of the workspace (cua.space.image: a path table and the bytes of each distinct file). Before a call the adapter scans the workspace (find, stat, sha256sum through cua-spacesd) and commits any change made outside the calls together with the call's start mark; after the call it commits the call's diff in the same log transaction as its outcome. The image is always the workspace as of the last committed call. After a crash the Space is made equal to it again: files that are missing or differ are written back and files the log does not have are deleted. A Space that was lost is rebuilt from the log alone.

• auto: transactional when all of these hold, otherwise checkpoint:

  Condition	Transactional	Checkpoint
  Space kind	container, gvisor, or any kind with only workspace checkpoints available	microvm or vm when the snapshot provider is whole-machine (SandboxSnapshots), or an unknown kind with such a provider
  Workspace	at most transactional.maxFiles (1000) files and maxBytes (16 MiB)	larger
  Processes	no wrapped bash call left a process running in the workspace	a call left one (Linux, read from /proc)
  Tooling	find, stat, xargs, sha256sum; no name with a backslash or newline	missing
  History	new conversation, or one already using the image	a conversation that already has checkpoints

  The limits are checked at every call boundary. The first time one is crossed (before a call, or by what a call did) the conversation switches to checkpoint mode permanently, the reason is recorded in the cua.space document (mode, modeReason) and reported as a mode recovery event. When a call crosses the limit, its outcome and the switch are committed together and the checkpoint follows.

mode: "transactional" never falls back: a workspace over the limits is refused before the call, and one that outgrows them during a call marks the image stale, so the next call fails with an error telling you to use checkpoint. Tune the limits with transactional: { maxFiles, maxBytes, exclude } (exclude lists workspace-relative paths left out of the image).

What a call costs in transactional mode: two scans of the workspace (a few milliseconds for hundreds of files) and, in the log, only the bytes that changed (base64, plus the path table). The first call stores the whole workspace once, up to about 21 MiB at the default limit. Rewindable history keeps each call's delta, so the log grows with the amount of change, not with the number of calls times the workspace size.

Recovery in transactional mode, per call:

Every RecoveryEvent carries the mode it was decided in; transactional mode adds reverted and reimaged events.

Fencing#

A session's epoch is a number that only grows: the coordinator issues one for every drive attempt (see the driver below). Each resource a driver writes to refuses epochs below its floor, and the new driver raises the floor when it starts, which cuts off the older one:

• The Space. Commands carry at least the session epoch (epoch option or CUA_SESSION_EPOCH); the Space is advanced to it on first use (or earlier with durable.raiseEpoch(name)), killing older processes.
• The log. openFencedSqliteStorage(path, epoch) (or openFencedSqlite, which also returns the fence) returns a Pi SqliteStorage whose every commit first reads a one-row floor table (cua_fence) inside the same SQLite transaction, so a write with an older epoch is refused atomically and a newer epoch raises the floor. FencedFileSystem does the same for a JsonlStorage or a log on a volume, with epoch marker files, but a filesystem has no transaction: a stale writer that passed the check just before the floor was raised can still land one write, and it learns of the refusal on the check after it. Prefer SQLite.
• The stream. Live output and details are not forwarded once the fence reports a newer epoch (the check is throttled to 200 ms, so a chunk or two can still go out).

A refusal is a StaleEpochError, never a tool result: it is thrown up through the wrapped tool, durableSpaces({ onStale }) is called, and exec no longer maps it to an aborted result. Pi itself records a throwing tool as an error result, which is why the log must be fenced too: that write is refused and the stale driver can never add to the conversation. A driver stops when it sees a StaleEpochError or the onStale callback.

The driver entry#

runDurableSession is the process a coordinator (re)launches for each attempt of a durable session (cua agent run --durable). It reads this environment:

It opens the fenced log in the state directory, raises the Space's epoch, and calls your start to build the Harness (models and providers stay yours), then waits for the submission to settle. Exit codes: 0 finished, 78 permanent failure (a PermanentFailure, a bad environment, or a submission that ended unanswered), 75 fenced (a newer epoch exists; ignore this process), anything else crashed and is resumed with backoff.

import { durableSpaces, PermanentFailure, runDurableSession } from "@trycua/pi-durable"
 
const durable = durableSpaces({ spaces, snapshots, cwd: "/home/cua/work" /* epoch from CUA_SESSION_EPOCH */ })
process.exit(
  await runDurableSession({
    durable,
    start: async ({ session, log, fence, onStale }) => {
      // durableSpaces({ fence, onStale }) is rebuilt here if you need the fence on the extension
      const harness = await Harness.open(log.storage, { models, registry, env: durable.env }, context)
      harness.resume()
      const root = await harness.root(context, { agent })
      // A fixed request id: resuming (or a retry of the whole job) never admits the input twice.
      const submission = await root.submit({ type: "input", content: input, requestId: session.name }, context)
      return { wait: () => submission.wait(context), close: () => harness.close(context) }
    },
  }),
)

Pass onHalt as well: when the Space or the SDK cannot raise the epoch (for example a @trycua/cua whose JSON method table lacks ProcessService/AdvanceEpoch) the adapter throws a HaltError (permanent when the capability is missing, so the driver exits 78; otherwise exit 1 to retry), and the driver halts the log's fence so Pi cannot record that failure as a tool result. The epoch operation is also probed when a Space is first opened, so this fails before any call. Return ensureSpace: () => durable.ensureSpace(harness, root.id, context) from start: the driver calls it after the run settles, so a Space lost after the last call is rebuilt (from the newest checkpoint, or from the log's image in transactional mode) before the session is reported done.

Pass fence and onStale to durableSpaces so the call-level checks and the stream fence are active; a stale commit by Pi's own scheduler is caught by the fence as well, and runDurableSession returns 75 as soon as the log's fence is refused, even if the run never settles.

How it works#

• Commands. SpacesExecutionEnv runs each command as a cua-spacesd process with an idempotency key, <conversation>:<call>:<n>@<epoch>, and the Space's fencing epoch. cua-spacesd writes the key to its exec ledger and syncs it before the process starts. Starting a known key again does not run the command: it reattaches to the running process, or replays the recorded exit status and the last 64 KiB of output.

• Files. File operations go to Pi's own pi-env daemon, which runs in the Space and is reached over a byte pipe through cua-spacesd. File behavior is Pi's; the adapter passes Pi's ExecutionEnv conformance suite.

• Checkpoints. Wrapped calls run one at a time per Space. After a call's effects, and before Pi commits its result entry, the adapter records the outcome, checkpoints the Space, and records {callId, checkpointId} (and, once committed, the result entryId) in the conversation's cua.space document. The state before any call is therefore the newest recorded checkpoint.

• Recovery. When Pi resumes an interrupted call, the wrapper decides:

  What is known	What happens
  The outcome was recorded	It is adopted; the checkpoint is taken if the crash prevented it.
  The command is running	The call reattaches to it.
  The command exited	The recorded outcome is replayed; the command does not run again.
  The command never started	It runs.
  Unknown: the command was lost with the Space's daemon, or a file tool was cut off mid-write	The Space is restored to the checkpoint before the call under a new epoch. write and edit (and any tool declared replay: "safe") then run again; other tools end with an interrupted result that says the sandbox was restored, so the model can decide.

• Fencing. A restore first raises the epoch. cua-spacesd kills every process started with an older epoch, including a stale worker's commands and file daemon, and refuses new requests that carry it.

• Forks. A conversation forked at an entry gets its own Space, forked from the checkpoint its parent had recorded at that entry. The cua.space document is rewindable and copied asOf the fork entry, so the fork sees exactly that checkpoint.

• Lost Spaces. Machine checkpoints live with the cua daemon; with WorkspaceSnapshots, a CheckpointStore keeps a copy of every checkpoint outside the Space. When a conversation's Space no longer exists (its machine is gone, or the Space provider now points at another machine), the next tool call recreates it and restores the newest recorded checkpoint. A call whose effects were recorded only in the lost Space is treated as unknown: write and edit run again, other tools end interrupted.

Review what an agent changed#

The log and the checkpoints together are a reviewable history. Every mutating call has a checkpoint of the workspace taken right after it, recorded in cua.space with the call and its result entry. A person can compare the workspace between two entries, inspect the state the agent saw before a decision, or fork the conversation at an entry and continue by hand in a Space built from exactly that state.

Guarantees, crash campaign and cost#

What each mode guarantees, the crash campaign that measures it, what it costs and its limits are on Durable agent guarantees.