Bounded failure design for long-running agent workflows

The searches that lead here are usually typed mid-incident: claude code stuck while a session sits on the same plan for the third quarter hour, cursor stuck on planning next moves while the status line promises progress that never arrives, or the quieter variant — an agent loop left running overnight whose morning report nobody trusts. The fear behind those searches is legitimate but misdirected. Long-running agent workflows do not fail expensively because models are unreliable; they fail expensively because nobody decided, before launch, what a failure was allowed to cost. This guide is that decision, written down as a design method you can apply to any harness and reuse with attribution. YYLO's runner contracts are the worked examples, and every claim about them is anchored in the committed documentation.

The method has three properties, and every section below is a decision that produces one of them. A long-running workflow is safe to look away from when its failure is bounded (it cannot spend more compute, time, or repository change than a declared budget), observable (the failure surfaces in evidence a machine can read, not in console silence), and recoverable (a verified prefix survives the crash, and resuming never invents completion that was not earned).

Decide the blast radius before the first run

Bounded failure starts as a spatial decision: before any run, you declare where change may land and how much of it the run may buy. The iteration budget is the simplest form — YYLO caps every loop with -i <n>, and the default cap is one iteration, so a single bounded run is what you get without asking. Unbounded execution exists only as the explicit opt-in -i -1; treat it as a named exception, never as the ambient mode for work you intend to review. A budget turns "when do we stop?" from a judgment call made mid-incident into a number chosen while consequences were still cheap.

Budgets bound spend in time and money; boundaries bound it in space. YYLO's typed managed-agent steps make the spatial contract explicit: a stage_boundary declaration fixes the one worktree root, written as an absolute path, and enumerates exactly which paths the step may touch; the runtime hashes tracked and untracked content on both sides of the dispatch, and any mutation that lands outside the admitted paths stops the step — its successor never launches, and the runner deliberately performs no cleanup, because "clean it up and continue" is how small escapes become unreviewable ones. The architectural version of this argument belongs to the harness engineering blueprint; the design move to steal is smaller: the workflow declares its write surface up front, and machinery, not prompt prose, enforces the declaration.

Even the provider's own quota becomes a designed boundary rather than a surprise. YYLO's --on-hourly-limit accepts exactly two policies — wait (auto-retry) or raise (exit) — so hitting a rate limit is a decision you made on purpose, not an event that happens to your run.

Pair the numeric stop with a semantic one

A cap on iterations cannot stop a loop that is running in place. Five identical, useless cycles respect an iteration cap perfectly and produce nothing but cost; the failure mode is not overspending the budget but spending it without direction. That is why a bounded design always pairs its numeric stop with a semantic one: a terminal condition expressed in state the runner can read without judgment.

The cleanest semantic stop is "the queue is empty." YYLO's run-until-completion runner is built on exactly that condition: it runs at least once, then keeps going only while the board still holds backlog, todo, or in_progress work — progress is measured in board state, not in the agent's opinion that it is nearly done. "Until the model feels finished" is not a stop condition; "until no open work remains" or "until the gate passes" is. If you are starting from a single task rather than a queue, the bounded loop is the beginner-sized form of the same idea.

# fail-forward policy chosen before launch, not mid-incident
yylo loop --workflow flow.yaml -n 5 --on-error stop

The fail-forward policy is the third stop decision, and it is per workflow, not per mood. YYLO's yylo loop defaults to on_error: continue — a failed step is recorded, the rest of that iteration is skipped, and the next iteration starts, because evidence-gathering outranks tidiness. --on-error stop (or a per-step on_error: stop) launches nothing further after a failure. Inside the ordered Workflow Runner, generic steps likewise record failures without halting unless a step declares fail_workflow: true — reserve that marker for the steps where barreling on is the costlier branch: the gates that validate, the commands that destroy, and anything whose downstream steps would ingest a bad upstream answer and compound it. Whichever policy you choose, yylo loop holds one invariant across both: a failed command makes the final loop exit nonzero even when later iterations ran, carrying the first failure's code forward to the end. The ordered Workflow Runner's generic-step default is deliberately the opposite — a failed generic step lands in the manifest and the process still exits zero — so there a green exit tells you the process finished, not that every step succeeded; the manifest's failed-step list is the truth.

Make staleness machine-detectable

Stuck is the failure mode a human notices last and a machine could notice first. From the outside it looks like no observable progress: the planning step that never converges, the same edit attempted and abandoned in a loop, a console that has been quiet long enough that you stop believing it. The design question is not "how do I watch harder" but "what state does my workflow expose that a runner can diff between iterations to detect that nothing is moving?"

YYLO answers that with the task board. Run-until-completion tracks kanban state across iterations, and when a configurable number of consecutive iterations pass with no task change — three by default, adjustable with --stale-threshold — it executes the ON_STALE hook and exits instead of burning budget pretending to work:

# stale detection owns the no-progress risk this loop would otherwise carry
./.juno_task/scripts/run_until_completion.sh -s claude -i 5 --stale-threshold 3

The check can be disabled with --no-stale-check, and that flag is itself a design smell: if you switch it off, some other mechanism must own the risk it carried, and the design review in the last section asks which one.

Two quieter signals belong in the same design. First, when the detected agent command exits zero yet produces no answer, the contract fails the step — an empty result is something the runner refuses to grade as success, which closes the cheapest mimicry there is: doing nothing, successfully. Second, interruption is a designed stop, not an act of desperation: sending SIGINT or SIGTERM to a yylo loop stops the active child and prevents any later launch, so the operator's Ctrl-C is a first-class boundary the runner honors. The principle unifying all three: if the only way to tell that a run is stuck is a human reading scrollback, the workflow's design is not finished.

Checkpoint semantics that survive the crash

"Recoverable" is a property of checkpoints, and most systems that claim checkpoints do not have them — they have logs that happen to still exist. A checkpoint you can resume from earns the name by satisfying four rules:

Evidence is written before progress is claimed. A step counts as complete only after its stdout, stderr, response, and any declared receipts are durably persisted — never because a process returned.
Identity is bound. The checkpoint hashes the command, run identity, frozen inputs, and receipt bytes it attests, so "the same step" cannot silently mean different work at resume time.
Ambiguity is refused. Evidence that is half-written, borrowed from a different run, missing its middle, or altered since it was written gets refused rather than interpreted; recovery appends an interrupted record of the verified prefix and never infers semantic completion on its own.
History stays immutable. No amount of inconvenience licenses rewriting a finished run so that its evidence becomes reusable — corrections happen in a fresh attempt that revalidates what the old one actually proved.

YYLO's Workflow Runner implements all four in one artifact: a run's first attempt writes run_contract.json — one file serving as both the checkpoint and the attempt index — and a step earns reusability only once its stdout, stderr, response, and declared receipts persist atomically, hash-bound to their command, run, and attempt identity. The payoff arrives on the worst day: when the producer dies mid-run, the hash-bound prefix is still trustworthy, recovery names the first step it cannot vouch for, and resuming means re-verifying the unchanged workflow, rendered commands, frozen inputs, and receipt hashes before anything replays. The command-level recovery walkthrough — dry-run first, then resume at the reported step — is owned by the auditable workflows guide; what belongs here is the design obligation: decide, before launch, what "done enough to resume" means in your workflow, and let machinery verify it instead of a tired operator guessing at a manifest.

Recovery patterns for the stuck and the derailed

Stuck and derailed are different incidents and recover differently. A stuck run has stopped making progress but stayed inside its bounds; a derailed run is making progress in the wrong direction or has escaped them. Five named patterns cover the responses:

Stop-then-inspect (stuck). Interrupt — in a yylo loop, SIGINT/SIGTERM stops the active child and prevents later launches — then read the evidence before relaunching anything: the step responses, the board state, the working diff. A relaunch without a diagnosis is the same failure with fresh budget.
Verified-prefix resume (interrupted). After a crash, recover the verified prefix and take up again at the first step the recovery report marks invalid, letting the runner re-hash the unchanged inputs rather than trusting a directory listing.
State-anchored continuation (stuck conversation). When the thread itself is the asset — the context, the half-formed plan — continue it by session id instead of retyping the story into a fresh agent; the sessions and handoffs guide owns that discipline end to end.
Roll back the change, never the record (derailed). The work layer recovers through Git — revert, reset, remove the worktree; the evidence layer is append-only. A failed attempt is data about your system, not garbage to tidy away before anyone sees it.
Fail to the boundary (escaped bounds). When mutation lands outside the declared write surface, stop before the successor runs and record the escape. "Continue now, clean up later" is how a bounded incident becomes an unbounded one.

These are design patterns, not a taxonomy of everything that can go wrong — the point of naming them is that each one is a decision you can make in advance, in the workflow file, while the cost of deciding is zero.

The bounded-failure design review

Run this review against any long-running agent workflow before it earns unattended time. Each line is a yes/no question; every "no" is a specific, fixable gap rather than a vague worry:

Budget — does every loop carry a numeric bound, and is any unbounded mode a deliberate, named exception?
Stop — is there a semantic terminal condition expressed in runner-readable state, plus an explicit fail-forward policy per step?
Staleness — can the runner itself detect a no-progress iteration and exit on it?
Silence — when an agent exits zero and returns nothing, does the runner fail the step?
Checkpoint — is progress claimed only after hash-bound evidence is written, with ambiguity refused rather than interpreted?
Blast radius — is the write surface declared before launch and enforced by machinery?
Recovery — does resume land on the one step the recovery report marks invalid, re-verify the prefix, and never invent completion?
Interruption — does an operator signal stop the active child and prevent later launches?
Record — after any failure, is the past immutable, with corrections recorded as new attempts?

That checklist, the four checkpoint rules, and the five recovery patterns are the reusable core of this method. If they make your next long-running run cheaper to fail, cite this page when you reuse them — attribution keeps the method's owner findable while the ideas travel.