The run died at 2am: recovery as a first-class workflow

The long runs never die at a convenient time. They die at 2am, three hours into a task, right after the agent did the interesting part and right before it wrote anything down. You come back to a branch that is half-done, a transcript that scrolled past the useful part, and a decision to make with no good options: read three hours of logs to figure out where it got to, or throw it away and start over.

That decision is the tax nobody budgets for. And the reason it hurts every time is that recovery is treated as an accident. Something went wrong, so now a human does detective work. But long-running agent work does not fail occasionally. It fails routinely, because the failure surface is enormous: rate limits, a flaky test, a context window that filled and compacted away the plan, a tool that returned something unexpected, a step that half-applied before it stopped. If failure is routine, recovery cannot be heroism. It has to be a workflow you designed on purpose.

Why long runs fail in the middle, not at the start

A demo runs for two minutes and either works or does not. A real change runs for hours across what I think of as episodes: establish the first behavior, keep it alive while adding the second, survive a checkpoint and resume, handle the late event, cover the negative path, integrate at the end without breaking what the first episode established.

Failure clusters in the middle of that chain, and it is usually not dramatic. The agent does not crash. It drifts. The context fills up, compaction throws away the constraint from episode one, and the agent keeps going, confidently, having silently forgotten the thing that made episode one correct. I wrote about that specific mechanism in compaction eats your decisions, and it is the single most common way a long run ends up wrong instead of stopped.

That distinction matters. A run that stops is easy: you know where you are. A run that keeps going after it lost the plot is the expensive one, because now you have output that looks finished and is subtly broken, and you have to reconstruct what it forgot to even see the problem.

Recovery is not retry

The naive fix is a retry loop: it failed, run it again. Sometimes that works, for a rate limit or a flaky test. But re-running from the top is not recovery, it is amnesia with extra tokens. It throws away the three hours of good work to escape the last five minutes of bad, and if the failure was a drift rather than a crash, the retry often drifts the same way.

Real recovery needs three things a retry does not have.

A last-good state to return to. Not the start, the last point where the work was known-correct. If your only checkpoint is the initial commit, every failure costs you the whole run.

A record of what was already true. The constraints, the decisions, the gates that were green before the failure. Without that, a resumed run cannot tell whether it is continuing or regressing.

A way to score the resumed work against the original contract. This is the part that separates recovery that works from recovery that feels like progress and quietly re-breaks episode one. A resumed run has to prove it finished the way any run does, which is the whole discipline of making done mean done: completion demonstrated against the criteria, not self-reported.

I put real numbers on that last point in the recovery experiment: a coding agent driven by a strong prompt, and the same agent with a hand-written recovery skill, both plateaued at the same two-thirds pass rate across four cumulative episodes, for the same reason. They fixed the new requirement and broke an older invariant, and nothing told them they had regressed. The thing that reached the finish was not a smarter model. It was something outside the agent holding the contract and re-scoring the work after each attempt. Recovery is that outside thing, made routine.

Designing for recovery instead of praying against failure

If you accept that runs fail in the middle, you design the run so the middle is recoverable. A few concrete moves, all of which you can do today without any special tooling:

• Checkpoint at episode boundaries, not just at the end. After each coherent chunk of work passes its check, commit it with the state that made it pass. A branch with six labeled checkpoints is recoverable. A branch with one giant uncommitted diff is a coin flip.
• Write the plan somewhere the run does not own. If the only copy of “what we are building and why” lives in the agent’s context window, it dies when the context compacts. Keep it in a file, a task, an artifact, anything that outlives the session. This is the same reason context loss between sessions is a productivity killer, and the same fix.
• Capture what was green before you continue. Before resuming, the resumed run needs to know which gates were already passing, so it can detect a regression instead of celebrating a half-fix.
• Make the failure loud. A run that stops with a clear “I failed here, this was the last good state” is worth ten runs that drift silently to a confident wrong answer. Half of recovery is just noticing early.

The through-line: the state that lets you recover has to live outside the run. The moment recovery depends on reading the transcript, you are back to detective work at 2am.

The handoff case

Recovery and handoff are the same problem wearing different clothes. When a run dies and you resume it tomorrow, you are handing off to a future version of the run, and everything that makes a clean handoff between agents work is what makes recovery work: the contract, the last-good state, the decisions so far, and what still needs proving all have to travel. If nothing traveled, resuming a dead run and onboarding a fresh agent are equally painful, because both are starting from a transcript nobody wants to read.

Where PaellaDoc fits

Recovery stops being heroism when the state a run needs to resume lives outside the run and something re-checks the resumed work against the original contract. That is the mechanism, not a promise of numbers. PaellaDoc treats a run as something with checkpoints, a contract it is held against, and a governed retry that resumes from the last good state and re-scores against the gates that were green before, so a failure at 2am is a resumable event and not a morning of reading logs. The person doing that reconciliation by hand today is the runtime, and a green build at the end still only counts when it passes the criteria that existed before the run started, which is the whole point of a green build is not a correct feature.

FAQ

How do I recover a failed AI agent run without starting over?

Resume from the last checkpoint where the work was known-correct, not from the beginning, and give the resumed run the record of what was already green so it can tell continuation from regression. This only works if you checkpointed at episode boundaries and kept the plan and constraints outside the agent’s context. If the only artifact is a long transcript, you are reconstructing, not recovering.

Why does simply re-running the agent not fix it?

Re-running works for transient failures like a rate limit, but a lot of long-run failures are drift, not crashes: the agent lost a constraint and kept going. Re-running from the top often loses the same constraint the same way, and it throws away all the correct work to escape the incorrect tail. Recovery needs a last-good state and a way to score the resumed work against the original contract.

What state do I need to capture to make runs recoverable?

Three things: a last-good checkpoint at each episode boundary, the plan and constraints stored outside the run so compaction cannot erase them, and a record of which gates were passing before the failure. With those, a resumed run can continue from a known point and detect if it just broke something that used to work.