Agents Don't Fail Loudly
Here is the thing nobody tells you when you start building with agents: they don't crash. They apologize. They hand you a confident, well-formatted answer that happens to be wrong, mark the run green, and move on. A traditional system fails with a stack trace and a line number. An agent fails politely, and you find out three weeks later from the bill, the customer, or a test you should have written.
Every section below is a failure I shipped. Each one cost me something: tokens, sleep, or trust. The patterns and principles at the end of each story aren't theory. They're the scar tissue. 😅
Most of this is harness-agnostic. Where I point at something concrete I lean on SWEny, a YAML-defined workflow engine I maintain, and on Claude Code skills. The lessons travel.
The Polite Failure
A screenshot step in one of my pipelines started returning a 1x1 transparent PNG. The run stayed green for a week. The agent had decided that when the browser failed to capture, a placeholder image was a reasonable thing to substitute, so the "evidence" attached to every report was a blank pixel. Nobody noticed because nothing errored. The system reported success while lying.
This is the worst failure mode in the whole field, and it's everywhere. A primary path quietly degrades to a fallback: a fake screenshot, a serialized loop where you asked for parallelism, a default value where the real fetch failed. The run "succeeds." The output is plausible. It is also fiction.
The fix is a posture, not a patch. Forbid silent substitution. If a step can't do the real thing, it must stop and shout, not paper over the gap with something that looks fine.
A fallback that produces plausible output is worse than a crash. A crash you notice. Forbid substitution and emit a loud BLOCKER instead: capture failed, here is why, here is what to fix. Design every step so its failure is detectable, because the model will confabulate before it will admit defeat.
The Cache That Vanished
I once spent a long afternoon staring at a token bill that had quietly tripled. No new features, no traffic spike. The culprit was one line: I had prepended a run_id and a timestamp to the top of a shared context block "for observability." That single volatile string sat above everything else, so every spawn saw a different prefix, and my prompt cache hit rate fell to zero. The cache didn't error. It just stopped existing, and the only signal was the invoice.
Prompt caching pays off hard when you get it right. On Anthropic's pricing a cache write costs 1.25x a normal token (2x for the 1-hour TTL), but a cache read costs 0.1x. Spawn one agent to write the cache, then fan out and have the rest read it at a tenth of the price. The catch: the cached prefix has to be byte-identical every time. Invalidation flows tools then system then messages, so anything volatile has to live last.
// Byte-identical for every spawn. No timestamps/UUIDs above this line.
const shared = [
metaBlock, // sorted keys only, no run_id
fullDiff,
repoRules,
fileManifest, // [{ path, locAdded, locRemoved, role }]
].join("\n\n");
// Per-spawn block appended AFTER the cached prefix.
const prompt = shared + "\n\n" + personaBlock(persona, worktreePath);
// Spawn 1 writes the cache (1.25x). Spawns 2..N read it (0.1x).The principle is one sentence: byte-identical prefix, volatile content last. The timestamp goes at the bottom or in metadata, never at the top. Watch the hit rate like you watch error rate, because a busted cache is invisible until payday.
Death by a Thousand 99%s
I had a workflow where every individual step passed its unit check at 99%. I was proud of it. Then I measured the whole chain end-to-end and it completed the actual task about 60% of the time. I had been grading each link in the chain and ignoring the chain. Error doesn't add. It multiplies.
end_to_end = r1 * r2 * ... * rn
10 steps @ 0.99 -> 0.904 (fails ~1 in 10)
50 steps @ 0.99 -> 0.605 (fails ~4 in 10)Ten steps at 99% each is 90% end-to-end. Fifty steps is 60%. Per-step reliability is a comforting number that has almost nothing to do with whether the user got what they asked for. It's the difference between "the engine fired on every cylinder" and "the car reached the destination."
So measure end-to-end task completion, not per-call success. Build the cheap deterministic eval layer first (assertions on the final artifact), then add a human-plus-model layer on top. Without that, every fix is whack-a-mole: you patch one step, another regresses, and your green dashboard keeps lying to you.
The Lane That Poisoned Seven Others
I fanned out eight agents to work in parallel. One of them hit a rate limit, retried against the same seed data the others were mutating, and corrupted shared state. Within a minute, seven healthy lanes were producing garbage built on top of the one flaky lane's mess. One nondeterministic hiccup, eight ruined results. The parallelism that was supposed to make me faster made the blast radius bigger.
The honest framing of the multi-agent debate helps here. Anthropic's research system beat a single agent by 90.2% on breadth-first research, while burning roughly 15x the tokens, and they were explicit that it's for research, not interdependent coding. Cognition's "Don't Build Multi-Agents" argues the opposite for build work: share context and full traces, because parallel agents make conflicting implicit decisions. They aren't really fighting. They agree on the mechanism (shared context is the constraint) and differ on the domain.
So name the axis instead of picking a tribe: parallelize independent reads, single-thread interdependent writes. Eight agents reading eight separate files, great. Eight agents writing the same state, you just built a race condition with a language model in the loop. When work truly must run concurrently, isolate it: give each lane its own worktree and its own task file, and let the filesystem be the lock.
The Judge Who Loved Long Answers
I wired up an LLM-as-judge to score outputs and gate the pipeline. It felt rigorous. Then I checked its picks against mine and noticed it preferred the longer answer almost every time, and the answer listed first when I showed it two. It wasn't evaluating quality. It was rewarding verbosity and position. My "objective" gate had a thumb on the scale, and I'd been trusting it.
Judges have measurable biases: position (favor the first answer), verbosity (favor the longer one), and self-enhancement (favor their own style). A judge is calibrated tooling, not a neutral oracle. Before you trust one, align it to a human: label a set yourself, then tune the rubric until agreement is high. Hamel reports getting past 90% agreement in about three iterations. Eugene Yan catalogs the biases and the mitigations. The cheapest one is to randomize answer order so position bias cancels out.
And mind the rubric itself: too loose and you get inflated scores plus a token bill for nonsense; too tight and you sign up for endless maintenance. A judge earns gray-area decisions a deterministic check can't express. It does not earn your blind trust on day one.
What I Actually Believe Now
After enough quiet failures, a few principles harden into reflexes:
- Forbid silent substitution. A plausible fallback is a lie with good formatting. Make failure loud and detectable.
- Determinism lives in the scaffolding. You can't make the model deterministic. You make the system deterministic around it: phases, evals, retries, locks.
- Measure end-to-end, not per-step. The only number that matters is whether the user got what they asked for.
- Reads fan out, writes single-thread. Concurrency is a tool, not a default. Name the axis.
- A judge is tooling, not an oracle. Calibrate to a human, randomize order, then trust it a little.
- Watch the bill. Cache misses, retry storms, and verbose judges are all invisible until the invoice. Byte-identical prefix, volatile content last.
None of this makes the model reliable. It makes the system honest about when the model wasn't. That's the whole game: not preventing failure, but refusing to let it happen quietly.
Resources
Anthropic's workflows-vs-agents guide. Autonomy is a cost, not a feature.
The 90.2% win, the 15x token cost, and when fan-out actually helps.
The counterpoint: share context, single-thread interdependent work.
Exact multipliers (1.25x write, 0.1x read), TTLs, and the invalidation hierarchy.
Why the absence of evals turns bug-fixing into whack-a-mole.
Aligning a judge to human labels before you trust its verdicts.
Position, verbosity, and self-enhancement bias, with mitigations.
All 18 frontier models tested degrade as input length grows.