The Harness Is the Product

Every team I talk to now runs a frontier model. Claude Sonnet 4.5 on the API, Opus reserved for the calls that need the extra depth. Same model, same weights, same tool-use format. The model is table stakes. It is the amp everyone already owns — and nobody wins a gig by owning the amp.

The product is not the model. The product is the wiring around it.

Setup

Take a 20-step agent pipeline. Read a ticket, pull the repo, plan an edit, write code, run tests, read the diff, open a PR — the ordinary chain of an agent doing real work. Assume each step succeeds 95% of the time. That is a good step. A model that gets it right 19 times in 20 feels reliable when you watch a single call.

Now chain twenty of them and let the errors compound.

Observation

0.95^20 = 0.3585. The pipeline completes end-to-end 36% of the time.

That number is not a vibe. It is exact arithmetic, and it is the whole argument. A chain is only as strong as the product of its links, and probabilities below 1 collapse fast when you multiply them. Drop to 90% per step — still a step that “usually works” — and 0.90^20 = 0.1216. Twelve percent. Four runs in five fail somewhere in the middle, and the failures land at random, so you cannot even reproduce them cleanly.

Push the other way. 0.99^20 = 0.8179. Move each step from 95% to 99% and end-to-end reliability jumps from 36% to 82%. Same model. Same task. The only thing that changed is the four points of per-step reliability — and every one of those points comes from the harness, not the weights.

That gap, 36 to 82, is where the product lives.

Mechanism

The harness is the set of things that raise per-step reliability and contain the failures the model still produces:

• Verification loops. A second pass that checks the first. A critic call reads the output against the spec before the pipeline accepts it. Cheap. It converts a silent wrong answer into a caught one.
• Bounded retries with backoff. A step fails, you retry it — not forever, three times, with growing delay. Transient failures die here. Half of that lost 64% is transient, and a retry policy reclaims it.
• Checkpoint-resume. State persists after every step. A failure at step 18 resumes from step 18. Without it, one bad call at the end throws away seventeen good ones and you pay the full token bill again.
• Structured outputs and schema validation at the hand-offs. The boundary between steps is where agents rot. Validate the shape before the next step reads it. A malformed hand-off caught at the seam never becomes a wrong answer three steps downstream.
• Human gates. Not a compliance checkbox. A deliberate stop before an irreversible action, placed where a wrong call is expensive to undo.

None of that is model work. All of it is engineering, and all of it is where the reliability points come from.

Building Gizmo — my own agent harness — taught me the ordering. The first version chained model calls and trusted them. It was quick to demo and miserable to depend on. The failures were never in a single call; a single call almost always looked fine in isolation. They were in the joints between calls, and in the absence of a way to resume, so every failure cost the whole run. The work that made Gizmo dependable was not a better prompt. It was verification at the seams, retries with limits, and state that survives a crash. I spent far more lines on the harness than on anything the model touches, and that ratio is the point.

Take

Swapping Sonnet 4.5 for the next model buys you a few points of raw per-step reliability. Real. Not nothing. But the model gets better for everyone at once — your competitor got the same bump the same morning. The harness is the part you own.

Stop shopping for models. Everyone has the amp. Go build the thing that turns 36% into 82% and contains what still breaks. The model is the instrument. The harness is the record.