October 12, 2026PracticeAgents · Testing · Governance

Testing an Agent Is Not Proving: It Is Sampling

When an agent produces non-deterministic output, a green test suite stops being a gate that proves the system is correct and becomes a sample with a confidence interval: "green" means "I did not catch a defect on this run," not "it is correct." This essay carries acceptance sampling from manufacturing over to agent verification and argues that the architect lever moves from inspecting the output to designing the process so the defect becomes impossible.

You run the test suite against the service your agent generated, everything comes back green, and you ship it. You did this hundreds of times with hand-written code and the rule never failed: if the tests pass, the behavior you tested is correct. The difference is that that code produced the same output every time you ran it. The agent does not. Give it the same input twice and it hands you back two different things, both plausible, one of them carrying a bug that today's test never touched.

And that is where the rule breaks without warning. With non-deterministic output, a passing test no longer proves the system is correct. It proves something far more modest: that on this run you did not catch a defect. Green stopped being a gate and became a sample.

Why did a test over a deterministic system actually prove anything?

Because you ran it once and that once stood in for all of them. An if that returns true for an input will always return it for that input; there is nothing to sample, there is a single reality and you observed all of it. The test was a gate in the literal sense: you either pass or you do not, and if you pass, it is proven for that case.

Non-determinism breaks exactly that assumption. The agent's output for a fixed input is not a value, it is a distribution: most of the time it is right, sometimes it drifts. Observing one run is drawing a marble from an urn you never empty. The next marble can be a different color, and today's test knows nothing about the marbles it did not draw.

Green is acceptance sampling, not a certificate

Manufacturing has been living with exactly this problem for a century and gave it a name: acceptance sampling. When inspecting one hundred percent of a lot is too expensive, destroys the product, or takes too long, you do not inspect everything: you take a sample, count the defects, and decide whether to accept the whole lot. The statistics behind it (the sampling plans, the acceptable quality limit, the operating characteristic curve) were born for World War II ammunition lots and are still alive in the ISO standards.

The first thing that discipline tells you is uncomfortable: sampling does not prove the lot is clean. It gives you a probability of accepting it given a defect level, with its associated risk. Accepting a lot never meant "there are no defects"; it meant "the sample did not find enough to reject it." That is exactly the jump between "the tests passed" and "the system is correct," and in software we had been ignoring it because determinism handed us one-hundred-percent inspection for free on every run.

It is worth pausing on why this metaphor holds and is not just a pretty picture. Gregor Hohpe has a test for that in Mighty Metaphor: a metaphor is good for thinking, not just for decorating, when the dynamics match, not when the images resemble each other. Let us apply it. In acceptance sampling, inspecting one hundred percent is unworkable (it destroys the product, it costs too much) and so you sample; in an agent, inspecting one hundred percent is impossible because the space of inputs and outputs is effectively infinite, so you sample too. In both cases the result is not a binary verdict on the lot, it is a probability with a risk. And in both cases the real decision is not "do I pass or reject this lot?" but "what defect rate am I willing to accept and what sample size backs that decision?"

The dynamics match, so the metaphor lets you think past what I wrote: if green is a sample, then a test that runs once has a sample size of one, the worst sampling plan that exists; then it makes sense to talk about an acceptable quality limit for your agent's output, just as a factory declares its AQL; and then "flaky" stops being an insult to the test and becomes information about the distribution you are sampling. I argued none of that and you are already deducing it. That is what separates a model from an ornament.

The napkin math: how much does a green prove?

Let us put numbers on it, with the assumptions in plain sight so you can redo the math with yours. This is not a measurement of a real system; it is arithmetic for thinking.

Suppose the agent's output is defective with probability $p$ on each run, and that the runs are independent and share the same $p$ (two strong assumptions; we come back to them). If you run the test once and it passes, all you know is that you landed in the 1 − p good fraction. The probability that $N$ independent runs catch at least one failure is:

$P(\text{catch} \ge 1) = 1 - (1-p)^N$

Let us drop in a 5% defect, that is $p = 0.05$ . With a single run ( $N = 1$ ) the probability of catching it is exactly that 5%. Put the other way around, and here is what stings: a green with a single run leaves you a 95% probability that you simply did not step on the bad path. It is not that the system is correct; it is that you did not look hard enough.

How many runs do you need to be reasonably sure, say 95%, of catching a defect that shows up 5% of the time? Solve for it:

$1 - (1-p)^N \ge 0.95 \;\Longrightarrow\; (0.95)^N \le 0.05 \;\Longrightarrow\; N \ge \frac{\ln 0.05}{\ln 0.95} \approx 58.4$

That is 59 runs to cross the line. You went from one test to fifty-nine, and all you bought was 95% confidence against a fairly fat defect. If the defect were 1%, the same math asks for about 299 runs. The intuition that "it passed the test" equals "it works" falls apart the moment you put the exponent on it.

A green with a single run is not proof that the system is correct; it is proof that you did not catch it this time.

Two verification models side by side: on the left the deterministic gate, a test that passes and closes; on the right the sample, N runs estimating a defect rate with a confidence band that never fully closes.

The deterministic gate proves; the sample estimates with an interval. They are not the same gesture with more repetitions: they are two different things.

Two identical panels of a green test, the left one labeled "the system is correct" and the right one "I did not catch the defect this time."

I know what you are about to say

Then I run the suite a thousand times in the pipeline and I am done, a big sample and I sleep easy.

If only it were that simple, and the sampling statistics themselves explain why it is not. The math above leans on two assumptions that are fragile in an agent. The first is that the runs are independent: if you fix the seed, lower the temperature, or cache, a thousand runs look more like one run repeated a thousand times, and your effective sample size drops back near one. The second is that $p$ is fixed: it is not. You change the system prompt, the provider updates the model underneath you, an input lands from a region of the space you never sampled, and the distribution you worked so hard to sample is already a different one. You just received a new lot from a supplier who changed the production line without telling you, and your sampling plan was for the old lot.

Multiplying the runs helps, but it has diminishing returns and a ceiling: no matter how many you run, you never reach "proven," only "a narrower confidence interval." And every run costs tokens and time. At some point, sampling more is more expensive than attacking the problem from the other side.

And what is the other side?

Acceptance sampling brings its own lesson, and it is the one that really matters. Industry did not keep inspecting lots forever: Toyota realized that inspecting more cheaply is not the same as having fewer defects, and moved the effort from inspecting the output to designing the process so the defect becomes impossible. It is Shigeo Shingo's poka-yoke: the part that only fits in the right position, the connector that will not go in upside down. You do not detect the error afterward; you make it unable to happen.

Translated to an agent: stop asking "did the output pass the test?" and start asking "what constraint makes this class of output not even possible?" If the agent should never have deleted a production resource, sampling that catches it 5% of the time is the wrong answer; the answer is that it should not have the permission to do it. That is where a platform like Amazon Bedrock AgentCore stops being infrastructure and becomes the place where the poka-yoke lives: identity, permissions, and guardrails that bound what the agent can touch, so the whole class of defect falls outside the output space and there is nothing to sample. And where the spec-driven flow of Kiro turns the spec into the explicit acceptance criterion the generated output is checked against, instead of letting "whatever makes the test pass" be the implicit definition of correct.

This does not replace sampling; it reserves it for what cannot be made impossible. Whatever can be bounded with a constraint gets bounded, and leaves the lot you have to inspect. It is the same lever shift I argued in Code Is Inventory, Not an Asset: when producing becomes cheap, the architect's control moves from reviewing every piece to designing the constraints the factory reads before producing it.

My suggestion

On Monday, before you treat a green as a certificate again, decide two numbers and write them into the service's ADR: what defect rate you are willing to accept in the agent's output, and how many runs back that "green." If the number of runs is one, you already know your sampling plan is the worst one there is and that you are calling a sample size of one "proven."

And for the half you do not want to sample, ask the poka-yoke question about each class of defect that scares you: do I catch this with more runs, or do I make it impossible with a constraint? Move to guardrails and specs everything that can be made impossible, and reserve sampling for what truly is a distribution to be estimated. If at the end of the exercise your suite still runs once and there is not a single constraint that makes the worst defect impossible, you do not have a quality gate: you have a marble from an urn you never emptied, and you are calling it proof. ;)