The testing problem with agents
You've built an eval. You've found the gaps. You have hypotheses about how to fix them: maybe a documentation rewrite, a different CLI output format, or a new prompt template. The natural next step is to ship the change and measure again. Don't do that just yet.
Every improvement you think of is a hypothesis. "If I add a warning callout to the release notes, the agent will use the CLI instead of attempting a manual upgrade." It sounds reasonable. It might even be obvious. But LLMs don't reason the way you expect them to, and obvious changes produce counterintuitive results with uncomfortable regularity.
The case that proved the rule
Microsoft's team hit this in a SharePoint Framework (SPFx) project upgrade scenario. SPFx projects upgrade incrementally, version by version, with each step introducing its own configuration and dependency changes. The agent needed to discover a CLI tool from documentation. The documentation included a tip recommending the CLI. Straightforward, right?
The agent ignored the tip in every single run. The tip was there. The agent read the page. It did nothing differently. The intuition was that a recommendation in the docs would influence the agent's behavior. The reality was that the agent had already constructed a plan from its training data before it fetched the page. The documentation confirmed parts of that plan (package names, version numbers), and the tip got filed away as optional side information. Advisory content does not override an existing plan. The agent treats "you could also try X" as noise when it already has a strategy it believes will work.
What worked, and what didn't
The team tested a dozen variations. They moved the tip to a different position on the page. They reworded it. They included the exact command to run. Nothing made a difference. The most useful mental model for understanding why: the agent processes an entire page and gravitates toward actionable step-by-step content. A tip, by definition, is not a directive.
What finally worked was a warning that told the agent its current plan would fail. Not "here's an alternative" but "your approach will result in build failures." Five out of five runs flipped to using the CLI. The agent abandoned its plan because the documentation told it the plan was broken, not because a better alternative existed.
Nobody on the team predicted this. They assumed stronger wording in the tip would suffice. They were wrong, and they would have shipped a useless documentation change to production if they hadn't tested it first.
The hit rate on other hypotheses
The team ran over a dozen more experiments in the same engagement, and the hit rate was not encouraging. Removing links to a competing migration guide helped CLI adoption but destroyed overall quality. Dependency accuracy dropped from 34/50 to 18/50 because the four out of five runs that still didn't find the CLI no longer had the guide to work from. Adding structured JSON output to the CLI improved configuration correctness from 72/85 to 85/85, but switching to JSONL (which seems superficially similar) cost 2.6x more tokens for a tiny gain in latency.
The pattern is consistent: agent changes look reasonable, ship them, and watch the metric you cared about move the wrong way. The only defense is a test harness that can run the agent against the change in a controlled environment, with the same evaluation you used to find the gap in the first place.
The operational rule
The lesson is to treat every agent change as a hypothesis and every test as the only way to know whether the hypothesis is right. The eval is the design tool, not just the verification tool. If you can't run an experiment that compares the old behavior to the new, you don't have a feedback loop. You have a guess.