The Question Nobody Asks Before Shipping
You built a skill that calls an API. The agent uses it, users get results, everyone moves on. But how do you actually know those results stay correct? How do you know your last change didn't quietly break the happy path, or that switching models won't regress everything you already got right?
The honest answer for most teams is that they don't know, because they never built a way to measure it against real production traffic without paying a real cost for finding out.
The Hidden Cost of Evaluating Against Reality
If the API being tested belongs to an external service, every evaluation run costs money. Fifty scenarios across three models with five repetitions each adds up to 750 calls in a single session, multiplied across every iteration as prompts and models get tuned. Even when the API is free to call, mutating writes, PATCH calls and DELETE calls, change live state as a side effect of measurement. Reset it manually between runs, or accept that later results are contaminated by earlier ones.
There's a subtler problem too. If other systems or people write to that same API, evaluation results stop being deterministic. A score drops between two runs not because anything changed on your end, but because someone else's team updated a record your test scenarios happened to depend on.
Faced with all that, most skill builders simply skip rigorous evaluation. They test manually, ship, and hope. The result is skills that work fine on a Tuesday demo and quietly break on a Friday when the underlying API returns slightly different data, with nobody the wiser because nobody measured.
Why a Hand-Built Mock Isn't the Answer Either
The obvious fix is standing up a local mock server. In practice, that means handling multiple endpoints, supporting filtering and pagination, returning realistic payloads, and correctly implementing PATCH and DELETE semantics, an afternoon of work for a simple API and an ongoing maintenance burden for anything larger, one that drifts out of sync the moment the real API changes.
There's a deeper issue as well: pointing a skill at localhost means changing the URLs inside the skill file itself, which means the skill under test is no longer the one that will actually ship. Every token in a model's context shapes its reasoning, and swapping a production URL for a local one introduces exactly the kind of variable you're trying to eliminate from the measurement.
A Lower Bar for "Good Enough"
It turns out the bar for adequate API emulation is lower than most people assume. What's actually needed is stable URLs and realistic payloads with correct HTTP semantics for whatever operations the skill performs, not a full replica of the backend.
Tools like Dev Proxy achieve this through a JSON configuration defining a base URL, endpoints, and seed data, intercepting HTTP traffic from the agent and answering locally with no server to build or maintain. The skill keeps calling its real production URLs the entire time; it has no idea a proxy is answering instead of the actual service. Data resets on every run, so there are no production records harmed, no API costs incurred, and no awkward conversation about who deleted the test orders. Remove the proxy when testing is done, and the skill works against production without a single code change.
Evaluation Isn't Optional Anymore
Nobody would run integration tests against a production database. Running eval scenarios against a live production API is the exact same mistake wearing a different hat. Whether the fix is a maintained mock or a lightweight proxy, the underlying principle doesn't change: pick an approach and use it, because the alternative is shipping skills that have never been systematically measured at all.