[MS] How to test agent skills without hitting real APIs - devamazonaws.blogspot.com
You shipped a skill that calls an API. The agent uses it, user get results. But how do you know the results are good ? How do you know your last change didn't quietly break the happy path, or that switching models won't regress the scenarios you already got right? You need to evaluate. Run your skill through a set of scenarios, score the outputs, compare across iterations. Without that, you're shipping blind and hoping for the best. The problem nobody warns you about The moment you start evaluating a skill that calls an API, things get complicated. If that API belongs to an external service, every eval run costs money. Say you're running 50 scenarios across 3 models with 5 repetitions per scenario. That's at least 750 API calls per session. Multiply by every iteration as you tune prompts and try different models. The meter is running, and you're paying for test traffic that produces no user value. But even when the API is yours and costs nothing to c...