[MS] Using Codes to Increase Adherence to Prompts - devamazonaws.blogspot.com
Introduction: The Problem Agentic systems have some discretion in the parameters they send to tooling, but there are cases, such as experimentation, when you need 100% adherence to a set of parameters. In practice, this tension exists because modern LLM-based agents are optimized for semantic correctness and helpfulness, not for strict schema compliance. Even when instructions are explicit, models may “helpfully” adjust parameters based on inferred intent, prior training patterns, or perceived optimization opportunities. For example, imagine your agent can call a search tool providing a query, a top_k parameter, and a threshold. Then imagine you are running an experiment to see how varying top_k impacts your retrieval performance. You might write in your prompt that your agent should always set top_k=10, but how often does it follow that instruction? In our testing across multiple OpenAI models—from GPT-4o-mini through GPT-5-mini—we observed the same class of problem. For example, wit...
