[MS] Knowledge cutoff is a poor proxy for model capability - devamazonaws.blogspot.com

A model's knowledge cutoff marks the latest date its training data could cover. But its actual knowledge of a product might end years earlier or be absent entirely. I learned this while trying to answer what seemed like a straightforward question: how much does a model know about a product? If the model's knowledge ended around a particular date, I should have been able to translate that date to a product version.

The model knows the product up to roughly version X. Anything introduced after that needs additional context.

On paper, that would be useful when building agent extensions. You could establish what the model already knows and supply the missing documentation, saving the context window for the work. In practice, the results showed no consistent boundary.

Looking for the line

To test it, I used GPT-5.6 Luna with the GitHub Copilot SDK against two developer products: Dev Proxy and SharePoint Framework (SPFx). I fed the changelogs and release notes for both products to GPT-5.6 Sol and asked it to extract product changes, excluding things like dependency updates and CI infrastructure work. I then asked a model to classify whether each change was suitable for evaluation, based on whether it introduced enough difference from the previous behavior to test.

For each suitable change, a model generated a task prompt and an evaluation rubric. I ran the tasks in Vally with GPT-5.6 Luna as the model under test. GPT-5.6 Terra judged each result against its rubric.

Measuring internal model knowledge requires a strict information boundary. As I covered in Your AI coding agent evaluation is only as good as its sandbox, even a local product installation can give the agent access to the answer and invalidate the measurement. For this experiment, I removed access to external information, including documentation and web search.

OpenAI lists GPT-5.6 Luna's knowledge cutoff as February 16, 2026. Dev Proxy 2.1.0, released on January 28, was the latest stable release available by then. Yet when I asked the model where its product knowledge ended, it placed the boundary around Dev Proxy 0.29.0 and SPFx 1.21.1. Dev Proxy 0.29.0 had shipped on June 25, 2025, almost eight months before the model's official cutoff. This was already evidence that a date cutoff and the model's knowledge of a specific product are different boundaries.

If the model's product estimates gave us a useful boundary, the results should still have had a recognizable shape. Older releases would perform relatively well and performance would deteriorate as the tasks approached those versions, then newer releases would consistently fail.

Instead, I got noise. Across 336 Dev Proxy tasks spanning 53 versions, the model passed 61, or 18%. Dev Proxy 0.3.0 passed four of five tested tasks, while 0.4.0 passed none. Much later, versions 2.3.4, 3.0.0 and 3.1.0 each passed one of two, followed again by releases with no passing tasks.

Meanwhile, SPFx looked remarkably similar. Across 413 tasks covering 40 versions, the model passed 61, or 15%. Correct answers and failures appeared throughout the product history, with no clean transition at any version.

A model can fail before its cutoff

A knowledge cutoff sets a ceiling. It tells us the latest knowledge the model could have encountered during training, without telling us how well any particular product is represented. Its knowledge of that product might stop two years earlier or be absent entirely. Even when the knowledge is present, the model might struggle to recall and apply it when needed.

The SPFx results showed this clearly. One task for SPFx 1.11.0 asked about using a third-party module loader in the local workbench. The model produced plausible advice about SPFx's module-loading model, AMD, webpack and iframe isolation. But it missed the fundamental fact: SystemJS support had been removed from the local workbench in that release. The model knew enough about the surrounding technology to construct a convincing answer while missing or failing to recover the one historical fact needed to solve the task.

Dev Proxy showed the same pattern. For a Dev Proxy 0.5.0 task, I asked which executable should be invoked. The model confidently answered devproxy. The required answer for that version was however mgdp, when Dev Proxy was still named Microsoft Graph Developer Proxy.

So product knowledge varies by task. A model can know a product's architecture and common patterns while missing one particular fact. Features that predate the cutoff can still sit beyond the model's capability.

A model can succeed beyond the expected boundary

At the same time, the model solved tasks for releases far beyond where I expected its product knowledge to be useful. At first, those passes looked like a problem with the experiment. If I was looking for the last version the model knew, why did it occasionally answer questions about much newer versions correctly? Sometimes it could infer how a feature worked.

In When the model has never seen your code, I described how models map unfamiliar technology to the closest patterns they already know. That behavior can produce plausible mistakes, but it can also help. A model might recognize older Dev Proxy concepts in a newer feature, while its general knowledge of proxies and software fills the remaining gap.

Dev Proxy 2.3.4 shipped on April 23, 2026, followed by 3.0.0 on May 28 and 3.1.0 on July 1. All three releases came after GPT-5.6 Luna's February 16 knowledge cutoff, yet the model passed one of the two tasks I tested from each release.

Those passes could not come from direct exposure to the released versions in training data bounded by the communicated cutoff. They do not prove that every underlying idea first became public on its release date. The model may have inferred the answers from familiar patterns or guessed correctly. Either way, it solved tasks tied to product versions that did not exist at its cutoff.

Capability has several sources

When a model produces a correct answer, we can't reliably determine whether the product fact was represented in its weights. It might have extrapolated from an earlier version or transferred a pattern from another product. General programming knowledge may have been enough, or the model may simply have guessed correctly.

From the outside, each route produces the same observable result: the model solved the task. The knowledge cutoff is therefore a poor proxy for what I was trying to measure. I thought I wanted to know:

What's the latest version of this product the model knows?

But what I actually wanted to know was:

How capable is this model of working with this product without additional information?

The first question asks where the model's knowledge came from. The second asks whether it can do the work, and it's actually more useful when working with an AI agent.

Stop looking for the version

Suppose you're building applications that use Microsoft Graph and want your agent to use Dev Proxy. One option is to determine the newest Dev Proxy version represented in the model's training data, then feed the agent documentation for everything released afterward.

The model may miss functionality from before the cutoff and infer functionality from after it. Supplying every newer feature wastes context on knowledge the model might already be able to apply. Meanwhile, older gaps remain. So instead, start with the work you expect the agent to perform. Create representative tasks from that work, then test the model without additional product knowledge. If you don't have a recurring domain, use the specific problems you need to solve as the evaluation set.

Measure a workload because capability depends on the tasks and context in which you use the model. In What AI benchmarks are not telling you, I explain how to build that workload-based comparison and why public scores cannot replace it. Cutoff dates have the same limitation: they are broad signals that cannot tell you whether the model can do the work that matters to you.

Once you have that baseline, use the controlled comparison to measure the effect of adding documentation or agent extensions. Now you know where additional knowledge helps.

Trust but verify

Still, knowledge cutoffs tell us something useful about a model's training. When a feature was released after the cutoff, the model may never have encountered it directly. The trouble starts when we use that date to predict what the model can do.

There was no clean point in my results where knowledge disappeared. Correct answers sat among failures, and failures among correct answers. The model understood the shape of a technology while missing individual facts. Sometimes it reasoned its way into functionality that a cutoff suggested it shouldn't know.

To assess a model with your technology, give it the work, measure what it can do and supply knowledge where gaps appear. The cutoff gives you a date but it's the eval that tells you whether the model can do the work.

Post Updated on September 21, 2026 at 08:51AM
Thanks for reading
from devamazonaws.blogspot.com