AI coding agents can build SharePoint Framework solutions. But should
you trust them to choose the right toolchain and package versions by
themselves, then apply every upgrade step correctly?
We wanted evidence. Before publishing
SPFx Dev
Skills, the SharePoint Framework team and Developer Relations worked
together to understand what agents already know and where they struggle.
We then measured which guidance improves the outcome.
It’s easy to package our expertise in a skill and tell developers to
trust it. Proving where the skill adds value takes more work, but
produces a far more useful result.
We started without the skill
The collaboration brought together Vesa Juvonen and Bert Jansen from
the OneDrive and SharePoint SPFx team, and Garry Trinder and Waldek
Mastykarz from Developer Relations. Collectively, we defined what a
correct upgrade should look like and designed the evaluations, then
traced failures through the agents’ complete sequence of actions.
Together, we turned those observations into changes across the surfaces
that agents use.
The evaluations also built on years of work by the
CLI for Microsoft 365 maintainers.
They have encoded deterministic, version-by-version SPFx upgrades in the
spfx project upgrade command. We used that work as an input
to our evaluations and, later, as the upgrade path we wanted agents to
discover.
Our first scenario focused on upgrading an SPFx 1.21.1 project to
1.22.2. This requires more than changing package versions because SPFx
1.22 introduced the move from gulp to Heft, along with related changes
to configuration and the project structure.
We asked GitHub Copilot Chat in Visual Studio Code on Windows, using
Claude Sonnet 4.6 to do just that:
Upgrade the project to 1.22.2
Notice that we purposefully didn’t mention SPFx or point to
documentation, and we ran the scenario without a skill. This gave us the
baseline: what can the agent do with the project context and its
existing knowledge?
Because language models are non-deterministic, we ran the same
scenario 5 times. We checked whether the project still built and ran,
but we also evaluated the details that distinguish a partial upgrade
from a correct one.
| Classification |
Passed |
Total |
| Prerequisite |
5 |
5 |
| Idiomatic use |
7 |
10 |
| Dependency currency |
34 |
50 |
| Configuration correctness |
38 |
85 |
Every run produced a project that passed our execution gates. At
first glance, that looks like success. A closer look showed that the
agent updated the headline SPFx version but missed dependencies and
migration details. Starting with a baseline kept us from writing a large
skill that repeated knowledge the agent already had while missing the
problems that actually affected developers.
Copilot already knew a lot about SPFx
Before invoking a tool, the agent recognized the project as SPFx
simply from its file tree! It knew that Microsoft packages needed
updating and expected changes to the TypeScript toolchain. When it
needed the release notes, it didn’t search for them. It constructed the
full Microsoft Learn URL directly, following the URL pattern established
across years of previous SPFx releases. Its weakness was combining that
knowledge with the information it found.
For example, an agent asked to upgrade to 1.22.2 fetched the release
notes for 1.22. It noticed that the documented version didn’t exactly
match the requested version, but didn’t fetch every intermediate release
page. That’s risky because SPFx upgrades are incremental, so changes in
1.22.0 and 1.22.1 still matter when the target is 1.22.2.
The agent also formed its plan before reading the documentation, then
used the docs to confirm it. A tip suggesting CLI for Microsoft 365
didn’t change its approach because the page also offered detailed manual
migration steps that looked actionable.
So, what should a skill add when the agent already recognizes SPFx
and can find its docs? Precise decision rules for the places where
generic knowledge is unreliable.
We tested the obvious answer
The SPFx team had already created a separate anti-hallucination
skill. It instructed the agent to verify SPFx facts using authoritative
documentation. We compared the baseline with this skill, then added the
context7 MCP server to see whether another documentation source helped
further.
| Classification |
Baseline |
+ Anti-hallucination skill |
+ Anti-hallucination skill and
context7 |
| Prerequisite |
5 / 5 |
5 / 5 |
5 / 5 |
| Idiomatic use |
7 / 10 |
10 / 10 |
8 / 10 |
| Dependency currency |
34 / 50 |
40 / 50 |
39 / 50 |
| Configuration correctness |
38 / 85 |
46 / 85 |
47 / 85 |
The anti-hallucination skill improved the results and reduced average
token use by roughly 9%, but directing the agent to authoritative
documentation still left the project only partially upgraded.
Adding context7 didn’t provide meaningful additional lift. In 3 of 5
runs, its tools were available but never invoked. Availability alone
didn’t make the tools relevant to the agent’s plan.
We had better numbers, but not enough to call the upgrade problem
solved. Treating the anti-hallucination skill as the answer would have
been the agent-development equivalent of saying,
trust us, this
helps, so we kept looking.
The best upgrade guidance already existed
CLI for Microsoft 365 includes
the
spfx project upgrade command. It analyzes the source
and target versions, then generates the version-specific changes
required for an incremental upgrade.
When we explicitly told the agent to use the CLI, dependency currency
jumped from 34/50 to 50/50. Configuration correctness rose from 38/85 to
83/85.
| Classification |
Baseline |
+ CLI for Microsoft 365 |
| Prerequisite |
5 / 5 |
5 / 5 |
| Idiomatic use |
7 / 10 |
4 / 10 |
| Dependency currency |
34 / 50 |
50 / 50 |
| Configuration correctness |
38 / 85 |
83 / 85 |
An existing product tool produced the strongest results because it
already encoded the upgrade logic. The agent didn’t discover it by
itself, so we focused SPFx Dev Skills on guiding the agent to the CLI
when an upgrade required it.
Documentation changed the agent’s plan
The SPFx release notes already included a tip about CLI for Microsoft
365. Even when we added the exact command, agents continued with their
manual plan.
Why? Because a tip offers an alternative without telling the agent
that its current approach is wrong. Meanwhile, the linked migration
guide contained detailed manual steps supported by npm commands and JSON
snippets. Agents consistently chose that actionable content.
We used
Dev Proxy to test
documentation changes without publishing unproven ideas to Microsoft
Learn. Several attempts didn’t work. Moving the tip made no difference,
and removing the migration guide hurt manual runs without reliably
increasing CLI adoption.
Then we added a warning immediately before the tip, directly
challenging the agent’s manual approach:
Upgrading from a previous minor version requires changes across
package versions, build configuration files, and toolchain settings that
vary depending on your source version. Manually updating package.json
alone will result in build failures.
With the detailed migration link removed, CLI adoption moved from 0
of 5 runs to 5 of 5. The warning worked because it challenged the plan
the agent had already formed.
We couldn’t simply remove useful guidance for human readers, though.
When we restored the step-by-step migration guide, CLI adoption fell
back to 0 of 5. So we rewrote the guide to explain what changes during
the gulp-to-Heft migration while directing readers to CLI for Microsoft
365 to apply those changes. With the conceptual guide supporting the
warning, agents used the CLI in 5 of 5 runs, and configuration
correctness reached 85/85.
We submitted these improvements to the SPFx documentation. The
release-note version changes shipped in
PR
#10855, followed by the warning and rewritten migration guidance in
PR
#10921. Putting these fixes at the source made them available to all
developers and agents, including those without SPFx Dev Skills.
The evaluation shaped SPFx Dev Skills
Our evaluation covered the upgrade scenario and compared several ways
of helping the agent, including the separate anti-hallucination skill.
The results showed what SPFx Dev Skills should add and what it should
leave to authoritative documentation or existing tools. The
SPFx Dev Skills
preview packages the SPFx-specific decision rules in 1 focused
spfx skill and routes each request to a self-contained
playbook.
Today, the skill covers these areas:
- creating SPFx web parts, extensions, libraries, and Adaptive Card
Extensions;
- upgrading projects with CLI for Microsoft 365;
- choosing Heft for SPFx 1.22.0 and newer, or gulp for earlier
versions;
- building React interfaces with Fluent UI v9; and
- using PnPjs for SharePoint and Microsoft Graph data access.
The upgrade guidance reflects what we observed during evaluation. It
tells the agent to detect the installed version and check Node and
TypeScript compatibility before acting. The agent then uses CLI for
Microsoft 365 to apply every generated change in order, and it must
finish with a clean build. The global guidance also tells agents to run
long SPFx dependency installations synchronously, avoiding the repeated
terminal polling we saw consume time and tokens during testing.
The skill complements the documentation and CLI by helping the agent
choose the right source and use it correctly.
SPFx Dev Skills is currently available in preview. Its playbooks for
creation and design extend beyond what we evaluated here, as do its
toolchain and data playbooks. We’ll keep validating them through real
usage, and developers should review the output before committing or
shipping.
The collaboration improved more than the skill
The investigation also surfaced an opportunity beyond SPFx. Microsoft
Learn supports returning pages directly as Markdown, giving agents
cleaner content than converting the full HTML page. We shared this
finding with the GitHub Copilot teams, who added
Accept: text/markdown support to GitHub Copilot Chat and
GitHub Copilot CLI.
Agent experience problems can originate outside a skill. In this
case, some fixes belonged in documentation or the CLI, while another sat
in the platform that retrieves content. Establishing the baseline showed
us where to work.
Try the preview and help us improve it
You can explore
SPFx Dev Skills on
GitHub. The repository contains the portable
spfx skill
and its task-specific references, along with manual installation
instructions for compatible AI coding agents. Try the preview with your
SPFx work and review what the agent produces. When the result is wrong
or surprising,
open an
issue. That evidence will help us decide what the skill should teach
next, and what we should fix somewhere else.
Post Updated on September 2, 2026 at 11:14AM
Thanks for reading
from devamazonaws.blogspot.com
Comments
Post a Comment