I Grepped the Source Code Behind Visual Studio’s Modernization Agents (So you don't have to)

The July 2026 Visual Studio update added something super cool: built-in skills, authored by the .NET and Azure teams, that appear in the Copilot tool picker whenever the matching workload is installed.

Side note: of the nerdy posts I've written over the years, I'd like to propose this one in my top 10.

The interesting part is that they aren't a black box. They ship from two public repositories: dotnet/skills and microsoft/azure-skills as plain Markdown with YAML frontmatter, under an MIT license, alongside the evaluation suites Microsoft uses to grade them.

Which means that for the first time, you can read exactly what the platform vendor believes a coding agent should know about modernizing software, and exactly how they verify it knows it. That is a rare artifact. So I cloned both and read them.

Here's an inventory.

What's in the Modernization Plugin

dotnet/skills currently carries 16 plugins and 96 skills. The one that matters for this post is dotnet-upgrade, described in the repo as covering migration and upgrade of .NET projects across framework versions, language features, and compatibility targets.

It contains six skills:

Five of the six are modern-to-modern. One, thread-abort-migration, reaches back into .NET Framework, and it's a well-built skill: it handles ThreadAbortException, Thread.ResetAbort, the SYSLIB0006 diagnostic, and the ASP.NET Response.End case where Thread.Abort is called on your behalf. That is exactly the kind of semantic trap that breaks a naive port.

Elsewhere in the repo, msbuild-modernization handles legacy .csproj to SDK-style conversion, packages.config to PackageReference, and AssemblyInfo.cs removal. It also draws its own boundary explicitly, scoping itself out of .NET Framework projects that cannot move to SDK-style at all.

Now the grep. Across all 96 SKILL.md files in the repository:

• VB6 — 0 mentions
• MS Access — 0 mentions
• Visual Basic — 0
• WebForms / Web Forms — 0
• PowerBuilder — 0
• Delphi — 0
• ActiveX — 0
• Silverlight — 0
• Crystal Reports — 0

The Azure side tells the same story from a different direction. microsoft/azure-skills ships 28 skills. azure-cloud-migrate covers AWS Lambda to Azure Functions, Beanstalk/Heroku/App Engine to Azure App Service, and Fargate/EKS/Cloud Run to Container Apps. azure-upgrade covers hosting plan changes and Java SDK namespace modernization (com.microsoft.azure → com.azure).

Every one of those is a hosting migration or a dependency migration. None crosses a language boundary. None crosses a UI paradigm. The codebase is assumed to already be written in something the agent can read fluently and something the target runtime already speaks.

How a Migration Skill Gets Graded

This is where the repository becomes instructive, because Microsoft publishes the eval harness too: 97 eval.yaml files, one per skill, each defining stimuli, graders, and an LLM rubric.

Here's the grading block from the msbuild-modernization eval, condensed:

 graders:
  - type: output-contains
    config: { substring: "Microsoft.NET.Sdk" }
  - type: prompt
rubric:
  - Identified project as non-SDK-style and suggested migration to SDK-style
  - Identified explicit Compile Include items as unnecessary
  - Identified AssemblyInfo.cs as replaceable by SDK auto-generation

The skill passes if the agent's response text contains the right strings and an LLM judge agrees it suggested the right things. Nothing in that eval builds the converted project.

The harness does have a stronger grader. It's called exit-success, and the repository's own coverage tooling annotates it as meaning the project builds and tests pass. That's a real bar because it executes code.

But it is applied unevenly. Of the 97 evals in the repo, 31 contain at least one exit-success assertion. The other 66 are graded on text matches, file contents, and rubric judgments.

Inside dotnet-upgrade specifically:

Read the first three rows in order and you can watch Microsoft's evaluation standards tighten in real time. The 8→9 and 9→10 skills are graded on 36 and 50 regex matches respectively against what the agent said. The newest one, 10→11, attaches a build assertion to every single stimulus.

Microsoft is converging on the right idea: the code has to compile. Hold onto that, because it's the setup for the real problem.

Why "Builds and Tests Pass" Is the Ceiling, Not the Floor

Take exit-success at its strongest reading, the migrated project compiles and its test suite is green and ask what it proves about a legacy migration.

It proves the output is syntactically valid C# and that the tests which came with the code still pass.

For a .NET 9 microservice with 80% coverage, that's meaningful. For a 1.4-million-line VB6 or PowerBuilder estate written in 1998 and extended by six developers who have since retired, it proves almost nothing, for two reasons:

1. The Verification Void

The assertion presumes a test suite exists to carry forward. In legacy estates, automated tests are nonexistent. The verification method with the highest rigor in Microsoft's entire harness is structurally unavailable to precisely the systems where operational risk is highest.

2. Syntactic Validity vs. Semantic Equivalence

"Compiles and passes its own tests" is not equivalence. Consider these classic runtime edge cases:

• A VB6 Variant comparison that silently coerces strings to numbers depending on context.
• A PowerBuilder DataWindow holding implicit update state across transaction boundaries.
• An implicit SQLCA commit in legacy database calls.
• A form-level global variable that survives a modal dialog invocation.

Each of these can be transcribed into C# that compiles cleanly, executes without exceptions, and computes a fundamentally different answer than the system it replaced.

That divergence does not surface in a build log or a compiler error. It surfaces in a reconciliation report three quarters later when the accounting team notices financial totals have drifted.

 +-----------------------------------------------------------------------+
|                       THE VERIFICATION GAP                            |
+-----------------------------------------------------------------------+
|  Microsoft Skill Harness Ceiling  -->  [ Builds & Tests Pass ]        |
|                                                  |                    |
|  Operational Modernization Risk   -->  [ Unhandled Runtime Semantics ]|
|                                        [ Missing Test Suites ]        |
|                                        [ Silent Data Coercions ]      |
+-----------------------------------------------------------------------+

What Migration-Grade Verification Requires

If "builds and tests pass" is the absolute ceiling of what an IDE-resident agent harness can assert, then modernizing a mission-critical system of record requires a completely different class of instrumentation:

1. Behavioral Equivalence over Compilation: The unit of verification can't be "does it build." It must be "given identical inputs, does the modernized system produce bit-identical outputs?" The legacy system and target system must run side-by-side against production data, with automated differential audits at the database level. The legacy runtime is the spec.
2. Deterministic Rules Engine + Probabilistic LLM: Relying solely on LLM context windows for bulk code rewrite introduces non-deterministic hallucination into runtime logic. High-rigor modernization demands Abstract Syntax Tree (AST) deterministic transformations for business logic, reserved for LLMs only where semantic intent needs synthesis.
3. Per-Unit Self-Test over Whole-System Regeneration: Verification must be scoped to bounded contexts and individual endpoints. If an equivalence test fails, you fix the specific transformation rule. You don't prompt the LLM to "try regenerating the module again." Regenerating the whole system resets every equivalence result you've already earned.
4. Complete Transformation Provenance: Modernization requires a line-of-sight audit trail: exact mapping showing which legacy construct produced which target C# construct, and under which specific rule. Without that provenance, a divergence discovered nine months post-migration becomes an impossible exercise in software archaeology.

The Boundary Is Clear

None of this is a criticism of Microsoft's skills. They are well-authored, honestly scoped, and openly tested. The DO NOT USE FOR clauses in their frontmatter display far more discipline than most commercial marketing. Microsoft has drawn a boundary and labeled it clearly.

The point is that the boundary is now legible. Anyone can read where the platform's built-in agent knowledge ends: at the edge of the modern .NET language family, verified by string match, and occasionally by build assertion.

Everything past that edge: the millions of lines of VB6, PowerBuilder, Clarion, Access, Delphi, and legacy WinForms running global payroll, dispatch, and claims processing remains unencoded. Not through neglect, but because an implicit coercion rule or state-bound transaction cannot be verified by checking whether Microsoft.NET.Sdk appears in a Copilot chat response.

That is a different engineering problem. And it demands a different instrument.

Methodology & Reproducibility

Analysis performed August 4, 2026, against dotnet/skills (commit 6fce087) and microsoft/azure-skills (commit 1d88f75), both main branches.

To reproduce these metrics on your local machine:

 # Clone the repositories
git clone https://github.com/dotnet/skills.git
git clone https://github.com/microsoft/azure-skills.git

# Count skills and plugins
find dotnet/skills/plugins -name SKILL.md | wc -l
find dotnet/skills/tests -name eval.yaml | wc -l

# Run legacy tech greps across dotnet/skills
grep -rnwi "dotnet/skills" -e "VB6" -e "Visual Basic" -e "PowerBuilder" -e "Delphi" -e "ActiveX" -e "WebForms"