Technical Brief: Microsoft Open-Sources `code-testing-generator`

Source Article: MarkTechPost ("Microsoft Open Sources code-testing-generator: a Polyglot Unit-Test Agent That Hits 92.1% Task Completion Versus 78.9% for Stock Copilot")

Article Date: August 6, 2026

Repository: dotnet/skills (MIT License | dotnet-test plugin)

Date Saved: August 8, 2026

Saved Path: ~/topics/research/microsoft_code_testing_generator_2026.md

1. Executive Summary

Microsoft has open-sourced code-testing-generator, a polyglot autonomous agent that writes unit tests, executes them locally, and self-corrects until all tests pass. Distributed as part of the dotnet-test plugin inside the MIT-licensed dotnet/skills repository, the tool solves a core gap in LLM code generation: AI assistants frequently generate syntactically plausible unit tests that fail during actual test-runner execution due to unstated framework conventions, incorrect imports, or missing mocks.

On Microsoft's internal 152-task benchmark across multiple languages, code-testing-generator achieved a 92.1% task completion rate (140/152) compared to 78.9% (120/152) for stock GitHub Copilot operating under identical base models and prompts.

2. Key Architecture & Mechanics

Repository-First Context Inspection

Standard LLM prompts like "generate unit tests for this function" fail because they lack codebase conventions (assertion styles, test file naming patterns, mocking libraries, project file structures).

code-testing-generator executes a pre-generation repository scan:

1. Framework Discovery: Identifies active testing frameworks (e.g., xUnit, NUnit, MSTest for .NET; pytest for Python; Jest/Vitest for TypeScript; go test for Go).

2. Conventions & Fixtures: Parses existing test files to extract mock setup patterns, dependency injection rules, and naming styles.

3. Target Scope: Identifies public interfaces, edge conditions, and error branches requiring coverage.

Closed-Loop Execution & Self-Correction

Rather than producing unverified code, the agent operates in an iterative execution loop:

1. Plan & Draft: Generates unit test files matching existing codebase patterns.

2. Execute Local Test Runner: Invokes the native CLI test runner (dotnet test, pytest, npm test, go test) inside a sandboxed subprocess.

3. Parse Diagnostics: If a test fails or compilation errors occur, the agent captures stderr/stdout, parses stack traces, and refines the test code.

4. Verified Pass: Iterates until 100% test execution success is confirmed or max retry depth is reached.

3. Benchmark Comparisons (Microsoft 152-Task Benchmark)

| Evaluation Metric | Stock Copilot | code-testing-generator Agent | Delta Improvement |

| :--- | :---: | :---: | :---: |

| Task Completion Rate | 78.9% (120/152) | 92.1% (140/152) | +13.2% Absolute (+16.7% Relative) |

| First-Pass Compilation Rate | ~68% | 86.4% | +18.4% |

| First-Pass Execution Pass Rate | ~54% | 81.2% | +27.2% |

| Multi-Turn Error Recovery | Failed on 72% of errors | Resolved 68% of initial runner errors | High Error Resilience |

Note: Both evaluated pipelines used the identical underlying LLM backend and base system instructions.

4. Integration & Engineering Takeaways for Sentinel Integrations

1. Alignment with TDD / RED-GREEN-REFACTOR Skills:

code-testing-generator reinforces Sentinel's Test-Driven Development (TDD) and requesting-code-review skills. Integrating repo-first context discovery into local test-generation tools eliminates phantom imports and broken mocks.

2. Local Runner Integration on Local Inference Node / Orchestrator Node:

The dotnet-test plugin pattern (repo-scan $\rightarrow$ generate $\rightarrow$ local execution $\rightarrow$ diagnostic parsing) can be incorporated into Hermes local coding agents (such as Claude Code and local SFT models on Local Inference Node).

3. Repository Reference: Available in the official GitHub repository: github.com/dotnet/skills.