Build agentic testing systems to validate AI generated code | ODSP912
Microsoft Developer argues that as AI accelerates software creation, “vibe coding” and traditional manual checks don’t provide enough rigor to keep AI-generated logic reliable at scale.
Overview
The session introduces Agentic Testing: using autonomous agents to validate autonomous systems. It focuses on patterns for building test agents that can:
- Execute end-to-end workflows autonomously
- Detect failures across multi-step processes
- Produce shareable evidence of what happened during a run
- Continuously verify behavior as part of CI/CD
TestMD framework integration for CI/CD
The presenter highlights integrating a Markdown-based TestMD framework into CI/CD so tests can be authored and executed in a way that keeps up with agent-generated changes.
Shareable evidence: video logs and trace runs
A key theme is making failures diagnosable by producing artifacts such as:
- Video logs of runs
- Trace runs that capture execution details
KNCLI interactive mode and agent environment setup
The demo walkthrough includes starting test execution and opening KNCLI in interactive mode, then initializing a KNCLI session within an agent environment.
Demo: validating an end-to-end workflow
The session demonstrates KNCLI supporting AI agents as they build and test an end-to-end workflow, including:
- Order placement
- Test data generation for checkout
- Completing workflow steps and validating outcomes
- A successful run where the workflow completes and the test passes