Agents in production: Observe evaluate and optimize with Foundry | BRK251
Deepraj Khot, Han Che, and Divya Chirasani present a practical continuous-improvement workflow for agents after deployment. The session combines Foundry demonstrations with a Tata Consultancy Services (TCS) case study.
From production behavior to continuous improvement
Deploying an agent is the start of an ongoing process: observing its behavior, evaluating its outputs, investigating unexpected issues, and improving performance. Because AI agents are non-deterministic, teams need evidence from actual runs as well as structured evaluations to assess whether they remain trustworthy in production.
The session connects several parts of this workflow:
- Tracing to examine agent behavior and support investigations.
- Datasets and evaluation frameworks to assess quality systematically.
- Production insights and continuous analysis to identify issues and support evidence-backed debugging.
- Human feedback to inform the improvement loop.
- Agent Optimizer workflows to address findings and improve agent quality.
- Fleet monitoring to track quality, reliability, latency, safety, and cost across multiple agents.
Evaluation and debugging demonstrations
The demonstrations cover traces, datasets, synthetic data generation, and offline evaluations. The presenters also introduce Foundry evaluation features, including rubric evaluators and decision models.
The production workflow extends evaluation to live traffic, helping teams identify quality gaps that emerge after deployment. Production insights and continuous analysis then provide evidence for debugging, while Agent Optimizer workflows connect those findings to quality improvements.
Monitoring across an agent fleet
Beyond investigating individual runs, the session addresses observability across multiple agents and cost management. The TCS case study illustrates token reduction and quality improvements using Foundry observability.
Session chapters
- 00:00 — Introduction: The challenge of trustworthy AI agents in production.
- 01:49 — Observability and evaluations: Why they matter for non-deterministic AI agents.
- 03:56 — Building reliable agents: Traces, datasets, and evaluation frameworks.
- 06:18 — Evaluation features: Rubric evaluators and decision models.
- 07:21 — Demo: Traces, datasets, synthetic data generation, and offline evaluations.
- 15:50 — Live traffic evaluations: Identifying production quality gaps.
- 17:08 — Production insights: Continuous analysis and evidence-backed debugging.
- 25:56 — Agent Optimizer: Workflows for addressing insights and improving agent quality.
- 32:34 — Fleet monitoring: Cost management and observability across multiple agents.
- 36:39 — TCS case study: Token reduction and quality improvements using Foundry observability.
Resources
This foundational breakout session, BRK251, is part of the Agent Platform track at Microsoft Ignite 2026 and is presented in English (US).