AI Observability and Optimization with Microsoft Foundry at Ignite 2025
This session featuring Abhi Bhatt, Sebastian Kohlmeier, and Sam Naghshineh from Microsoft Events explores how to achieve total visibility and optimization for AI agents using Microsoft Foundry, with practical demos and enterprise best practices.
AI Observability and Optimization with Microsoft Foundry at Ignite 2025
Speakers: Abhi Bhatt, Sebastian Kohlmeier, Sam Naghshineh
Event: Microsoft Ignite 2025
Session Code: BRK190
Overview
This advanced session demonstrates using Microsoft Foundry to monitor, optimize, and scale AI agent fleets with robust observability and control tools. Attendees learn how Foundry enables:
- Full visibility into deployed AI agents
- Real-time dashboards, diagnostics, and fleet management
- Continuous monitoring and evaluation of production AI workflows
- Integration of observability stack with Azure services
- Deployment gate processes enforcing safety and risk checks before rollout
- Human-in-the-loop release approvals
- Centralized scoring and thresholds stored in GitHub for traceability
Key Highlights
- Evaluation Platform: New risk and safety evaluators to ensure agent reliability
- Demo: Weather agent scenario showing intent resolution and evaluation
- Production Tools: Overview of Foundry's observability and continuous evaluation features
- Integration: Seamless connection with Azure for enterprise-grade monitoring
- Partner Case Study: CarMax VP of Data & AI Engineering shares real-world deployment experiences
- Release Management: Each Sky release undergoes mandatory AI evaluations, with results logged in GitHub
- Decision Gates: Human decision-makers approve or reject releases based on defined thresholds and scores
Chapters
- Details on evaluation platform and safety evaluators
- Agent demo: intent resolution and evaluation
- Introduction to production observability
- Observability stack integration with Azure
- Partner insights (CarMax)
- Deployment gate processes and evaluation checks
- Using GitHub for audit and access to evaluation results
- Session wrap-up and related resources
Resources & Further Learning
Takeaways
By leveraging Microsoft Foundry's AI observability stack, organizations can proactively monitor, validate, and optimize their AI deployments, ensuring safety, reliability, and compliance for agent fleets at scale.