Code with Cosmos: Operating AI Agents in Production
Marko Hotti and Madeline Egan explain how to move an AI agent from a demo to a production-ready system using Azure Cosmos DB, focusing on observability and evaluation practices like tracing, regression testing, and before/after metrics to improve retrieval quality, latency, token usage, memory efficiency, and RU consumption.
Overview
The video presents a practical framework for operating AI agents in production with Azure Cosmos DB, with an emphasis on measuring and improving agent behavior across multiple layers of the system.
Key resources referenced:
- GitHub repo: https://aka.ms/cosmos-travel-planner
- Series playlist: https://aka.ms/CodeWithCosmosPlaylist
Production framework for AI agents
The presenters focus on taking an agent beyond a “working demo” by putting repeatable measurement and improvement loops in place.
Observe
They highlight using traces to understand what the agent is doing end-to-end, so you can pinpoint where quality or performance issues are coming from.
Evaluate
They describe building evaluation suites to assess agent behavior and outputs in a consistent way, rather than relying on ad-hoc manual testing.
Optimize
They discuss using before-and-after metrics to validate that changes actually improve the system, rather than shifting problems around.
What to measure and improve
The video calls out concrete areas where instrumentation and evaluation can drive improvements:
- Retrieval quality (improving what the agent fetches and uses)
- Latency (reducing end-to-end response time)
- Token usage (controlling LLM cost and throughput)
- Memory efficiency (improving how the agent stores/uses state)
- Azure Cosmos DB request unit (RU) consumption (reducing database cost and bottlenecks)
Layers to consider when debugging and tuning
They frame optimization work across multiple layers, rather than treating the agent as a single black box:
- Data layer
- Retrieval layer
- Memory layer
- LLM layer