Why AI agents Cost So Much

Takuto Higuchi interviews Sindhura Bandhakavi, Director of Engineering in Microsoft CoreAI, about why AI agents can become expensive to run at scale and how to approach cost as an engineering and operations problem.

Overview

The episode focuses on the emerging discipline of AI token economics (“tokenomics”) and how to optimize agent performance without sacrificing quality.

Why AI agents get expensive at scale

• As agent usage grows, operational costs become more visible and harder to predict.
• Cost drivers discussed include:
  • Token consumption patterns
  • Model choice and per-request model costs
  • The relationship between quality, latency/performance, and spend
• A key theme is that reducing tokens alone is not always the right optimization strategy; it can trade away quality or shift costs elsewhere.

Measuring ROI and treating AI as a managed investment

• The discussion frames AI spend as something to measure and manage rather than treat as a fixed platform cost.
• Topics covered include:
  • Measuring costs and evaluating agent effectiveness
  • Establishing controls for usage spikes
  • Ensuring each request hits an acceptable balance of quality, performance, and cost

Using Azure AI Foundry to optimize cost and performance

The episode highlights Azure AI Foundry capabilities aimed at improving efficiency and maintaining quality as workloads scale.

Model Router

• Described as a way to automatically select the right model for each request.
• The episode includes a live demo segment showing Model Router in action.

Agent Optimizer (Foundry Agent Service)

• Presented as a capability to improve agents over time.
• The episode includes a live demo focused on optimizing prompts and overall agent performance.
• The conversation also describes building a continuous optimization “flywheel” for ongoing improvements.

Resources

• The economics of agent optimization: from pilots to measurable returns: https://azure.microsoft.com/blog/the-economics-of-agent-optimization-from-pilots-to-measurable-returns/
• Model router for Microsoft Foundry concepts: https://learn.microsoft.com/azure/foundry/openai/concepts/model-router
• How to use model router for Microsoft Foundry: https://learn.microsoft.com/azure/foundry/openai/how-to/model-router
• Agent optimizer in Foundry Agent Service overview: https://learn.microsoft.com/azure/foundry/agents/concepts/agent-optimizer-overview
• Introducing Agent Optimizer in Foundry Agent Service: https://devblogs.microsoft.com/foundry/agent-optimizer-build2026/
• What's New in Microsoft Foundry, June 2026: https://devblogs.microsoft.com/foundry/whats-new-in-microsoft-foundry-june-2026/

People

• Host: Takuto Higuchi
• Guest: Sindhura Bandhakavi (Director of Engineering, Microsoft CoreAI)

Chapters (from the video description)

• 00:00 In this episode
• 00:26 Welcome to Tokenomics
• 01:00 Why AI Agents Cost More Than You Think
• 04:00 The Hidden Cost of Chasing Fewer Tokens
• 07:00 Measuring ROI for AI Agents
• 10:00 Choosing the Right Model with Model Router
• 12:00 Live Demo: Model Router in Action
• 15:30 Improving Agents Over Time with Agent Optimizer
• 18:00 Live Demo: Optimizing Prompts and Agent Performance
• 21:20 Building a Continuous Optimization Flywheel
• 22:20 Your AI Budget Review Checklist
• 23:16 Key Takeaways and Resources