How to locally evaluate AI agents in .NET Microsoft Agent Framework

Authorised Territory demonstrates how to build an AI agent with the .NET Microsoft Agent Framework using a local Ollama model, then evaluate the agent locally with built-in checks to confirm response quality and that required tools were actually invoked.

Overview

The video walks through building and evaluating an AI agent in a .NET console application.

Scenario: travel blogger agent with an email tool

• The agent acts as a travel blogger.
• It generates a paragraph about the history of Barcelona.
• It sends the generated content via email by invoking a tool.

Connecting Microsoft Agent Framework to a local Ollama model

• Connect the .NET Microsoft Agent Framework to a local Ollama model.
• Create an AIAgent using IChatClient.

Adding and using tools with an AI agent

• Add tools to the agent and invoke them as part of the agent workflow.
• Review how tool approval works in the Microsoft Agent Framework.

Local evaluation with LocalEvaluator

• Evaluate the agent locally using LocalEvaluator.
• Use built-in evaluation checks, including:
  • KeywordCheck to verify the agent response contains expected keywords.
  • ToolCalledCheck to verify a specific tool was actually invoked.
• Evaluate whether the agent completed the requested task.

Inspecting evaluation results and metrics

• Inspect evaluation results and metrics produced by the local evaluation.
• Review the output in a .NET console application.