Build custom AI agent evaluators in .NET Microsoft Agent Framework + Ollama

Authorised Territory walks through building custom AI agent evaluators in .NET using Microsoft Agent Framework (MAF) with a local Ollama endpoint.

Overview

The tutorial shows how to create a local AI agent and evaluate its responses using custom FunctionEvaluator checks. It focuses on building application-specific evaluation metrics (beyond generic LLM evaluation) and inspecting which checks pass or fail.

Starting solution referenced by the presenter: https://youtu.be/1T-OfUwcPDI

What the tutorial builds

• A local AI agent built with Microsoft Agent Framework
• A connection from MAF to a local Ollama endpoint
• Use of the Granite 4.1 8B model
• A custom evaluation pipeline that runs multiple checks and reports results

Custom evaluator 1: containsBarcelonaHistory (content evaluation)

This evaluator checks whether the agent response contains the information the app expects. The check verifies that the response:

• Mentions Barcelona
• Contains at least two historical terms
• Matches terms such as:
  • Roman
  • Sagrada
  • Catalan
  • Mediterranean
  • Olympics
  • 1992

The point of this example is to demonstrate a domain-specific evaluation metric rather than relying only on generic LLM scoring.

Custom evaluator 2: responseLength (output validation)

This evaluator enforces a simple structural requirement:

• The generated response must be longer than 500 characters

Even though it’s a basic check, it illustrates that custom evaluators can validate essentially any requirement you can express in C#.

Key .NET / MAF concepts used

• Defining custom evaluators with FunctionEvaluator.Create
• Combining multiple EvalCheck instances
• Creating a LocalEvaluator
• Running evaluation with EvaluateAsync
• Inspecting passed and failed evaluation metrics
• Building application-specific tests for AI-generated content