How we optimize GitHub Copilot for every new model
Julia Kasper, Product Manager at Microsoft, presents the evaluation harness behind GitHub Copilot’s coding agent at GitHub Universe 2026. The session focuses on integrating models from OpenAI, Claude, and Gemini while maintaining quality across different clients and configurations.
The harness behind model integration
The session’s central takeaway is that the harness is the product: supporting a new model involves more than making it available for selection. GitHub Copilot’s evaluation framework continuously tests, measures, and helps optimize the coding agent across a matrix of clients, models, and configurations.
Kasper covers Copilot’s approach to model choice, its model integration strategy, and the workflow used to bring new models into the product.
Evaluating quality and catching regressions
The framework combines several evaluation methods:
- Reproducible benchmarks to support repeatable testing.
- Thousands of autonomous coding runs to evaluate agent behavior on coding tasks.
- LLM-graded assertions to assess results and help identify regressions before they reach developers.
- Offline evaluations and internal dogfooding to test changes within the team.
- Public GitHub Copilot benchmarks as part of the broader evaluation process.
- Online experiments and key metrics to assess quality in use.
These methods support continuous improvement of the harness as models and configurations change.
Session chapters
- 00:00 — Optimizing GitHub Copilot for new models
- 01:04 — Understanding the GitHub Copilot harness
- 02:38 — Ensuring GitHub Copilot quality
- 03:24 — GitHub Copilot’s model choice identity
- 04:14 — GitHub Copilot model integration strategy
- 05:02 — The model integration workflow
- 06:34 — Continuous evaluation for harness improvement
- 07:06 — Offline evaluations and internal dogfooding
- 08:21 — Public GitHub Copilot benchmarks
- 08:57 — Online experiments and key metrics
- 11:03 — Key takeaways: the harness is the product