🍰 Compare Models & Harnesses with Agent Bake Off!
The Visual Studio Code team presents Agent Bake Off as a way to compare AI models and harnesses on a shared coding task, then review and combine their results.
Comparing agents on the same task
The demonstration runs six AI coding agents in parallel on a Windows settings reorganization. Using the same task provides a common basis for comparing different models and harnesses.
The comparison considers more than the finished changes:
- Results: What each agent produces.
- Tool use: How each agent uses tools during the task.
- Context: The context involved in each attempt.
- Cost: The cost of the different attempts.
Reviewing and synthesizing results
A judge highlights the strengths and weaknesses of each attempt. Synthesis then combines the strongest ideas into a finished change, rather than simply choosing one agent’s output.
The video also shows how this approach applies when starting without an existing project.
Video chapters
- 00:00 — Introduction: Comparing AI Models
- 01:42 — Reviewing and Synthesizing Results
- 04:04 — Comparing Models Without a Project
- 04:38 — Happy Coding