Content by waldek mastykarz (24)

Waldek Mastykarz examines whether naming a workspace folder ‘eval’ changes GitHub Copilot CLI behavior. Across about 900 runs with four models, the name increased agents’ suspicion that they were being tested without producing statistically significant behavioral differences. The analysis explains the experiment’s controls, limitations, and practical guidance for evaluation harnesses.

Waldek Mastykarz tests whether swapping a real GitHub repo for a local bare git remote changes AI coding agent behavior during evals, focusing on a rebase scenario where the agent must decide whether to push without being asked.

Waldek Mastykarz introduces the AX Practitioner Playbook, a practical method for evaluating how AI coding agents use your SDKs, docs, CLIs, and MCP tools, diagnosing why they fail, and validating fixes before shipping changes that improve agent behavior.

Waldek Mastykarz introduces the Agent Experience (AX) Practitioner Playbook, a practical method for evaluating how AI coding agents use your SDKs, docs, tools, and extensions—and for tracing failures back to their root causes so you can ship fixes to the sources agents rely on.

Waldek Mastykarz introduces Agent Experience (AX): how AI agents discover, choose, and correctly use your SDKs, APIs, and CLIs. He breaks AX down into propensity and efficacy, shows why you must measure both quality and task cost, and shares practical examples from GitHub Copilot Chat evaluations.

Waldek Mastykarz explains why a model’s published knowledge cutoff date doesn’t reliably predict whether it can do real work with a specific developer product, and shows how workload-based evaluations give a more accurate signal when building and testing AI coding agents and extensions.

Waldek Mastykarz explains why AI coding agent eval pass rates can be misleading when the sandbox leaks information, and how to design evaluations around information boundaries rather than individual tool deny-lists.

Waldek Mastykarz explains how to pick a “minimal viable model” for AI coding work by running your own evals instead of defaulting to the most expensive option. Using GitHub Copilot Chat in VS Code, he compares model quality scores and per-run costs across real upgrade scenarios to show why “smarter” can be worse value.

Waldek Mastykarz explains how to evaluate agent skills that call APIs without paying for test traffic or mutating production data. He outlines why direct evals against real endpoints create cost, state, and non-determinism problems, then shows a proxy-based approach using Dev Proxy to emulate APIs while keeping production URLs unchanged.

Waldek Mastykarz explains how to design AI coding-agent evaluations (AX evals) that produce real signal instead of misleading scores, focusing on representative prompts, precise pass/fail/skip criteria, judge calibration, and enough repeated runs to separate variance from actual improvements.

Waldek Mastykarz explains why AI coding-agent eval scores often fail to reproduce across machines, and how “hidden variables” like OS, shell choice, workspace paths, and language-server feedback can quietly change an agent’s behavior and your measured results.

Waldek Mastykarz tests the common advice to replace complex CLI flags with a single --json payload for AI agents, and shows that traditional args can be more reliable and far cheaper in tokens—especially once retries and shell-escaping issues (PowerShell vs Bash) are factored in.

Waldek Mastykarz shares results from 150 GitHub Copilot Chat agent runs comparing Claude Sonnet 4.6 vs 5, showing how a “cheaper” model can still cost more due to token spikes and variance, and how output quality can regress depending on the task and available documentation.

Waldek Mastykarz explains why public coding benchmarks like SWE-bench don’t predict how well an AI coding agent will perform in your real developer environment, and outlines a practical approach for running representative, repeatable evaluations against your own scenarios, extensions, and harness.

Waldek Mastykarz explains what goes wrong when an AI coding agent has zero training data about your internal SDKs or proprietary frameworks, and how to design extensions that teach the model effectively. He focuses on baseline evaluation, avoiding “closest match” hallucinations, and balancing correctness against token and context-window costs.

Waldek Mastykarz explains why “model preferences” (like “Claude prefers React”) are usually artifacts of missing or skewed context, not stable traits of an LLM. He breaks down how prompts, formatting, and attached project files change outputs, and how to evaluate AI developer tools using realistic, repo-shaped tests.

Waldek Mastykarz explains why “skills” for AI coding agents often fail when they return large chunks of documentation the model already knows, wasting tokens and pushing out more valuable context. He outlines a measurement-first approach to define a lean skill scope that targets only the gaps where the model actually fails.

Waldek Mastykarz explains why AI coding agent extensions that perform well alone can degrade results when installed alongside other extensions, and how to measure and reduce these composition effects in real developer workspaces.

Waldek Mastykarz explains why AI coding agents often keep using deprecated tooling (like legacy CLIs) even after teams ship replacements, and outlines practical ways to measure and correct agent behavior using extensions, explicit deprecation signals, and better naming.

Waldek Mastykarz explains how AI coding agents can silently scaffold outdated Node.js projects when they run npx without pinning versions, due to npm’s engine-aware manifest selection. The post breaks down why this happens and gives practical steps to make agent-driven scaffolding more predictable.

Waldek Mastykarz explains how to measure whether an AI coding agent extension actually improves generated code, using controlled comparisons, clear evaluation criteria, and repeatable scenario runs while tracking both quality and token cost.

Waldek Mastykarz explains why piling up dozens of agent “skills” can quietly burn your token budget and reduce response quality, and how to decide what should stay a skill versus what should become a manually-invoked prompt in tools like GitHub Copilot and VS Code.

Waldek Mastykarz breaks down what happens between a developer prompt and an AI agent generating code with your SDK, CLI, or API, focusing on how harnesses assemble context, how models choose (or skip) tools like MCP servers, and why response quality and error messages can make agents succeed or fail.

Waldek Mastykarz breaks down the “agent stack” behind AI coding agents and explains why most problems aren’t in the model or the agent harness, but in the extension layer you can actually control. He outlines common failure modes and a practical way to measure whether your extensions create lift or drag.