In today's software development ecosystem, AI agents have become must-have tools for speeding up code creation, automating repetitive tasks, and improving team productivity. However, measuring whether they are actually working as we expect is not trivial. Traditional assessments often yield numbers that seem solid, but hide a much more complex reality: poorly defined criteria, environments that do not reflect real usage, and poorly placed trust in metrics. In this article, we explore how to build AX (Agent Experience) assessments that really provide useful information, avoiding the most common biases and aligning measurement with business objectives.
The first step to obtaining reliable signals is to understand that an evaluation is not an end in itself, but a means to identify the agent's shortcomings in the use of your technology. If the measurement system tells you that everything is going well when in fact the agent is generating incorrect or incomplete code, you will be optimizing in the wrong direction. Therefore, before launching a battery of tests, it is worth reflecting on what exactly we want to measure: the agent's ability to discover a library? The quality of the code generated? Integration with an existing ecosystem? Each objective requires a different evaluation design.
One of the most frequent mistakes is to write vague evaluation criteria, such as 'the code must handle errors correctly'. Two different reviewers will interpret that phrase differently, and the same goes for language models acting as judges. To obtain consistent results, the criteria must be so specific that there is no room for interpretation. For example, instead of asking for 'proper API usage', we can break down the condition into atomic checks: 'all calls to external services are wrapped in try-catch blocks', 'network errors are retried with exponential backoff', 'application errors are propagated to the caller while preserving the original code'. Each of these checks can be evaluated independently, with pass, fail, or skip states, providing a granular and actionable signal.
Another critical aspect is the need for multiple executions. Language models are inherently non-deterministic: the same instruction can produce different results in each invocation. Relying on a single run is tantamount to drawing conclusions based on a statistically insignificant sample. Experience indicates that with five repetitions per scenario you begin to observe a stable pattern, enough to make informed decisions about whether an extension or improvement is really helping. If the results are highly variable between executions, that is valuable information: it can indicate that the scenario is not well defined, that the environment has hidden variables, or that the agent is inherently unstable for that task.
The cleanliness of the assessment environment deserves special attention. User names, directory paths, project names, or any other semantic clue can skew agent behavior. A path like /home/demo/test-project is telling the model that this is a test scenario, and that can cause it to take shortcuts or not apply the same quality practices as in a real environment. Ideally, the environment should be neutral, containing no clues about what is being evaluated. In addition, the environment must faithfully represent the working context of the end users. If most of your developers use Windows and Visual Studio Code, evaluating Linux with a CLI agent won't do you much good. Differences in the operating system, shell, and tool ecosystem change model decisions, and what works in one environment may fail in another.
The judge's calibration is another fundamental pillar. It is not enough to write detailed criteria; it is necessary to verify that the model that acts as an evaluator interprets them correctly. To do this, between five and ten real agent outputs are selected – not artificial examples – the expected verdicts are manually noted and compared with those issued by the judge. When there are discrepancies, it is almost always because the criteria are ambiguous. Adjusting the wording, repeating the calibration and ensuring that the judge is correct in all the cases of evidence is an indispensable iterative process. It is also advisable to check consistency: execute the same judge on the same output several times and verify that the verdict does not change. If it does, the criterion needs more specificity, not a more powerful model.
Among the most common failures in automatic evaluators, the inability of certain models to reason about semantic versions (semver) stands out. A small judge can identify the dependencies of a project, but not reliably determine whether the version used is the latest or if it meets compatibility ranges. In those cases, the solution is not to force the model to do something for which it is not prepared, but to divide the task: that the judge identifies the version, and that an external tool – a deterministic script – performs the comparison. Similarly, asking the judge to calculate aggregate scores is often a source of arithmetic errors. It is much more reliable for the judge to return only individual verdicts (pass/fail/skip) with their evidence, and for an external process to calculate the final metrics deterministically.
For companies that are developing products based on AI agents, having a robust evaluation system is not a luxury, it is a competitive necessity. At Q2BSTUDIO, we understand that the quality of AI-generated software is directly dependent on how we measure and adjust agent behavior. That's why we offer bespoke application services and AI consulting for enterprises, where we integrate these assessment practices into the development cycle. We work with cloud technologies such as AWS and Azure, implement cybersecurity solutions, and develop dashboards in Power BI so that performance information is clear and actionable. Our team helps organizations design agent tests that truly reflect usage in production, avoiding the false positives that so often derail improvement efforts.
A practical recommendation for those starting from scratch: limit the initial scope to three or five scenarios that represent the most common tasks of your developers. Write prompts that sound like real requests, not optimized instructions for evaluation. Define criteria in the format of a checklist, with explicit pass, fail and skip conditions. Calibrate the judge with real outputs from the agent. Run each condition five times (baseline and with your extension). If the results are unanimous, you have your answer; if not, run more reps. Fixes the environment: records the operating system, tool versions, model, and harness. Every time you change something, recalibrate. And don't compare scores between versions of the rubric without recalibrating.
Variability is not an enemy, it is a source of information. If a scenario happens 90% of the time with your extension and 50% without it, you've found a case where your solution makes a real difference. But if the hit rate fluctuates wildly from one day to the next, there's probably an uncontrolled variable—an update to the base model, a change in the harness, an extension that was automatically updated. Keeping a detailed record of each execution will allow you to identify these drifts and correct them before they contaminate your conclusions.
Ultimately, building AX assessments that actually work requires discipline in rubric design, statistical rigor, and systematic hygiene of the environment. These aren't particularly complex tasks individually, but keeping them all aligned as the number of scenarios grows is where many teams fail. The reward is worth the effort: when you see a score change, you know it reflects a real improvement in the agent's ability. When your leadership asks if investing in AI agents is paying off, you have data, not anecdotes. If you're ready to take the next step and need support to implement a robust assessment system or develop custom software that integrates these practices, Q2BSTUDIO is ready to help you turn artificial intelligence into a real driver of productivity and quality.
