Agent evaluation leans almost entirely on outcome metrics such as success rate, which capture whether an agent finishes but not how steadily it behaves, the new paper argues arXiv abstract. The authors introduce the Behavioral Consistency Metric (BCM), a process-level score meant to sit beside success rate rather than replace it.
What the metric actually measures
BCM trains a model to predict whether a task succeeded from the behavioral features of an agent’s execution trace, then derives a per-trajectory feature-attribution vector and measures the mean pairwise similarity of those vectors within an agent system arXiv abstract. High similarity means the agent follows a stable strategy; low similarity means its behavior fragments from task to task. The study drew on roughly 9,000 trajectories from six language model agents working on software engineering tasks.
Why consistency splits from success
The central result is that cross-task and within-task consistency are separate axes that can pull apart. Some systems repeat themselves faithfully on the same task yet show no stable strategy across different tasks; others stay consistent at both scales arXiv paper. Because prior work measured only same-task reproducibility, it could not see this separation. The authors also show consistency is not a proxy for success rate: systems with similar win rates can diverge sharply in how consistently they behave.
A second finding is that the consistency gap between frontier and open-source systems persists even when task difficulty is held constant within a task arXiv paper. That controls for the easy explanation that frontier models simply face harder problems, and suggests the gap is structural rather than a side effect of difficulty.
The consistency work is one of several recent arXiv studies examining how agents actually behave under load arXiv abstract. A related zBrandco report examines an agent-memory model that bars stale and retracted data arXiv agent-memory model bars stale, retracted data. Another covers edge agents that cut latency violations under load arXiv study: edge agents cut latency violations to 3%.
For teams shipping agents into production, BCM offers a way to catch systems that look fine on pass-rate dashboards but behave erratically when conditions shift arXiv paper. The authors are explicit that the metric is meaningful only under specific conditions, which they spell out, so it should complement outcome metrics rather than become a standalone grade. As agent deployments move from demos to real workflows, a score that captures behavioral drift could become part of the standard evaluation toolkit.
