A research team has proposed a way to measure whether an “agentic learning harness” — an LLM wired to memory or retrieval that improves from feedback without retraining — is actually helping, without the labeled test sets that seldom exist in live security work DOI record. The study, posted on 11 August 2026 arXiv paper, was accepted at CAMLIS 2025, a conference on applied machine learning for information security.
Why labeled benchmarks fall short
Conventional evaluation leans on gains against labeled benchmarks, but the authors argue those labels are scarce, stale, and unrepresentative once a system runs in production arXiv paper. They also report that LLM-as-a-judge gives little signal, because a judge model is no stronger than the agent it grades, and that distilling from scarce, sporadic, biased labels is unreliable. The gap matters most in cybersecurity, the domain that motivated the work, where incidents arrive faster than anyone can label them for a clean test set.
How the teacher-student score works
The proposed framework rests on the scaling hypothesis. A stronger teacher model supplies sparsely sampled corrections to a smaller student paired with a continual-learning harness, and the harness is scored by how much the student converges toward the teacher over time arXiv paper. Across security tasks, model families, and harness designs, the authors show that a harness’s improvement relative to its teacher correlates with its improvement against a held-out gold standard, which validates teacher-relative lift as a proxy for true uplift when labels are missing DOI record. The same experiments confirm that LLM-as-a-judge between similarly powered models yields no usable signal.
What it means for builders
The result reaches past automated scoring. Because teacher-relative lift tracks real improvement, the authors suggest a teacher-sized model itself could be improved through the same harness when people supply the same kind of sparse, high-precision corrections DOI record.
For teams shipping agentic systems, the work reframes what evaluation must cost: rather than hunting for representative labeled data, engineers can watch a student chase a stronger model and read convergence as evidence of value. The approach also hints at a cheaper path to improving large models themselves — if sparse human corrections can lift a student, the same pattern may apply when people coach a frontier model. The challenge mirrors earlier findings that agentic harnesses reached vendor parity in coding assistants GitHub Copilot agentic harness hits parity with vendor tools, where judging real uplift is the hard part.
