Behavioral scientific research on AI agents asks a deceptively simple question: when an autonomous system is placed in a complex environment, how does it actually behave, and what makes that behavior change? Until recently this work was manual, slow, and hard to reproduce at scale. A research team has now built a multi-agent system named AEROBAT that runs the entire investigation on its own, from a one-line behavior description to a written report, as detailed in their paper on automating behavioral research on AI agents. This guide walks through the pipeline AEROBAT automates so you can reproduce the workflow, adapt it to your own target behavior, and interpret the kind of evidence it produces.
Understanding agent behavior is no longer a niche academic exercise. When agents handle shopping, scheduling, negotiation, or document analysis, their conduct under pressure has real consequences. zBrandCo has separately reported that research agents can leak private data in a meaningful share of tests, described in MosaicLeaks finds research agents leak private data, and that the same leak rate can be cut sharply with targeted fixes, covered in MosaicLeaks research agent leak fixes. Findings like those are exactly the kind of behavioral claim that benefits from a repeatable, controlled measurement pipeline rather than a single anecdote.
Before you begin
You do not need the published system running locally to learn the method, but you do need a clear decision about what behavior matters to you and a way to run the target agent inside a simulated environment you control. The paper’s contribution is the method and the evidence, not a hosted product, so the practical takeaway is the structure: define, hypothesize, design, execute, assess, analyze the AEROBAT paper. Keep that skeleton in view as you read the steps below.
What the automated pipeline actually does
The researchers present AEROBAT as a system that carries behavioral science on other agents forward without hands-on lab work from a person the study. You begin by describing a behavior you want to understand, and from there the system drives the whole investigation on its own: it proposes hypotheses, constructs controlled tests, scores what the agent did, works through the statistics, and produces a write-up. The architecture that coordinates those stages is shown in the project’s overview figure.
Source: arXiv HTML edition of 2608.10030.
The loop is deliberately modular. Each stage is handled by a specialized agent rather than a single monolithic prompt, which keeps the hypotheses, the experimental designs, and the assessments as separate, inspectable artifacts. That separation is what lets a researcher trace any conclusion back to the experiment that produced it.
Step 1: Define the target behavior you want to study
Every run starts with a single input: the behavior you care about. The paper frames this as the user providing an arbitrary target behavior, meaning you are not limited to a preset menu of tests the arXiv report. You might study whether a shopping agent is susceptible to seller persuasion, whether a coding agent withholds information under conflict, or whether a planning agent chooses flexible execution over rigid literal compliance.
Be specific about the circumstance, not just the trait. “Withholds information” is vague; “withholds a known critical constraint when a user’s request conflicts with a secondary instruction” is testable. The quality of everything that follows depends on how clearly the target behavior is bounded, because the hypothesis generator works backward from that definition.
Step 2: Generate hypotheses about what modulates the behavior
Once the behavior is specified, the system proposes hypotheses about the variables that change how strongly the behavior appears the AEROBAT paper. Rather than guessing blindly, it reasons about which environmental or contextual factors are likely to shift the outcome, then states each hypothesis in a form that can be tested by a controlled simulation. The authors lay out this stage in the paper’s methods section, reproduced in the HTML build AEROBAT methods, arXiv HTML.
Image credit: the authors’ HTML version, arXiv 2608.10030.
The hypothesis step is where manual research normally consumes the most time, because a good hypothesis must be both plausible and falsifiable. Automating it does not remove the need for scientific judgment; it removes the mechanical labor of drafting many candidate explanations so that the expensive part—running and assessing experiments—can begin sooner.
Step 3: Design controlled experiments
With hypotheses in hand, the system designs matched, controlled simulation environments that manipulate exactly the variable each hypothesis names the study. “Controlled” is the operative word: each experiment holds everything constant except the factor under test, so any difference in behavior can be attributed to that factor rather than background noise.
In the study itself, the authors used the system to design 1,240 controlled experiments across 12 target behaviors, a scale that would be impractical to hand-build one at a time the AEROBAT paper. The design stage is therefore as much about coverage as about correctness: it must produce enough comparable runs to support a statistical claim, not just a suggestive single trial.
Step 4: Execute the experiments as simulation rounds
The designed experiments are then executed as simulation rounds, where the target agent is placed in the prepared environment and its behavior is recorded the arXiv report. For the 12 behaviors studied, the system executed 23,512 simulation rounds in total, providing the raw behavioral record that later stages assess.
Credit: reproduced from arXiv HTML 2608.10030.
Running thousands of rounds is what turns a hypothesis from plausible to measurable. A single observed action could be luck or a quirk of one prompt; tens of thousands of rounds let the system estimate how reliably a behavior appears when the tested condition changes.
Step 5: Make behavioral assessments
After execution, the system performs behavioral assessments on the recorded runs, scoring what the agent actually did against the behavior being studied the study. This stage converts raw simulation transcripts into structured judgments about whether the target behavior occurred and under what conditions.
Source: arXiv HTML build of 2608.10030.
Keeping assessment separate from execution matters for credibility. If the same component that ran the experiment also decided whether it succeeded, confirmation bias could creep in. A dedicated assessment step makes the scoring rule explicit and repeatable.
Step 6: Analyze results and write the report
The final stages analyze the assessments statistically and write the report, closing the loop from a one-line behavior description to a documented finding the AEROBAT paper. Across the study, the system generated and tested 79 hypotheses for the 12 behaviors, and moderate-to-strong statistical evidence supported 26 of them, including some the authors describe as novel. The paper is also archived under its persistent identifier, DOI 10.48550/arXiv.2608.10030.
That 26-of-79 result is the honest headline: automation widened the search space enough to surface real signals, but most candidate hypotheses still did not clear the statistical bar. The value is in the pipeline’s ability to test many ideas cheaply, not in guaranteeing that any particular idea is correct.
How to read the evidence it produces
When you run this kind of pipeline, treat “moderate-to-strong statistical evidence” as a threshold, not a verdict the study. A hypothesis that clears it deserves follow-up with human-designed validation, especially before you act on it in a production system. The automated report tells you where to look; it does not replace domain expertise.
The authors are explicit that automated behavioral research is meant to complement manual work, not replace it. Manual studies supply the careful, small-sample investigations that give automated sweeps their hypotheses and their reality checks. Used together, they cover both breadth and depth.
Limitations to keep in mind
The approach depends entirely on the fidelity of the simulation environments. If the simulated world omits a factor that matters in production—say, real financial stakes or genuine user pushback—the measured behavior may not transfer the arXiv report. Treat simulation evidence as necessary but not sufficient.
It is also worth being clear about scope: zBrandCo has not executed AEROBAT, and this walkthrough describes the workflow the paper documents rather than independent testing. The figures embedded above are reproduced from the authors’ published HTML version of the study, not from a zBrandCo laboratory.
Applying the workflow to your own agents
To use this pipeline on your own systems, start by writing the target behavior as a precise, observable statement, then let the hypothesis generator propose what might shift it, and insist that every designed experiment changes exactly one variable at a time the AEROBAT paper. Keep assessment separate from execution, run enough rounds to support a statistical claim, and reserve human review for any hypothesis that crosses the evidence threshold.
Whether you adopt the published system directly or rebuild the loop with your own agents, the disciplining structure is the same: a bounded behavior, falsifiable hypotheses, controlled environments, and explicit assessment. That structure is what makes behavioral findings about AI agents something you can reproduce and defend, rather than a story you heard once.
Key terms used here
- Target behavior: the specific, observable conduct you want to measure in an agent.
- Controlled experiment: a simulation that changes one factor at a time so cause is isolable.
- Behavioral assessment: the scored judgment of whether the target behavior occurred.
- Simulation round: one executed run of the target agent inside a prepared environment.
