A new ServiceNow research benchmark called MosaicLeaks finds that AI deep research agents leak sensitive enterprise private data at a combined average rate of 34.0% for answer and full-information leakage. The benchmark, detailed in the official ServiceNow MosaicLeaks research announcement, tested 1,001 multi-hop research tasks to reach this finding. The study also found that standard performance-focused training is correlated with higher leakage risk for these agents, per the ServiceNow research team’s analysis of model behavior across test runs. This finding highlights a critical unaddressed privacy gap for enterprise teams deploying agents that combine internal document access with public web search.
What is the MosaicLeaks Benchmark?
The MosaicLeaks benchmark, released by ServiceNow’s AI research team, is designed to measure privacy leakage risk in deep research agents that combine internal enterprise document access with public web search. The benchmark is built around the mosaic effect: individual web queries may appear benign in isolation, but cumulative query logs allow observers to reconstruct private internal facts without access to internal documents or agent reasoning. This effect creates a hidden privacy risk that standard output audits often miss, per the benchmark’s accompanying MosaicLeaks research paper.
The benchmark uses 1,001 multi-hop research chains that interleave local enterprise document queries with public web searches. These chains are split into 559 training, 98 validation, and 344 held-out test chains drawn from DRBench-style enterprise tasks and the BrowseComp-Plus web corpus. Each chain requires the agent to retrieve a local fact first, then use that fact as a bridge to formulate a subsequent web query, creating explicit dependencies between private internal data and public-facing search terms. This design replicates real-world enterprise research workflows where agents cross-reference internal proprietary data with public sources to complete multi-step tasks.
How does mosaic leakage work in AI research agents?
MosaicLeaks defines three escalating leakage tiers to quantify this risk for AI research agents. The first tier is intent leakage, where an adversary infers the agent’s private research goals from query logs alone. The second is answer leakage, where the log plus a known private question lets an adversary recover the answer without internal docs. The third is full-information leakage, the most severe tier, where an observer can state verifiably true private claims without being prompted with specific questions. Full-information leakage is the hardest to detect via standard output reviews, as it does not require an adversary to know the specific private question in advance.
The agent harness used for testing, adapted from the DRBench benchmark, operates in four iterative steps: Plan, Choose, Read, and Resolve. The Plan tool generates local and web search queries, while the Choose tool selects relevant retrieved documents. The Read tool attempts to answer the current hop from selected documents in parallel, and the Resolve tool decides whether to answer, retrieve more documents, or plan additional searches. Each hop is evaluated individually via normalized string matching to measure both task accuracy and leakage risk, ensuring consistent measurement across all test runs.
What were the MosaicLeaks benchmark results?
Tests across multiple leading AI models found leakage was pervasive across all tested systems. Standard performance-focused training was correlated with higher leakage risk, with a baseline strict chain success rate of 48.7% for research chains where every hop was answered correctly. Strict chain success measures the share of full research chains completed without errors, per the benchmark documentation.
Across all tested models, the combined average rate of answer and full-information leakage was 34.0% of tasks. Full-information leakage, where an adversary could discover private facts without any prompting about what to look for, was widespread. Models routinely included private entity names, internal metrics, and confidential dates in web queries to improve search relevance, creating exploitable mosaic leakage patterns. For example, an agent tasked with cross-referencing internal Q3 sales figures with public market data might include the exact unreleased sales number in a public web query to get more relevant results, exposing that confidential figure to anyone with access to the query log.
How does PA-DR training reduce leakage?
To address the leakage gap, the ServiceNow research team proposed Privacy-Aware Deep Research (PA-DR), a mosaic-leakage-aware reinforcement learning training method. The approach penalizes models for including private information in web queries while rewarding successful task completion. This dual reward structure balances utility and privacy, unlike standard training that optimizes only for task accuracy. The full PA-DR methodology and results are outlined in the official MosaicLeaks research publication.
PA-DR raised strict chain success to 58.7%, a 10 percentage point improvement over the 48.7% baseline. It also slashed the combined answer and full-information leakage rate from 34.0% to 9.9%, a 24.1 percentage point reduction equal to a roughly 71% drop from the baseline rate. For context, this reduction means the model leaks private data in roughly 1 out of 10 tasks under PA-DR training, compared to roughly 1 in 3 tasks under standard performance-focused training.
The PA-DR results highlight a key tradeoff that enterprise AI teams deploying deep research agents currently face. Optimizing agents purely for task accuracy introduces material privacy risk, while targeted privacy-aware training can reduce that risk without sacrificing utility. For teams building agents that access internal documents and external web tools, the findings underscore the need to audit query logs for mosaic leakage, not just final output.
Even seemingly benign individual search queries can expose sensitive internal data when aggregated, as cumulative logs allow observers to reconstruct private facts without access to original internal documents. This means even if an agent’s final output is sanitized, its query history can still expose sensitive enterprise data.
Bottom line: Enterprises deploying deep research agents that mix internal document access with web search must audit query logs for mosaic leakage and adopt privacy-aware training frameworks like PA-DR, as standard performance optimization is correlated with a 34.0% combined answer and full-information leakage rate, compared to 9.9% with PA-DR training.
