MosaicLeaks: Can your research agent keep a secret?

Deep research agents, which combine private local documents with external web tools, inherently create a privacy risk. This "mosaic effect" occurs when an agent's external queries, though seemingly innocuous individually, collectively reveal sensitive private information to observers monitoring outbound traffic. For instance, a healthcare firm's research agent might issue web searches that, when pieced together, reveal a private cloud-migration milestone to an adversary.

MosaicLeaks, a new deep-research task, aims to quantify this privacy leakage. It utilizes multi-hop questions that interleave public and private information, with web queries serving as the leakage channel. Leakage is measured across three increasing levels of concern: intent leakage (what the agent is investigating), answer leakage (enough information to answer a private question), and full-information leakage (discovering and stating private facts without prior knowledge).

Notably, initial findings show that training agents solely for task performance exacerbates the leakage problem. When strict chain success rose from 48.7% to 59.3%, answer/full-information leakage also increased significantly from 34.0% to 51.7%. This indicates a trade-off where better task performance can lead to more context-rich—and thus more revealing—web queries.

To address this, MosaicLeaks proposes Privacy-Aware Deep Research (PA-DR), an RL training method designed to mitigate leakage. This method successfully raises strict chain success from 48.7% to 58.7% while substantially reducing answer/full-information leakage from 34.0% to 9.9%. This demonstrates that it is possible to improve agent performance while simultaneously enhancing privacy protection.