Predicting model behavior before release by simulating deployment

Ensuring the safety and responsible behavior of AI models before release is paramount, especially as their capabilities grow. Traditional evaluation methods, while valuable, often rely on synthetic or hand-picked prompts that may not fully capture real-world usage. This can lead to a limited understanding of how a model might behave in diverse, complex interactions.

Deployment Simulation addresses these limitations by replaying privacy-preserved past conversations with a new candidate model. This allows researchers to observe how the model responds in realistic contexts, identifying emerging undesired behaviors and estimating their frequency before the model reaches users. This approach offers a deployment-like preview, providing complementary insights to traditional red-teaming and targeted evaluations.

The technique has been successfully applied across multiple GPT-5 series deployments, significantly improving estimates of undesired behavior rates and surfacing novel forms of misalignment. It has also proven effective in evaluating complex agentic rollouts involving tool use, extending its utility beyond standard chat applications to more sophisticated AI systems.

By proactively identifying blind spots in traditional evaluations and informing mitigation strategies, Deployment Simulation plays a crucial role in the model development process. It helps validate pre-deployment forecasts and ensures a more comprehensive understanding of model behavior under realistic conditions.