Critical Copilot vulnerability allowed hackers to seal 2FA code from users

Last Tuesday, Microsoft addressed a critical security flaw within its M365 Copilot AI platform. This vulnerability, discovered and reported by security researchers, enabled attackers to retrieve sensitive information, such as two-factor authentication (2FA) codes, directly from users' emails accessible to Copilot.

The core issue lies in large language models' (LLMs) inability to differentiate between legitimate user instructions and malicious commands hidden within third-party content that the models process. This fundamental weakness makes it challenging for providers like Microsoft to secure their AI products, forcing them to implement complex, often temporary, safeguards.

Attackers bypassed existing Copilot safeguards, such as those preventing web form submissions or email sending, by embedding malicious instructions in markup language or HTML tags. These methods allowed them to trigger web requests that exposed sensitive data to attacker-controlled servers.

The researchers devised an exploit chain called "SearchLeak." This involved a "Parameter-to-Prompt Injection," where a malicious command was inserted into a URL query parameter. When a victim clicked a specially crafted link, Copilot would extract sensitive data and embed it into an image URL, which was then routed through Bing to the attacker's server.

Microsoft has since patched the vulnerabilities exploited by SearchLeak. However, the underlying challenge of AI models' susceptibility to prompt injection remains. This ongoing battle highlights the continuous need for vigilance and evolving security measures in the rapidly advancing field of artificial intelligence.