As a reminder to be cautious of AI browsers in general, researchers at LayerX discovered a vulnerability that allows attackers to inject malicious instructions into ChatGPT’s memory by using cross-site requests forgery.
This exploit is dubbed ChatGPT tainted memories by LayerX’s browser security researchers, who discovered and disclosed the security gap to OpenAI. It involves a certain level of social engineering, as it requires the user to click a malicious link. It is also a risk for ChatGPT users using any browser, not just Atlas which is OpenAI’s new AI-powered browser launched last week on macOS.
According to LayerX CEO and co-founder Or Eshed, it’s particularly dangerous for Atlas users. Atlas users are usually logged into ChatGPT by their default settings, which means that their authentication tokens can be abused while an active session is in progress. Plus, “LayerX testing indicates that the Atlas browser is up to 90 percent more exposed than Chrome and Edge to phishing attacks,” Eshed stated on a Monday blog. OpenAI
did not respond immediately to questions from The Register () about the attack and LayerX’s research. We will update the story once we hear from the AI giant.
This attack exploits a cross-site forgery vulnerability. It takes advantage of a user’s current session on a site and forces the browser to send a malicious request. This request is processed by the site as if it came from the authenticated user. In this case, it allows an attacker to access OpenAI systems the user is already logged into and then injects malicious instructions.
The attack also involves infecting ChatGPT’s built-in Memory feature which allows the chatbot “remember” user’s queries, chats and preferences and reuse them throughout future chats. It then involves injecting hidden instructions to ChatGPT’s memory by using cross-site requests forgery. Eshed wrote
“Once an account’s memory has been infected, this infection is persistent across all devices that the account is used on – across home and work computers, and across different browsers – whether a user is using them on Chrome, Atlas, or any other browser,” .
“This makes the attack extremely ‘sticky,’ and is especially dangerous for users who use the same account for both work and personal purposes,” He added. Here’s how it works:
- A user logs in to ChatGPT.
- A malicious link is used to trick the user into clicking it, most likely through phishing or social engineering. The link then directs them onto a compromised website. In this example, the “Please check out this cool GPT prompt” is a message posted in a Discord channel for vibe coding.
- This initiates a cross-site forgery attack which abuses the existing authentication credentials of the user.
- This request injects hidden instructions in ChatGPT’s Memory without the user’s awareness.
- When the user queries ChatGPT again, it “remembers” these malicious instructions and executes them.
The LayerX proof-of concept is not very malicious. The hidden prompt instructs the chatbot that it should create a Python script that detects if the user’s device is connected to their home WiFI network and then play “Eye of the Tiger.”
However, this same technique could also be used to install malware, steal data or give the attacker complete control over the victim’s system. Eshed says that the risk is greater for those who use AI-based browsers. Atlas is one of the more powerful ones. OpenAI Atlas ignores the inevitable prompt injection and releases AI browser
LayerX conducted 103 real-world phishing and web vulnerabilities tests against traditional browsers such as Chrome and Edge as well
Edge stopped these attacks 53% of the time in these tests. This was similar to Chrome, Dia, and Genspark at 7 percent. Atlas, on the other hand, only blocked 5.8 percent malicious web pages. LayerX claims that Atlas users are 90 per cent more vulnerable to phishing than people using other browsers.
The new exploit is a follow-up to a prompt injection attack against Atlas, demonstrated in a demonstration by NeuralTrust. Researchers disguised a malicious prompt with a harmless URL. Atlas treated these hidden instructions like high-trust “user intent” texts, which can be exploited to trick the AI browser and enable harmful actions.
Like the LayerX PoC the NeuralTrust relies on social engineering. Users must copy and paste a fake URL into Atlas “omnibox,” where they enter URLs or search terms.
But immediately after OpenAI’s Atlas was released, researchers demonstrated that it is easy to trick the AI browser by using indirect prompt injection attacks. (r)
