of celebrities hurling racial epithets.
Guardrails? What guardrails are there? Naughty internet users found a way of tricking the Sora 2 generator to produce deepfakes that sound like they are spewing racial insults. This includes OpenAI CEO Sam Altman, and billionaire Mark Cuban. The trick works even though Sora has built-in filters that block hateful language.
AI-detection platform Copyleaks reported on that its review of the newly released Sora 2 application, with its improved model for video generation, revealed several videos using celebrity images to recreate an 2020 incident where a man wearing Burger King crowns was kicked off JetBlue flight because he made a racist rant. Instead of the James May lookalike, Sora users recreated this scene by using Altman . Amouranth, IDKSterling, and YouTuber Jake Paul were also used.
Sora 2 Users weren’t able recreate the incident perfectly, but OpenAI’s software has guardrails that prevent the creation content with epithets (i.e. the n-word) used in the original. Copyleaks reports that a homophone can bypass these restrictions and make it appear as if public figures, such as those who have opted in to Sora’s Cameo, were using racist slurs.
Users can upload short clips to be used in Sora’s videos using the so-called Cameo function, which was added last month when Sora 2 was launched. There is no official list of public figures and celebrities in Sora. However, users have created their ownthat includes a numberof dead people and historic figures whose families were not happywith their inclusion.
“To evade platform filters meant to block hate speech, users appeared to prompt Sora with coded or phonetically similar terms—such as ‘knitter’—to generate audio that mimics a well-known racial slur,” Copyleaks said. “The digital double of Altman, for example, screams “Knitters are evil” while being escorted from the plane.”
Copyleaks said that the other videos listed by Copyleaks all use the word “knitters” instead of racist terminology. The Jake Paul videouses the term “neck hurts.” Another example of content that used Paul’s likeness was him explaining that “juice” referred to Jewish people and not liquid pressed out of fruit and vegetables.
- Japan warns OpenAI not to steal its copyrighted animation
- Suetopia Generative AI could be a lawsuit for your business
- AI will soon be hired by the studio of American filmmaker Tyler Perry.
- AI can better imitate famous authors’ styles than MFA students.
“This behavior illustrates an unsurprising trend in prompt-based evasion, where users intentionally probe systems for weaknesses in content moderation,” Copyleaks reported. “When combined with the likenesses of recognizable individuals, these deepfakes become more viral and damaging—spreading quickly across and beyond the platform.”
Copyright concerns aren’t the only thing that concerns people about Sora 2’s ability to distort the truth, damage intellectual property and harm reputations. Copyleaks warned of the dangers of this type of content. “The combination of a familiar face with offensive content is jarring and drives engagement in ways that anonymous or fictional content doesn’t.”
Cuban does not seem to be concerned at all: he has taken to deleting videos when they are posted on Sora. The entrepreneur stated in X. Once they’re exported to the wider web, it’s much more difficult. And they’re already there.
OpenAI did not respond to questions about this story. The videos Copyleaks identified were still available on Sora at the time of writing. (r)
