Making deepfakes becomes cheaper and easier thanks to AI

It wouldn’t be completely out of place for comedian-turned-podcaster Joe Rogan to endorse a “libido-boosting” coffee brand for men.

But when a video circulating on TikTok recently showed Mr Rogan and his guest, Andrew Huberman, peddling the coffee, some eagle-eyed viewers were shocked, including Dr Huberman.

“Yes, that’s wrong”, Dr. Huberman wrote on Twitter after seeing the ad, in which he appears to be praising the testosterone-boosting potential of coffee, even though he’s never done it.

The ad was part of a growing number of fake social media videos made with technology powered by artificial intelligence. Experts said Mr Rogan’s voice appeared to have been synthesized using AI tools that mimic the voices of celebrities. Dr. Huberman’s comments were taken from an unrelated interview.

Making realistic fake videos, often called deepfakes, once required elaborate software to put one person’s face on top of another. But now, many tools for creating them are available to everyday consumers, even on smartphone apps, and often for little or no money.

The new edited videos — mostly, so far, the work of meme creators and marketers — have gone viral on social media sites like TikTok and Twitter. The content they produce, sometimes called cheapfakes by researchers, works by cloning the voices of celebrities, altering mouth movements to match an alternate sound, and writing persuasive dialogue.

The videos and the accessible technology that underpins them have some artificial intelligence researchers worried about their dangers and have raised new concerns about whether social media companies are prepared to moderate growing digital fraud.

Disinformation watchdogs are also bracing for a wave of digital fakes that could mislead viewers or make it harder to know what’s true or false online.

“What’s different is that anyone can do it now,” said Britt Paris, an assistant professor of library and information science at Rutgers University, who helped coin the term “cheapfakes.” “. “It’s not just about people with sophisticated computer technology and fairly sophisticated computer know-how. Instead, it’s a free app.

Tons of manipulated content has circulated on TikTok and elsewhere for years, usually using more artisanal tricks like careful editing or swapping one audio clip for another. In a video on TikTok, Vice President Kamala Harris appeared to say that all people hospitalized with Covid-19 were vaccinated. In fact, she said the patients weren’t vaccinated.

Graphika, a research firm that studies disinformation, spotted deepfakes of fictitious news anchors that pro-China bot accounts distributed late last year, in the first known example of the technology being used. for state-aligned influence campaigns.

But several new tools are bringing similar technology to everyday internet users, giving comedians and fans the ability to make their own compelling parodies.

Last month, a fake video circulated showing President Biden declaring a national plan for war between Russia and Ukraine. The video was produced by the team behind “Human Events Daily”, a podcast and live stream run by Jack Posobiec, a right-wing influencer known for spreading conspiracy theories.

In a segment explaining the video, Posobiec said his team created it using AI technology. A tweet about the video from The Patriot Oasis, a conservative account, used a breaking news tag without indicating that the video was fake. The tweet has been viewed more than eight million times.

Many music videos featuring synthesized voices appeared to use technology from ElevenLabs, an American start-up co-founded by a former Google engineer. In November, the company launched a speech cloning tool that can be trained to reproduce voices in seconds.

ElevenLabs came to attention last month after 4chan, a message board known for its racist and conspiratorial content, used the tool to share hate messages. In one example, 4chan users created an audio recording of anti-Semitic text using a computer-generated voice that impersonated actor Emma Watson. Motherboard earlier reported 4chan’s use of audio technology.

ElevenLabs said on Twitter that it introduce new guarantees, such as limiting voice cloning to paid accounts and providing a new AI detection tool. But 4chan users said they would create their own version of the voice-cloning technology using open source code, releasing demos that sound like audio produced by ElevenLabs.

“We want to have our own custom AI with the power to create,” an anonymous 4chan user wrote in a post about the project.

In an email, a spokeswoman for ElevenLabs said the company was looking to collaborate with other AI developers to create a universal detection system that could be adopted across the industry.

Videos using cloned voices, created with ElevenLabs’ tool or similar technology, have gone viral in recent weeks. One, posted on Twitter by Elon Musk, the site’s owner, showed a fake profane conversation between Mr. Rogan, Mr. Musk and Jordan Peterson, a Canadian men’s rights activist. In another, posted on YouTube, Mr Rogan appeared to interview a fake version of Canadian Prime Minister Justin Trudeau about his political scandals.

“Producing such forgeries should be a felony with a mandatory ten-year sentence,” Mr Peterson said in a tweet about the fake videos featuring his voice. “This technology is dangerous beyond belief.”

In a statement, a YouTube spokeswoman said the video of Mr. Rogan and Mr. Trudeau did not violate the platform’s policies because it “provides sufficient context”. (The creator had described it as a “fake video.”) The company said its misinformation policies prohibit deceptively doctored content.

Experts studying deepfake technology have suggested that the fake ad featuring Mr Rogan and Dr Huberman was most likely created with a voice cloning program, although the exact tool used is unclear. Mr. Rogan’s audio was integrated into a real interview with Dr. Huberman discussing testosterone.

The results are not perfect. The clip of Mr Rogan was taken from an unrelated interview published in December with Fedor Gorst, a professional pool player. Mr. Rogan’s mouth movements do not match the audio and his voice sounds unnatural at times. If the video won over TikTok users, it was hard to say: it got a lot more attention after being flagged for its impressive counterfeit.

TikTok’s policies prohibit digital forgeries “that mislead users by distorting the truth of events and cause significant harm to the subject of the video, to other people, or to society.” Several of the videos were deleted after The New York Times reported them to the company. Twitter also removed some of the videos.

A TikTok spokesperson said the company uses “a combination of technology and human moderation to detect and remove” manipulated videos, but declined to elaborate on its methods.

Mr. Rogan and the company featured in the false advertisement did not respond to requests for comment.

Many social media companies, including Meta and Twitch, have banned deepfakes and manipulated videos that mislead users. Meta, which owns Facebook and Instagram, held a competition in 2021 to develop programs that could identify deepfakes, resulting in a tool that could spot them 83% of the time.

Federal regulators have been slow to respond. A 2019 federal law required reporting on the weaponization of deepfakes by aliens, required government agencies to notify Congress if deepfakes targeted elections in the United States, and created an award to encourage research into tools capable of detecting fraud. deepfakes.

“We can’t wait two years for laws to be passed,” said Ravit Dotan, a postdoctoral researcher who leads the Collaborative AI Responsibility Lab at the University of Pittsburgh. “By then, the damage could be too great. We have elections coming up here in the United States. It’s gonna be a problem. »