Doctored footage and hijacked accounts: anatomy of a deepfake scam network

The Labour party has a plan to “pull British citizens out of the crisis”, says leader Keir Starmer in an advert published recently on Facebook.

The footage looks real. But it is a deepfake – it has been edited and altered using artificial intelligence (AI). The original clip was taken from a speech Starmer gave in Scotland last summer about an energy proposal before being modified using AI technology.

It is one of a number of faked videos of politicians including Starmer and Rishi Sunak that circulated on Facebook and Instagram between 2022 and early 2024 as part of a scam campaign.

They have been manipulated to appear to be revealing details of high-return investments under the brand name Quantum AI. They were published across the world and viewed around one million times in the UK alone, with Meta, the platforms’ parent company, seemingly unable to halt their spread.

The ads raise questions about the ability of tech platforms to curb misinformation in the runup to major elections across Europe. The issue is highlighted in a new investigation by the Bureau of Investigative Journalism (TBIJ) that reveals thousands of scam adverts featuring politicians to have been circulating on Facebook in recent months.

The Quantum AI ads, which also feature public figures such as Boris Johnson, Elon Musk, Jeff Bezos, Angelina Jolie and Ryan Reynolds, appear to have tricked a number of people into handing over their cash.

TBIJ found multiple online reviews posted by people claiming to have lost money to Quantum AI after being fooled by the doctored videos. One said they had been scammed for “hundreds of dollars”, while another said she had lost all of her savings.

Despite numerous ads being blocked by Meta, new ones have continued to be launched and have attracted thousands of views each.

The campaign appears to have been targeted across the world, with videos tailored to different countries. Doctored footage had been drawn from broadcasters in the UK and Ireland, as well as media from Italy, France, Hungary, Poland, Germany and the Netherlands.

In the UK, the scam was presented as a government initiative, while ads targeting EU countries featured the president of the European Commission, Ursula von der Leyen, and appeared to promote “a new investment platform for earning in Europe” called European Partnership. The latter advert included a counterfeit website promising daily returns of €300 for every €300 invested.

As of February this year, a total of 1,200 Meta ads mentioning Quantum AI were found to be active, including those using AI-generated videos.

In the UK, 321 Quantum AI ads listed as concerning social or political issues have were viewed as many as 1.2m times, costing the scammers somewhere between £15,000 and £31,000. But the real numbers are probably much higher: not all Quantum AI’s ads were given this label and TBIJ found identical videos that had been labelled differently.

Scammers also appeared to have hijacked various “verified” Facebook pages, including those of political organisations or figures in Bolivia, India, Mexico and Brazil.

Besides running on Meta platforms, Quantum AI accounts were also identified on TikTok, YouTube, and X, formerly Twitter, which all hosted deepfake videos promoting the scam. All of these platforms ban AI-generated or edited content that misleads the audience.

A Meta spokesperson told TBIJ the company had removed the violating ads, in line with its policies.

They added: “Meta has specialised systems to detect celeb-bait, invests heavily in trained review teams, shares tips on avoiding scams and offers tools to report potential violations. We also work with law enforcement and take legal action.”

A Google spokesperson told TBIJ: “We first detected this scam last year and have taken extensive enforcement action, including immediately suspending multiple advertiser accounts and terminating the associated YouTube channels.”

TikTok told TBIJ it had removed the videos flagged and would be carrying out additional searches for similar content.