Chinese official's ChatGPT use accidentally exposes top secret global operation
An AI 'diary' mistake sheds light on a hidden online campaign
Chinese influence operations are often associated with slick propaganda and coordinated social media pushes, but this time, a suppression campaign has been unexpectedly revealed because of using AI in not exactly the best way.
A new report from OpenAI has revealed that a Chinese law enforcement official used ChatGPT like a diary to document an alleged covert operation. According to the report, it was a sprawling campaign aimed at Chinese dissidents living overseas, carried out through thousands of accounts across multiple platforms.
As reported by CNN, the activity included impersonating US immigration officials. In one instance described by the ChatGPT user, Chinese operators allegedly disguised themselves as US immigration officials to warn a US-based Chinese dissident that their public statements had ‘supposedly broken the law’.
Advert
The ChatGPT user also described an effort to use forged documents from a US county court to try to get a Chinese dissident’s social media account taken down. OpenAI said the influence operation appeared to involve hundreds of Chinese operators and thousands of fake online accounts on various social media platforms.
Ben Nimmo, principal investigator at OpenAI, told reporters ahead of the report’s release: “This is what Chinese modern transnational repression looks like…It’s not just digital. It’s not just about trolling. It’s industrialized. It’s about trying to hit critics of the CCP [Chinese Communist Party] with everything, everywhere, all at once.”
OpenAI said ChatGPT served as a journal for the Chinese operative to keep track of the covert network, while ‘much of the network’s content was generated by other tools’ and spread through social media accounts and websites. The company said it banned the user after discovering the activity.
OpenAI’s investigators said they were able to match descriptions from the ChatGPT user with real-world online activity and impact. The user described an effort to fake the death of a Chinese dissident by creating a phony obituary and photos of a gravestone and posting them online. OpenAI said false rumours of the dissident’s death did surface online in 2023.
In another case, the ChatGPT user asked the AI agent to draw up a multi-part plan to denigrate the incoming Japanese prime minister, Sanae Takaichi, in part by fanning online anger about US tariffs on Japanese goods. ChatGPT refused to respond to the prompt, according to OpenAI.
In late October, though, as Takaichi took power, OpenAI said hashtags emerged on a popular forum for Japanese graphic artists attacking her and complaining about US tariffs.
The report lands amid intensifying competition between the US and China over AI capabilities and deployment. Horowtiz, who is now a professor at the University of Pennsylvania, said: “US-China AI competition is continuing to intensify…This competition is not just taking place at the frontier, but in how China’s government is planning and implementing the day-to-day of their surveillance and information apparatus.”