OpenAI bans Chinese accounts using ChatGPT to edit code for social media surveillance

OpenAI has banned the accounts of a group of Chinese users who had attempted to use ChatGPT to debug and edit code for an AI social media surveillance tool, the company said Friday. The campaign, which OpenAI calls Peer Review, saw the group prompt ChatGPT to generate sales pitches for a program those documents suggest was designed to monitor anti-Chinese sentiment on X, Facebook, YouTube, Instagram and other platforms. The operation appears to have been particularly interested in spotting calls for protests against human rights violations in China, with the intent of sharing those insights with the country's authorities.

"This network consisted of ChatGPT accounts that operated in a time pattern consistent with mainland Chinese business hours, prompted our models in Chinese, and used our tools with a volume and variety consistent with manual prompting, rather than automation," said OpenAI. "The operators used our models to proofread claims that their insights had been sent to Chinese embassies abroad, and to intelligence agents monitoring protests in countries including the United States, Germany and the United Kingdom."

According to Ben Nimmo, a principal investigator with OpenAI, this was the first time the company had uncovered an AI tool of this kind. "Threat actors sometimes give us a glimpse of what they are doing in other parts of the internet because of the way they use our AI models," Nimmo told The New York Times.

Much of the code for the surveillance tool appears to have been based on an open-source version of one of Meta's Llama models. The group also appears to have used ChatGPT to generate an end-of-year performance review where it claims to have written phishing emails on behalf of clients in China.

"Assessing the impact of this activity would require inputs from multiple stakeholders, including operators of any open-source models who can shed a light on this activity," OpenAI said of the operation's efforts to use ChatGPT to edit code for the AI social media surveillance tool.

Separately, OpenAI said it recently banned an account that used ChatGPT to generate social media posts critical of Cai Xia, a Chinese political scientist and dissident who lives in the US in exile. The same group also used the chatbot to generate articles in Spanish critical of the US. These articles were published by "mainstream" news organizations in Latin America and often attributed to either an individual or a Chinese company.

This article originally appeared on Engadget at https://www.engadget.com/ai/openai-bans-chinese-accounts-using-chatgpt-to-edit-code-for-social-media-surveillance-230451036.html?src=rss https://www.engadget.com/ai/openai-bans-chinese-accounts-using-chatgpt-to-edit-code-for-social-media-surveillance-230451036.html?src=rss
Created 1mo | Feb 21, 2025, 11:20:07 PM


Login to add comment

Other posts in this group

iOS 18.4 is available now with new emoji, Apple News+ Food and priority notifications

Apple has released iOS 18.4, iPadOS 18.4, macOS 15.4 and vision

Apr 1, 2025, 12:50:09 AM | Engadget
iOS 18.4 is available now with new emoji, Apple News+ Food and priority notifications

Apple has released iOS 18.4, iPadOS 18.4, macOS 15.4 and vision

Mar 31, 2025, 8:20:18 PM | Engadget
Alexa+ is out, but missing a lot of features Amazon demoed last month

If there’s one thing you can count on with AI tools is that they’ll launch without most of the promised features. It happened with

Mar 31, 2025, 8:20:17 PM | Engadget
WD's 1TB C50 Xbox storage card is 30 percent off in the Amazon Spring Sale

The only real way to increase the storage of an Xbox Series X/S is to use a specialized expansion card. Luckily, a really good one is heavily discounted right now. The WD 1TB C40 storage card

Mar 31, 2025, 8:20:16 PM | Engadget
Substack updated its app to make it even more like TikTok

Substack is taking one more step towards becoming an all-in-one social media / creator platform by adding a vertical video feed to its app. Technically, the new TikTok-style feed is a redesign of t

Mar 31, 2025, 8:20:16 PM | Engadget