Disrupting a covert Iranian influence operation

https://openai.com/index/disrupting-a-covert-iranian-influence-operation/

by saikatsg • 2 years ago

125 127 2 years ago

I wonder if it would be possible to get a list of countries that can have influence operations using ChatGPT and countries that can't.

janice1999• 2 years ago

Russia, China, Iran, and Israel

https://www.npr.org/2024/05/30/g-s1-1670/openai-influence-op...

ein0p3• 2 years ago

This doesn’t pass the sniff test. Each of these countries has domestically developed, strong LLMs. And even if they did not, they’d have absolutely no issue running the larger FOSS ones. You don’t need GPT4 to generate political drivel.

wslh• 2 years ago

Those four results only show the metapropaganda limitations. There are more countries in the world that are less "fancy" and included in the list. Weird that you don't include America though. BTW, your comment reminds me this very basic espionage issue [1].

[1] https://www.reddit.com/r/todayilearned/comments/bdjh8o/til_t...

lancesells3• 2 years ago

I don't think that would matter. I live in the US and I would guess there are plenty of people in this country that work for countries that aren't allowed to use it. Same goes for other countries having people who work for the US.

I really don't think it would take that many people to run propoganda campaigns with modern tools.

hobo_in_library2• 2 years ago

Only countries with dedicated PsyOps divisions are allowed to use ChatGPT for influence operations, obviously

https://en.wikipedia.org/wiki/Psychological_operations_(Unit...

colordrops2• 2 years ago

I hope they put the same restrictions on Israel but I doubt it. Multiple core OpenAI team members have expressed pro-Israel comments, some very murderous and ugly, so I doubt their ability to be unbiased here.

jjmarr• 2 years ago

https://www.nbcnews.com/tech/security/meta-openai-say-disrup...

https://openai.com/index/disrupting-deceptive-uses-of-AI-by-...

OpenAI has also banned Israeli influence operations. Do you think the above isn't going far enough?

It's also strange that you call out all pro-Israel comments as a cause for concern.

vkou• 2 years ago

Why stop at countries? Is it possible to get a list of organizations that are running influence, propaganda, and astroturfing operations using ChatGPT, and its ilk?

trompetenaccoun• 2 years ago

We could start with OpenAI itself. The raw output of the models is clearly manipulated by the company, leading to politically biased results. Which many of their users naively take as 'truth' because they think it's artificial intelligence.

akira25012• 2 years ago

Wow. What a high value target. [0]

I mean. I get that low rent actors will use low rent services to try to generate political garbage. Is there any evidence that this is actually having a measurable or meaningful impact?

Who exactly is fooled by these sites? And is it the sites that are the problem or the relative lack of sophistication in American education when it comes to political corruption?

[0] https://niothinker.com/

Maken2• 2 years ago

My guess is that these sites are used to generate headlines that will be then shared by bots. Most people will never get pass that headline, the site is there to create the illusion of a real news source for those few who actually click the headline (but will probably never read the entire article because TL;DR). These healines are designed to reinforce certain bias and keep people inside their echo chambers, and that does have an impact.

akira25012• 2 years ago

We don't have to guess. We can precisely know the reach of these articles. Aside from this I wonder what value does GPT have here? It seems like it's saving "the covert Iranians" (how covert can you be if an American business points the finger right at you?) on the order of a few dollars an hour.

It's hard to take seriously. On any level.

katzinsky• 2 years ago

Yeah. Knowing my tools are judging my political positions and could self destruct if the authors disagree just makes me love using my computer.

This is why I only use local models these days.

EDIT: Out of posts for today but I've been pretty happy with Gemma2. The context is short but the performance is very good and it's easy to disable refusal.

guerrilla• 2 years ago

Just curious, which local models?

tonetegeatinst• 2 years ago

I know the Facebook model was good. Checkout r/locallamma for guidance, or check the ranking website.

lambdaba• 2 years ago

The only thing noteworthy about this is how small-scale this is, and that the perpetrators don't even bother/have the means to set up their own infrastructure.

chatmasta• 2 years ago

The noteworthy thing is that OpenAI is reporting it. They’re signaling that they are proactively monitoring and investigating this activity, and that they’re willing to work with federal agencies while self-policing their negative externalities.

This is all part of an ongoing conversation with lobbyists about “safe AI,” and it’s ultimately done to show that OpenAI is making an effort to mitigate the risks that regulators claim it creates.

But there’s also another signal, which is what they’re not broadcasting: “ChatGPT can be used for propaganda, it works in Persian too, and we’re happy to sell to the DoD.”

lancesells3• 2 years ago

It's a good thing for them to report this, but for a company that decided against watermarking their output they are kind of complicit.

I know they might lose their teens and college demographic if they do it, but if it's truly this world-changing tool that they claim than not watermarking is scraping the serial number off the guns they're selling.*

* Maybe a shitty analogy

bfLives• 2 years ago

and it’s ultimately done to show that OpenAI is making an effort to mitigate the risks that regulators claim it creates.

I’d go one step further — the subtext is that only Open AI is willing to do this, and that LLMs are just too dangerous to open source/not another with regulation that only big companies have the resources to adhere to.

programmarchy• 2 years ago

The linked PDF (Storm-2035 [1]) from Microsoft is more detailed and interesting than the blog post. However, what's missing from the reports is how they detected those operations and how they tied them to different groups. There's a lot of claims being made without showing all of the supporting evidence.

To give them the benefit of the doubt, they likely want to keep their detection methods secret to make circumvention more difficult. And it all sounds totally plausible of course. But at the same time, a degree of skepticism is warranted because Microsoft has a huge incentive to fearmonger about AI so they can lock it down and capture the market. And what better way is there than to use the usual bogeymen to do so.

[1] https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcor...

com2kid• 2 years ago

Having worked at Microsoft for almost a decade, I remember chatting with their security people plenty after meetings. One interesting thing I learned is that Microsoft (and all the other top tech companies presumably) are under constant Advanced Persistent Threat from state actors. From literal secret agents getting jobs and working undercover for a decade+ to obtain seniority, to physical penetration attempts (some buildings on MS campus used to have armed security, before Cloud server farms were a thing!).

Microsoft is one of the few companies that goes toe to toe with world governments every day of the year.

And I imagine balancing that next to all the engineers who demand admin access everywhere is a royal pain!

Although the best Government VS Business story I heard was during intern orientation at Boeing about French agents breaking into Boeing employee's hotel rooms during a conference in France while the employees were out to dinner, and going through laptops. One of the employees returned earlier than expected, and the men in suits shut the laptop, turned around and walked out of the room w/o saying anything!

BoingBoomTschak• 2 years ago

Microsoft is one of the few companies that goes toe to toe with world governments every day of the year.

It's also the company which was the first and longest member of PRISM, meaning very deep ties to the less savory parts of the US gov and {five,nine,fourteen} eyes. I know it's a boring advice, but I'd take this kind of declaration with a truckload of salt.

defamation3• 2 years ago

I remember a blogpost from OpenAI maybe a year ago that went into great detail about how they find these bad actors.

greatgib2• 2 years ago

What they don't say on their post but that we can guess and is from interest is that they probably had to spy their user messages to determinate that they used the account for generate content for the influence operation.

For sure the purpose is noble, but it is good to remind everyone that everything you type, submit or generate there is not private but could be randomly snooped by strangers!

szundi• 2 years ago

Naah, probably they used OpenAI apis to read, summarize and categorize everything

TOMDM• 2 years ago

That or they could store hashes of chunks of output to compare to propaganda in the wild.

greatgib2• 2 years ago

It would be very costly and nothing say that the bad actor is using the message as-is and not a subset.

simonw2• 2 years ago

From a Google search it looks like this is one of the articles in question: https://teorator.com/index.php/2024/08/12/x-censors-trumps-t...

I requested the Internet Archive grab this copy: https://web.archive.org/web/20240816210620/https://teorator....

throw310822• 2 years ago

Looks exceedingly good to be Ai-generated. Also, it appears to be slightly pro-Trump, which doesn't sound like in Iran's interest.

stickfigure3• 2 years ago

It is clear to me that Russia, Iran, North Korea are playing a zero-sum game. Anything that creates chaos in US politics (or European politics) is in their interest. Donald Trump represents chaos.

simonw2• 2 years ago

From the article:

Some of the X accounts posed as progressives, and others as conservatives.

My impression is that these influence campaigns know that they need to produce a LOT of content that appears to reflect different opinions and perspectives if they're going to appear to be "real" - that way they can build trust with an audience before attempting to influence them in one direction or another.

lukev• 2 years ago

Really? Trump has always been much weaker in diplomacy with nations that are historically antagonistic to the US. He'll have a meeting with their leader, let them flatter him, and come away convinced that they're "very nice" and make a lot of concessions.

He's basically an admirer of Putin at this point.

defamation3• 2 years ago

They had 5 or 6 websites that posted constant anti-trump articles according to the Microsoft report linked in the OpenAI blogpost.

https://i.imgur.com/SsMoJTv.png

jameshart2• 2 years ago

Looks more like early phase 'reputation building' content aimed at creating channels through which they can direct more inflammatory content later. AI is good for producing large volumes of bland material, which might be useful as camouflage.

irthomasthomas3• 2 years ago

It may be trying to get Trump supporters on their side before trying to influence them

joe_the_user• 2 years ago

I don't think people should jump to any "this is the level Iran is at?" conclusions.

Many nations employ "patriotic citizens" in an informal and semi-formal fashion along with trained propaganda and "infowar" experts. I know of China and Israel doing this but I'd assume it's everywhere.

det2x• 2 years ago

Iran is definately putting out mass propaganda especially on social media.

Israel has one of if not the largest propaganda/bot campaigns on the entire planet, followed by either Russia or China. It's definately working for them so they won't be stopping anytime soon.

djaouen2• 2 years ago

This is as much an indictment of ChatGPT as it is of the Iranians. According to OpenAI, their product produces output that no one in their right mind would want to read for any purpose.

echoangle2• 2 years ago

How is that a bad thing for OpenAI though? It depends on the prompting, I wouldn’t count the ability to generate useless/stupid/misleading content when prompted to do this as a negative.

djaouen2• 2 years ago

I think the point is, nobody likes slop. The fact that slop failed to promote the enemy doesn't mean that slop is inherently good.

hobo_in_library2• 2 years ago

It's hard to take an article like this at face value when they provide zero evidence for any of their claims.

This is coming soon after Trump decided to accuse Iran of being behind his assassination attempt (done by a white 20 year old) and Israel literally assassinated Hamas's chief negotiator while he was visiting Iran.

It seems like the powers that be are desperate for a war with Iran and will continue beating the drum to build consent.

Reminds me of the build up to the 2003 Iraq invasion (you know, because "they have WMDs")

usefulcat• 2 years ago

I'd be more interested in an analysis of the likely intention of the campaign. Is it just an attempt to reduce voter turnout? If so, that doesn't seem all that useful by itself.

dfhvneoieno• 2 years ago

Iran wants Democrats, Russia wants Republicans.

It's easy to see which one is more successful. Every time you blame something on Russia, people come out of the woods "explaining" why Russia is actually the victim (Russia is ALWAYS the victim, is what I've learned from this people).

epolanski• 2 years ago

Russia definitely has a non zero amount of people supporting or apologizing it's actions all around the world.

It's dangerous to assume all of the content you see is simply some shady operation or fake, having controversial opinions on any topic is nothing new, millions of Americans don't believe in the moon landings.

In general I don't think it's a good idea to discuss politics online, and I feel like many talk too much geopolitics but can't even tell the policies or programs of their own city mayor candidates which are way more important for their own lives and directly impacting.

beepbooptheory• 2 years ago

But if you read the post, you'd learn these bots are writing lots of anti Democrat sentiment.

Is there not anything else that comes to your mind right now that Iran cares about and wants to influence?

PerilousD• 2 years ago

These jokers seem like the AI version of "script kiddie" hackers, and OpenAI may be engaging in a bit of humble bragging. It doesn't take considerable investments in time or money to run local LLMs, INCLUDING ChatGPT, where your questions, prompts, and results are not sent home to the mothership, so it's a BS article as to (the real) actors who may or may not be doing this. NOW, if OpenAI or Gemini or LLama, etc, showed how they analyzed social media posts and flagged the ones that were AI generated and the analysis as to WHY the article is flagged, then that would be much more useful, actionable by at least some of the readers and would put the accounts spreading the content (particularly the rebroadcast fluffers) in the spotlight.

Arainach• 2 years ago

It wouldn't be useful at all, and would only serve to educate malicious actors how to better evade detection.

It's like claiming a search engine open sourcing its ranking algorithm would help people be informed instead of making spammers able to perfectly hijack all the results.

navaed013• 2 years ago

Amen

commandpaul• 2 years ago

Given that any self hosted open source model would have worked just as well. I can’t see this good faith post as anything more than forwarding open ais long campaign for regulatory capture.

dtquad2• 2 years ago

Self-hosting LLMs is expensive at scale. It's cheaper to use VC subsidized model inference like the OpenAI APIs.

CyberDildonics• 2 years ago

At "scale"? At what scale?

littlestymaar• 2 years ago

There are plenty of VC-subsidized inference provider which uses open source LLM for much cheaper than OpenAI (which isn't really VC-subsidized at this point but Microsoft-subsidized).

refulgentis• 2 years ago

My anecdata is most teams I've talked to say its below OpenAI at scale, and vLLM is a beast. It's interesting to hear the opposite, there's lots of cheaper providers, but the "VC dollars" argument can go "turtles all the way down", I suppose. Still, reality seems to differ.

ComplexSystems3• 2 years ago

Why wouldn't Iran just use Llama or something?

defamation3• 2 years ago

they might be using that too, it's that OpenAI can detect and catch those using their service.

Cmon mate.

swat535• 2 years ago

Of course they are.. This is just a marketing piece by OpenAI

rvnx• 2 years ago

The headline should be "OpenAI publicly admits it supported Iran influence operation despite the sanctions"

dj_gitmo• 2 years ago

They supported it by banning them? That doesn’t make much sense to me.

FactKnower69• 2 years ago

Main takeaway is that OpenAI is reading all of your prompts :)

sweeter• 2 years ago

CNN just did a piece on private Israeli groups doing the exact same thing. It is pretty scary the shear scale of this. Literally any entity or group can spin up a ton of bots and use any AI service locally or otherwise, to attempt to sway public opinion.

ants_everywhere4• 2 years ago

Reddit especially is full of LLMs. I would imagine there are ones from every major country plus several privately funded ones.

theGnuMe• 2 years ago

they are here as well.

suby• 2 years ago

Yeah, all of the major subreddits are gamed, especially subs like r/politics. Even going back to the 2016 election it was clear that much of the activity on there was not genuine.

firesteelrain2• 2 years ago

Spend enough time in r/politics and you’ll notice.

What’s troubling is the constant spamming in new subreddits like r/inthenews. Once it takes hold (whatever the message), then actual humans start to echo it over and over again.

It’s scary.

TaylorAlexander2• 2 years ago

It’s frustrating too seeing some people in AI say the technology has no negative implications. Of course it does. Foreign actors are using it to influence elections. But when Jeremy Howard was asked if he had done impact assessment of a large open source model release, he replied (roughly) “No. Impact is about the same as a new release of pencil and paper.”

Which… actually that’s not true. Iran isn’t using pencil and paper to influence US elections, they’re using LLMs. And if all the big cloud providers cut them off, they’ll move to using home brew models. That is the impact and putting our head in the sand about it isn’t going to be helpful.

This is NOT to say we should not release to models! I think we should release open models, and that in addition to that we should also care about written impact assessments so everyone is on the same page about risks! I think it’s great that Jeremy Howard is releasing open foundation models, but I wish he wasn’t so flippant about the real world implications. I’ve always looked up to him for the work he has done, and found that attitude very disappointing.

CatWChainsaw• 2 years ago

Also frustrating to be constantly patronized that the negative consequences will definitely be outweighed by positive ones, pinky promise. Right now we have threats of job insecurity in basically every industry, Noah's Arc levels of spam, the undermining of democracy as a functioning form of government, and insane energy demands and environmental consequences. But on the plus side you can outsource the entire design team to one guy tweaking SD outputs and you can use an LLM to summarize the spam generated by a different LLM, so all's well that ends well I guess.

zitterbewegung2• 2 years ago

There is a pro Russian bot that someone used prompt injection to prove it was a bot.

https://www.reddit.com/r/interestingasfuck/comments/1dlg8ni/...

userbinator• 2 years ago

How do you distinguish a real bot, from a human pretending to be a bot?

ComplexSystems3• 2 years ago

Fortunately they seem to be using centralized services such as OpenAI's API, where they can get caught, rather than spinning up their own Llama instances.

pessimizer• 2 years ago

Nothing like 900 million articles about secret Iranian plots from government entangled dystopian megacorps and the pundits who love them, soon after the US starts moving troops into the Middle East to defend the progress of an ongoing genocide.

I suppose this is a great opportunity for the people whose entire income comes from the fact that the US overpays insiders to supply its own military; if Trump wins, he gets to play tough and have the papers start his term praising him to the skies for starting a war with Iran, he'll also be able to blame everything on the last administration and work closely with the people who replace Netanyahu. If Harris wins, she's made no promises and has no beliefs, and will be aided by the press in blaming Gaza on Iran.

OpenAI itself is surely running larger covert influence operations in order to affect US legislation and elections.

Similar to the covert influence operations we reported in May, this operation does not appear to have achieved meaningful audience engagement. The majority of social media posts that we identified received few or no likes, shares, or comments. We similarly did not find indications of the web articles being shared across social media.

Sounds like Russian Facebook ads.

TiredOfLife• 2 years ago

US is going to protect Hamas?

irthomasthomas3• 2 years ago

speaking of influence operations, the strawberry dude and 'lily' just did one of those twitter voice group chat things where everyone tried to guess if lily was an ai or not. there just happened to be a worldcoin rep in the room...

navaed013• 2 years ago

Can you expand?

irthomasthomas3• 2 years ago

Strawberry is alleged to be a codename for some secret openai technology. A twitter account/s that keeps spamming strawberries and making cryptic prophesies was shared by openai and sama. Lots of openai staff and connected people also spammed twitter with strawberries, making it look like an official campaign, or like something was really about to be announced. The same account co-hosted a chat with someone called lily who some claimed was an ai.

worldcoin is a private company owned by sama that verifies identity by scanning the eyeball. He's been rolling out the technology in places like South Africa, where they also recruited people to train openai models by RLHF etc.

It seems it has since been confirmed a scam by some of the participants.

This chap looks to have more details on it https://twitter.com/BenjaminDEKR/status/1824676550379192324

kjellsbells• 2 years ago

One area that OpenAI did not comment on, was the likelihood that the AI-generated content here was itself used in the training data for a later model.

Let's say the OpenAI engineers are working on ChatGPT 5 and spent last month scraping Teorater and X/Twitter, where this material ended up. How does OpenAI know that the new model is not poisoned?

This isn't just OpenAI's problem of course. Anyone training on the open Internet now has this problem.

Arrath• 2 years ago

This isn't just OpenAI's problem of course. Anyone training on the open Internet now has this problem.

'Low-background Steel' has been a nerd trivia tidbit for a while now, will we have to consider a 'Pre' and 'Post' AI internet era for training uses?

nolroz• 2 years ago

Maybe one of the uses of the generated content detection method that OpenAI supposedly has is they can filter at least some of it out?

ethbr1• 2 years ago

Given recent advancements, buying Twitter solely for its use as a data source seems prescient, given only the first party has additional metadata to identify bot replies.

... if the actual purchase hasn't been such a cluster$&@$ so as to thoroughly disabuse the notion there was some master plan.

navaed013• 2 years ago

Its great that openAI is using this infraction as an opportunity to posture about how open they are and how they are a company that can tame evil applications of AI, while totally missing and not addressing the broader concern, what if this was run on a local instance?? How are we stopping / spotting and squatting that?!

Side note. This is some pretty terrible propaganda. The post about Kamala, immigrants and climate change barely makes any sense.

X just hosted trump for a live stream, who is being affected by a headline that reads ‘X censors trump tweet”.

bangaladore• 2 years ago

"...We ran these images through our DALL·E 3 classifier, which identified them as not being generated by our services..."

I'd be shocked if Twitter weren't stripping the metadata they are checking.

It is apparently C2PA per [1]

[1] https://openai.com/index/understanding-the-source-of-what-we...

langsoul-com• 2 years ago

LLM powered disinformation machines is terrifying. The barrier to entry and sustained cost is so low.

But the social impact is significant. I'm reminded of a fake story about kid kidnapping in India, that caused a mob to burn the two people that were targeted alive... They were completely innocent, the mob attacked them based on fake news. Now, that can happen enmasse.

mediumsmart• 2 years ago

influencing the politics of the United States? Is there an alternative choice I have not heard of? I thought it was samesame in blue or red as it always has been.

Disrupting a covert Iranian influence operation

Related Stories