Gavin Newsom vetoes SB 1047

https://www.gov.ca.gov/wp-content/uploads/2024/09/SB-1047-Veto-Message.pdf

778 449 2 years ago

gov.ca.gov

Perhaps worried that draconian restriction on new technology is not gonna help bring Silicon Valley back to preeminence.

jprete• 2 years ago

"The Democrat decided to reject the measure because it applies only to the biggest and most expensive AI models and doesn’t take into account whether they are deployed in high-risk situations, he said in his veto message."

That doesn't mean you're wrong, but it's not what Newsom signaled.

mhuffman• 2 years ago

and doesn’t take into account whether they are deployed in high-risk situations

Am I out of the loop here? What "high-risk" situations do they have in mind for LLM's?

JoshTriplett6• 2 years ago

Only applying to the biggest models is the point; the biggest models are the inherently high-risk ones. The larger they get, the more that running them at all is the "high-risk situation".

Passing this would not have been a complete solution, but it would have been a step in the right direction. This is a huge disappointment.

comp_throw74• 2 years ago

He's dissembling. He vetoed the bill because VCs decided to rally the flag; if the bill had covered more models he'd have been more likely to veto it, not less.

It's been vaguely mindblowing to watch various tech people & VCs argue that use-based restrictions would be better than this, when use-based restrictions are vastly more intrusive, economically inefficient, and subject to regulatory capture than what was proposed here.

jart4• 2 years ago

If you read Gavin Newsom's statement, it sounds like he agrees with Terrance Tao's position, which is that the government should regulate the people deploying AI rather than the people inventing AI. That's why he thinks it should be stricter. For example, you wouldn't want to lead people to believe that AI in health care decisions is OK so long as it's smaller than 10^26 flops. Read his full actual statement here: https://www.gov.ca.gov/wp-content/uploads/2024/09/SB-1047-Ve...

m463• 2 years ago

Unfortunately he also veto'd AB3048 which allowed consumers a direct way to opt-out of data sharing.

https://digitaldemocracy.calmatters.org/bills/ca_202320240ab...

brianjking• 2 years ago

https://www.wsj.com/tech/ai/californias-gavin-newsom-vetoes-...

dang7• 2 years ago

Thanks! The WSJ article was the submitted URL, but I've changed it to the governor's statement now. Interested readers will probably want to look at both.

guywithahat• 2 years ago

Why would you change it? The WSJ article already contains his reasoning, plus a lot of other interesting content from major players

dang7• 2 years ago

Related. Others?

OpenAI, Anthropic, Google employees support California AI bill - https://news.ycombinator.com/item?id=41540771 - Sept 2024 (26 comments)

Y Combinator, AI startups oppose California AI safety bill - https://news.ycombinator.com/item?id=40780036 - June 2024 (8 comments)

California AI bill becomes a lightning rod–for safety advocates and devs alike - https://news.ycombinator.com/item?id=40767627 - June 2024 (2 comments)

California Senate Passes SB 1047 - https://news.ycombinator.com/item?id=40515465 - May 2024 (42 comments)

California residents: call your legislators about AI bill SB 1047 - https://news.ycombinator.com/item?id=40421986 - May 2024 (11 comments)

Misconceptions about SB 1047 - https://news.ycombinator.com/item?id=40291577 - May 2024 (35 comments)

California Senate bill to crush OpenAI competitors fast tracked for a vote - https://news.ycombinator.com/item?id=40200971 - April 2024 (16 comments)

SB-1047 will stifle open-source AI and decrease safety - https://news.ycombinator.com/item?id=40198766 - April 2024 (190 comments)

Call-to-Action on SB 1047 – Frontier Artificial Intelligence Models Act - https://news.ycombinator.com/item?id=40192204 - April 2024 (103 comments)

On the Proposed California SB 1047 - https://news.ycombinator.com/item?id=39347961 - Feb 2024 (115 comments)

voidfunc• 2 years ago

It was a dumb law so... good on a politician for doing the smart thing for once.

x3n0ph3n3• 2 years ago

Given what Scott Wiener did with restaurant fees, it's hard to trust his judgement on any legislation. He clearly prioritizes monied interests over the general populace.

gotoeleven• 2 years ago

This guy is a menace. Among his other recent bills are ones to require cars not be able to go more than 10mph over the speed limit (watered down to just making a terrible noise when they do) and to decriminalize intentionally giving someone AIDs. I know this sounds like hyperbole.. how could this guy keep getting elected?? But its not, it's california!

baggy_trough• 2 years ago

Scott Wiener is literally a demon in human form.

johnnyanmac4• 2 years ago

Technically you can't go over 5mph of the speed limit. And that's only because of radar accuracy.

Of course no one cares until you get a bored cop one day. And with free way traffic you're lucky to hit half the speed limit.

deredede2• 2 years ago

I was surprised at the claim that intentionally giving someone AIDS would be decriminalized, so I looked it up. The AIDS bill you seem to refer to (SB 239) lowers penalties from a felony to a misdemeanor (so it is still a crime), bringing it in line with other sexually transmitted diseases. The argument is that we now have good enough treatment for HIV that there is no reason for the punishment to be harsher than for exposing someone to hepatitis or herpes, which I think is sound.

radicality• 2 years ago

I don’t follow politics closely and don’t live in CA, but is he really that bad? I had a look on Wikipedia for some other bills he worked on that seem to me positive:

* wanted to decriminalize psychoactive drugs (lsd/dmt/mdma etc)

* wanted to allow alcohol sales till 4am

* a bill about removing parking minimums for new constructions close to public transit

Though I agree the car one seems ridiculous, and on first glance downright dangerous.

simonw• 2 years ago

Also on The Verge: https://www.theverge.com/2024/9/29/24232172/california-ai-sa...

davidu• 2 years ago

This is a massive win for tech, startups, and America.

ken472• 2 years ago

[flagged]

khazhoux• 2 years ago

The US is the world leader in AI technology. Defeating a bad AI bill is good for the US.

cornercasechase3• 2 years ago

It was a bad bill but your gross nationalism is even worse. 1 step forward, 10 steps back.

richwater2• 2 years ago

gross nationalism

How on earth did you get that from the original posters comment?

SonOfLilit18• 2 years ago

A bill laying the groundwork to ensure the future survival of humanity by making companies on the frontier of AGI research responsible for damages or deaths caused by their models, was vetoed because it doesn't stifle competition with the big players enough and because we don't want companies to be scared of letting future models capable of massive hacks or creating mass casualty events handle their customer support.

Today humanity scored a self-goal.

edit:

I'm guessing I'm getting downvoted because people don't think this is relevant to our reality. Well, it isn't. This bill shouldn't scare anyone releasing a GPT-4 level model:

The bill he vetoed, SB 1047, would have required developers of large AI models to take “reasonable care” to ensure that their technology didn’t pose an “unreasonable risk of causing or materially enabling a critical harm.” It defined that harm as cyberattacks that cause at least $500 million in damages or mass casualties. Developers also would have needed to ensure their AI could be shut down by a human if it started behaving dangerously.

What's the risk? How could it possibly hack something causing $500m of damages or mass casualties?

If we somehow manage to build a future technology that _can_ do that, do you think it should be released?

atemerev4• 2 years ago

Oh come on, the entire bill was against open source models, it’s pure business. “AI safety”, at least of the X-risk variety, is a non-issue.

whimsicalism8• 2 years ago

“AI safety”, at least of the X-risk variety, is a non-issue.

i have no earthly idea why people feel so confident making statements like this.

at current rate of progress, you should have absolutely massive error bars for what capabilities will like in 3,5,10 years.

SonOfLilit18• 2 years ago

I find it hard to believe that Google, Microsoft and OpenAI would oppose a bill against open source models.

datavirtue4• 2 years ago

The future survival of humanity involves creating machines that have all of our knowledge and which can replicate themselves. We can't leave the planet but our robot children can. I just wish that I could see what they become.

SonOfLilit18• 2 years ago

Sure, that's future survival. Is it of humanity though? Kinda no by definition in your scenario. In general, depends at least if they share our values...

johnnyanmac4• 2 years ago

Sounds like the exact opposite plot of Wall-E.

raxxorraxor2• 2 years ago

Mountains out of scrap, rivers out of oil and wide circuit plains. It will be absolutely beautiful.

tbrownaw5• 2 years ago

https://legiscan.com/CA/text/SB1047/id/3019694

So this is the one that would make it illegal to provide open weights for models past a certain size, would make it illegal to sell enough compute power to train such a model without first verifying that your customer isn't going to train a model and then ignore this law, and mandates audit requirements to prove that your models won't help people cause disasters and can be turned off.

timr2• 2 years ago

The proposed law was so egregiously stupid that if you live in California, you should seriously consider voting for Anthony Weiner's opponent in the next election.

The man cannot be trusted with power -- this is far from the first ridiculous law he has championed. Notably, he was behind the (blatantly unconstitutional) AB2098, which was silently repealed by the CA state legislature before it could be struck down by the courts:

https://finance.yahoo.com/news/ncla-victory-gov-newsom-repea...

https://www.sfchronicle.com/opinion/openforum/article/COVID-...

(Folks, this isn't a partisan issue. Weiner has a long history of horrendously bad judgment and self-aggrandizement via legislation. I don't care which side of the political spectrum you are on, or what you think of "AI safety", you should want more thoughtful representation than this.)

johnnyanmac4• 2 years ago

you should want more thoughtful representation than this.

Your opinion on what "thoughtful representation" is is what makes this point partisan. Regardless, he's in until 2028 so it'll be some time before that vote can happen.

Also, important Nitpick, it's Scott Weiner. Anthony Weiner (no relation AFAIK) was in New York and has a much more... Public controversy.

dlx• 2 years ago

you've got the wrong Weiner dude ;)

rekttrader• 2 years ago

** Anthony != Scott Weiner

GolfPopper• 2 years ago

Anthony Weiner is a disgraced New York Democratic politician who does not appear to have re-entered politics after his release from prison a few years ago. You mentioned his name twice in your post, so it doesn't seem to be an accident that you mentioned him, yet his name does not seem to appear anywhere in your links. I have no idea what message you're trying to convey, but whatever it is, I think you're failing to communicate it.

akira25013• 2 years ago

and mandates audit requirements to prove that your models won't help people cause disasters

Audits cannot prove anything and they offer no value when planning for the future. They're purely a retrospective tool that offers insights into potential risk factors.

and can be turned off.

I really wish legislators would operate inside reality instead of a Star Trek episode.

whimsicalism8• 2 years ago

This snide dismissiveness around “sci-fi” scenarios, while capabilities continue to grow, seems incredibly naïve and foolish.

Many of you saying stuff like this were the same naysayers who have been terribly wrong about scaling for the last 6-8 years or people who only started paying attention in the last two years.

lopatin• 2 years ago

Audits cannot prove anything and they offer no value when planning for the future. They're purely a retrospective tool that offers insights into potential risk factors.

What if it audits your deploy and approval processes? They can say for example, that if your AI deployment process doesn't include stress tests against some specific malicious behavior (insert test cases here) then you are in violation of the law. That would essentially be a control on all future deploys.

Loughla• 2 years ago

I really wish legislators would operate inside reality instead of a Star Trek episode.

What are your thoughts about businesses like Google and Meta providing guidance and assistance to legislators?

trog• 2 years ago

Audits cannot prove anything and they offer no value when planning for the future. They're purely a retrospective tool that offers insights into potential risk factors.

Uh, aren't potential risk factors things you want to consider when planning for the future?

teekert• 2 years ago

The best episodes are where the model can't be turned off anymore ;)

comp_throw74• 2 years ago

this is the one that would make it illegal to provide open weights for models past a certain size

That's nowhere in the bill, but plenty of people have been confused into thinking this by the bill's opponents.

tbrownaw5• 2 years ago

Three of the four options of what an "artifical intelligence safety incident" is defined as require that the weights be kept secret. One is quite explicit, the others are just impossible to prevent if the weights are available:

(2) Theft, misappropriation, malicious use, inadvertent release, unauthorized access, or escape of the model weights of a covered model or covered model derivative.
(3) The critical failure of technical or administrative controls, including controls limiting the ability to modify a covered model or covered model derivative.
(4) Unauthorized use of a covered model or covered model derivative to cause or materially enable critical harm.

Terr_3• 2 years ago

Sounds like legislation that mis-indentifies the root issue as "somehow maybe the computer is too smart" as opposed to, say, "humans and corporations should be liable for using the tool to do evil."

concordDance3• 2 years ago

The former is a potentially extremely serious issue, just not one we're likely to hit in the very near future.

raxxorraxor2• 2 years ago

That is a very bad law. People and especially corporations in favor of it should be under scrutiny for trying to corner a market for themselves.

choppaface• 2 years ago

The Apple Intelligence demos showed Apple is likely planning to use on-device models for ad targeting, and Google / Facebook will certainly respond. Small LLMs will help move unwanted computation onto user devices in order to circumvent existing data and privacy laws. And they will likely be much more effective since they’ll have more access and more data. This use case is just getting started, hence SB 1047 is so short-sighted. Smaller LLMs have dangers of their own.

jimjimjim2• 2 years ago

Thank you. For some reason I hadn't thought of the advertising angle with local LLMs but you are right!

For example, why is Microsoft hell-bent on pushing Recall onto windows? Answer: targeted advertising.

jart4• 2 years ago

Why is it wrong to show someone ads that are relevant to their interests? Local AI is a win-win, since tech companies get targeted ads, and your data stays private.

seltzered_• 2 years ago

Is part of the issue the concern that runaway ai computing would just happen outside of california?

There's another important county election in Sonoma happening about CAFOs where part of the issue is that you may get environmental progress locally, but just end up exporting the issue to another state with lax rules: https://www.kqed.org/news/12006460/the-sonoma-ballot-measure...

alhirzel• 2 years ago

Like all laws, there will certainly be those who evade compliance geographically. A well-written law will be looked to as a precedent or "head start" for new places that end up wanting regulatory functions. I feel like the EU and California often end up on this "leading edge" with regard to technology and privacy. While this can seem like a futile position to be in, it paves the way and is a required step for a good law to find a global foothold.

metadat• 2 years ago

https://archive.today/22U12

dredmorbius5• 2 years ago

Hard paywall.

worstspotgain9• 2 years ago

Excellent move by Newsom. We have a very active legislature, but it's been extremely bandwagon-y in recent years. I support much of Wiener's agenda, particularly his housing policy, but this bill was way off the mark.

It was basically a torpedo against open models. Market leaders like OpenAI and Anthropic weren't really worried about it, or about open models in general. Its supporters were the also-rans like Musk [1] trying to empty out the bottom of the pack, as well as those who are against any AI they cannot control, such as antagonists of the West and wary copyright holders.

[1] https://techcrunch.com/2024/08/26/elon-musk-unexpectedly-off...

SonOfLilit18• 2 years ago

why would Google, Microsoft and OpenAI oppose a torpedo against open models? Aren't they positioned to benefit the most?

worstspotgain9• 2 years ago

If there was just one quasi-monopoly it would have probably supported the bill. As it is, the market leaders have the competition from each other to worry about. Getting rid of open models wouldn't let them raise their prices much.

CSMastermind• 2 years ago

The bill included language that required the creators of models to have various "safety" features that would severely restrict their development. It required audits and other regulatory hurdles to build the models at all.

hn_throwaway_997• 2 years ago

Yeah, I think the argument that "this just hurts open models" makes no sense given the supporters/detractors of this bill.

The thing that large companies care the most about in the legal realm is certainty. They're obviously going to be a big target of lawsuits regardless, so they want to know that legislation is clear as to the ways they can act - their biggest fear is that you get a good "emotional sob story" in front of a court with a sympathetic jury. It sounded like this legislation was so vague that it would attract a hoard of lawyers looking for a way they can argue these big companies didn't take "reasonable" care.

benreesman2• 2 years ago

Some laws are just bad. When the API-mediated/closed-weights companies agree with the open-weight/operator-aligned community that a law is bad, it’s probably got to be pretty awful. That said, though my mind might be playing tricks on me, I seem to recall the big labs being in favor at one time.

There are a number of related threads linked, but I’ll personally highlight Jeremy Howard’s open letter as IMHO the best-argued case against SB 1047.

https://www.answer.ai/posts/2024-04-29-sb1047.html

wrsh07• 2 years ago

I would note that Facebook and Google were opposed to eg gdpr although it gave them a larger share of the pie.

When framed like that: why be opposed, it hurts your competition? The answer is something like: it shrinks the pie or reduces the growth rate, and that's bad (for them and others)

The economics of this bill aren't clear to me (how large of a fine would Google/Microsoft pay in expectation within the next ten years?), but they maybe also aren't clear to Google/Microsoft (and that alone could be a reason to oppose)

Many of the ai safety crowd were very supportive, and I would recommend reading Zvi's writing on it if you want their take

nisten2• 2 years ago

Because it's a law that first intended to put opensource developers in jail.

rllearneratwork• 2 years ago

because it was a stupid law which would hurt AI innovation

mattmaroon• 2 years ago

First they came for the open models…

dragonwriter• 2 years ago

Excellent move by Newsom. [...] It was basically a torpedo against open models.

He vetoed it in part because the threshold it applies to at all are well-beyond any current models, and he wants something that will impose greater restrictions on more and much smaller/lower-training-compute models that this would have left alone entirely.

Market leaders like OpenAI and Anthropic weren't really worried about it, or about open models in general.

OpenAI (along with Google and Meta) led the institutional opposition to the bill, Anthropic was a major advocate for it.

worstspotgain9• 2 years ago

He vetoed it in part because the threshold it applies to at all are well-beyond any current models, and he wants something that will impose greater restrictions on more and much smaller/lower-training-compute models that this would have left alone entirely.

Well, we'll see what passes again and when. By then there'll be more kittens out of the bag too.

Anthropic was a major advocate for it.

I don't know about being a major advocate, the last I read was "cautious support" [1]. Perhaps Anthropic sees Llama as a bigger competitor of theirs than I do, but it could also just be PR.

[1] https://thejournal.com/articles/2024/08/26/anthropic-offers-...

arduanika• 2 years ago

He's a politician, and his stated reason for the veto is not necessarily his real reason for the veto.

raverbashing• 2 years ago

Anthropic was championing a lot of FUD in the AI area

inferiorhuman2• 2 years ago

Newsom vetoed the bill as a nod to his donors plain and simple. Same reason he just signed a bill allowing a specific customers at a single venue to be served alcohol later than 2 AM. Same reason he carved out a minimum wage exemption for Panera. Same reason he signed a bill to carve out a junk fee exemption specifically for restaurants.

He's just planning for a post-governor career.

EasyMark2• 2 years ago

Yes they definitely need to slow their roll and sit back and listen to both sides of this instead of those who think AGI will happen in a year or two and the T1000s are coming for them. I think LLM have a bright future, especially as more hardware is build specifically for them. The market can fix most of the problems and when it becomes evident we’re heading in the wrong direction or monopolies and abuses occur, that’s when the government needs to step in, no based some broad speculation from from the fringe of either side.

pbreit2• 2 years ago

Bills that could kill major new industries need to be reactive, if at all. This was a terrible bill. Thank you, Governor.

fwip2• 2 years ago

If the new industry is inherently unsafe, it is better to be proactive.

sshconnection• 2 years ago

Same. I am generally very politically aligned with him (esp housing and transit), but this one ain’t it.

SonOfLilit18• 2 years ago

I wondered if the article was over-dramatizing what risks were covered by the bill, so I read the text:

(g) (1) “Critical harm” means any of the following harms caused or materially enabled by a covered model or covered model derivative:

(A) The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.

(B) Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from cyberattacks on critical infrastructure by a model conducting, or providing precise instructions for conducting, a cyberattack or series of cyberattacks on critical infrastructure.

(C) Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from an artificial intelligence model engaging in conduct that does both of the following:

(i) Acts with limited human oversight, intervention, or supervision.

(ii) Results in death, great bodily injury, property damage, or property loss, and would, if committed by a human, constitute a crime specified in the Penal Code that requires intent, recklessness, or gross negligence, or the solicitation or aiding and abetting of such a crime.

(D) Other grave harms to public safety and security that are of comparable severity to the harms described in subparagraphs (A) to (C), inclusive.

(2) “Critical harm” does not include any of the following:

(A) Harms caused or materially enabled by information that a covered model or covered model derivative outputs if the information is otherwise reasonably publicly accessible by an ordinary person from sources other than a covered model or covered model derivative.

(B) Harms caused or materially enabled by a covered model combined with other software, including other models, if the covered model did not materially contribute to the other software’s ability to cause or materially enable the harm.

(C) Harms that are not caused or materially enabled by the developer’s creation, storage, use, or release of a covered model or covered model derivative.

handfuloflight2• 2 years ago

Does Newsom believe that an AI model can do this damage autonomously or does he understand it must be wielded and overseen by humans to do so?

In that case, how much of an enabler is an AI to meet the destructive ends, when, if the humans can use AI to conduct the damage, they can surely do it without the AI as well.

The potential for destruction exists either way but is the concern that AI makes this more accessible and effective? What's the boogeyman? I don't think these models have private information regarding infrastructure and systems that could be exploited.

SonOfLilit18• 2 years ago

“Critical harm” does not include any of the following: (A) Harms caused or materially enabled by information that a covered model or covered model derivative outputs if the information is otherwise reasonably publicly accessible by an ordinary person from sources other than a covered model or covered model derivative.

The bogeyman is not these models, it's future agentic autonomous ones, if and when they can hack major infrastructure or build nukes. The quoted text is very very clear on that.

anigbrowl3• 2 years ago

Newsom is the governor who vetoed the bill, not the lawmaker who authored it.

concordDance3• 2 years ago

Does Newsom believe that an AI model can do this damage autonomously or does he understand it must be wielded and overseen by humans to do so?

AI models might not be able to, but an AI system that uses a powerful model might be able to cause damage (including extinction of humanity in the more distant future) unintended and unforeseen by its creators.

The more complex and unpredictable the system the harder it is to properly oversee.

dang7• 2 years ago

(I added newlines to your quote to match what looked like the intended formatting - I hope that's ok. Since HN doesn't do indentation I'm not sure it helps all that much...)

ketzo2• 2 years ago

I’m sure people have asked this before, but would HN ever add a little more rich-text? Even just bullet points and indents might be nice.

w10-1• 2 years ago

--- (2) “Critical harm” does not include any of the following:

---

This exception swallows any rule and fails to target the difference with AI: it's actually better than an ordinary person at assimilating multiple fact streams.

That suggests this law is legislative theater: something designed to enlist interest and donations, i.e., to build a political franchise. That could be why it targets only the largest models, affecting only the biggest players, who have the most resources to donate per decision and the least goodwill to burn with opposition.

Regulating AI would be a very difficult legislative/administrative task, on the order of a new tax code in its complexity. But it will be impossible if treated as a political franchise.

As for self-regulation, with OpenAI's changing to for-profit, the non-profit form is insufficient to maintain a public benefit focus. Permitting this conversion is on par with the 1990's+ conversion of nonprofit hospital systems to for-profit.

AI's potential shines a bright light on our weakness in governance. While weak governance affords more opportunities, escaping the exploitation caused by governance failures is the engine of autocracy, and autocracy consumes property along with all other rights.

hn_throwaway_997• 2 years ago

Curious if anyone can point to some resources that summarize the pros/cons arguments of this legislation. Reading this article, my first thought is that I definitely agree it sounds impossibly vague for a piece of legislation - "reasonable care" and "unreasonable risk" sound like things that could be endlessly litigated.

At the same time,

Computer scientists Geoffrey Hinton and Yoshua Bengio, who developed much of the technology on which the current generative-AI wave is based, were outspoken supporters. In addition, 119 current and former employees at the biggest AI companies signed a letter urging its passage.

These are obviously highly intelligent people (though I've definitely learned in my life that intelligence in one area, like AI and science, doesn't mean you should be trusted to give legal advice), so I'm curious to know why Hinton and Bengio supported the legislation so strongly.

throwup238• 2 years ago

California’s Office of Legislative Counsel always provides a “digest” for every bill as part of its full text: https://leginfo.legislature.ca.gov/faces/billTextClient.xhtm...

It’s not an opinionated pros/cons list from the industry but it’s probably the most neutral explanation of what the bill does.

mmmore• 2 years ago

The concern is that near future systems will be much more capable than current systems, and by the time they arrive, it may be too late to react. Many people from the large frontier AI companies believe that world-changing AGI is 5 years or less away; see Situational Awareness by Aschbrenner, for example. There's also a parallel concern that AIs could make terrorism easier[1].

Yoshua Bengio has written in detail about his views on AI safety recently[2][3][4]. He seems to put less weight on human level AI being very soon, but says superhuman intelligence is plausible in 5-20 years and says:

Faced with that uncertainty, the magnitude of the risk of catastrophes or worse, extinction, and the fact that we did not anticipate the rapid progress in AI capabilities of recent years, agnostic prudence seems to me to be a much wiser path.

Hinton also has a detailed lecture he's been giving recently about the loss of control risk.

In general, proponents see this as narrowly tailored bill to somewhat address the worst case worries about loss of control and misuse.

[1] https://www.theregister.com/2023/07/28/ai_senate_bioweapon/

[2] https://yoshuabengio.org/2023/05/22/how-rogue-ais-may-arise/

[3] https://yoshuabengio.org/2023/06/24/faq-on-catastrophic-ai-r...

[4] https://yoshuabengio.org/2024/07/09/reasoning-through-argume...

crazygringo• 2 years ago

impossibly vague for a piece of legislation - "reasonable care" and "unreasonable risk" sound like things that could be endlessly litigated.

Nope, that's entirely standard legal stuff. Tort law deals exactly with those kinds of things, for instance. Yes it can certainly wind up in litigation, but the entire point is that if there's a gray area, a company should make sure it's operating entirely within the OK area -- or know it's taking a legal gamble if it tries to push the envelope.

But it's generally pretty easy to stay in the clear if you establish common-sense processes around these things, with a clear paper trail and decisions approved by lawyers.

Now the legislation can be bad for lots of other reasons, but "reasonable care" and "unreasonable risk" are not problematic.

hn_throwaway_997• 2 years ago

but "reasonable care" and "unreasonable risk" are not problematic.

Still strongly disagree, at least when it comes to AI legislation. Yes, I fully realize that there are "reasonableness" standards in lots of places of US jurisprudence, but when it comes to AI, given how new the tech is and how, perhaps more than any other recent technology, it is largely a "black box", meaning we don't really know how it works and we aren't really sure what its capabilities will ultimately be, I don't think anybody really knows what "reasonableness" means in this context.

leogao• 2 years ago

I looked into the question of what counts as reasonable care and wrote up my conclusions here: https://www.lesswrong.com/posts/kBg5eoXvLxQYyxD6R/my-takes-o...

hn_throwaway_997• 2 years ago

Thank you! Your post was really helpful in aiding my understanding, so I greatly appreciate it.

Also, while reading your article I also fell onto https://www.brookings.edu/articles/misrepresentations-of-cal... while trying to understand some terms, and that also gave some really good info, e.g. the difference between a "reasonable assurance" language that was dropped from an earlier version of the bill and replaced with "reasonable care".

ketzo2• 2 years ago

This was a great post, thanks.

svat• 2 years ago

Here's a post by the computer scientist Scott Aaronson on his blog, in support: https://scottaaronson.blog/?p=8269 -- it links to some earlier explainers, has some pro-con arguments, and further discussion in the comments.

hn_throwaway_997• 2 years ago

Oh, wow, thanks very much! Not only was that a very informative article, but it also has links to other detailed opinions on the topic (and some of those had links...), which left me feeling much better informed. Much appreciated!

nisten2• 2 years ago

Imagine being concerned about AI safety and then introducing a bill that had to be ammended to change criminal responsability of AI developers to civil legal responsability for people who are trying to investigate and work openly on models.

What's next, going after maintainers of python packages... is attacking transparency itself a good way to make AI safer. Yeah, no, it's f*king idiotic.

Lonestar14403• 2 years ago

This is no way to run a state. The Democrat-dominated legislature passes everything that comes before it (and rejects anything that the GOP touches, in committee) and then the Governor needs to veto the looniest 20% of them to keep us from falling into total chaos. This AI bill was far from the worst one.

"Vote out the legislators!" but for who... the Republican party? And we don't even get a choice on the general ballot most of the time, thanks to "Open Primaries".

It's good that Newsom is wise enough to muddle through, but this is an awful system.

https://www.pressdemocrat.com/article/news/california-gov-ne...

thinkingtoilet• 2 years ago

If California was it's own country, it would be one of the biggest most successful countries in the world. Like every where else it has it's problems but it's being run just fine. Objectively, there are many states that are far worse off in any key metric.

toephu23• 2 years ago

but it's being run just fine

As a Californian I have to disagree. The only reason you think it's being run just fine is because of the success of the private sector. The only reason California would be the 4th/5th largest economy in the world is because of the the tech industry and other industries that are in California (Hollywood, agriculture, etc). It's not because we have some awesome efficiently run state government.

ken472• 2 years ago

You're going to attribute even a small % of this to politicians rather than the actual innovators? Sure, then let's say they're responsible for some small % of its success. They're smart enough to not nuke their own economy.

LeroyRaz2• 2 years ago

The state has one of the highest illiteracy rates in the whole country (28%). To me, that implies they have some issue of governance.

Source: https://worldpopulationreview.com/state-rankings/us-literacy...

To be fair in the comparison, the literacy statistics for the whole of the US are pretty shocking from a European perspective.

cscurmudgeon2• 2 years ago

California is the largest recipient of federal money.

https://usafacts.org/articles/which-states-rely-the-most-on-...

(I know by population it will be different, but the argument here is around 'one of the the biggest' which is not a per capita statement.)

Objectively, there are many states that are far worse off in any key metric

You can apply the same logic to USA.

USA is one of the biggest most successful countries in the world. Like every where else it has it's problems but it's being run just fine. Objectively, there are many countries that are far worse off in any key metric.

kortilla4• 2 years ago

What is success in your metric? are you just counting GDP of companies that happen to be located there? If so, that has very little relationship to how well the state is being run.

It’s very easy to make arguments that they are successful in spite of a terribly run state government and are being protected by federal laws keeping the loonies in check (interstate commerce clause, etc).

tightbookkeeper2• 2 years ago

In this case the success is in spite of the governance rather than because of it.

The golden age of California was a long time ago.

hbarka• 2 years ago

High speed trains would do even more for California and would be the envy of the rest of the country.

aagha2• 2 years ago

Thank you.

I always think about this whenever someone says CA doesn't know what it's doing or it's being run wrong:

California's GDP in 2023 was $3.8T, representing 14% of the total U.S. economy. If California were a country, it would be the 5th largest economy in the world and more productive than India and the United Kingdom.

nradov2• 2 years ago

California is being terribly misgoverned, as you would expect in any single-party state. In some sense California has become like a petro-state afflicted by the resource curse: the tech industry throws off so much cash that the state runs reasonably well, not because of the government but in spite of it. We can afford to waste government resources on frivolous nonsense.

And this isn't a partisan dig at Democrats. If a Republicans controlled everything then the situation would be just as bad but in different ways.

anon2919• 2 years ago

California has unique geographic features that make it well positioned. It also has strategic geographic resources (like oil). This is like using Saudi Arabia as a standard of governance since they have greatly improved material conditions using oil money.

California does do several things very well. It also does several things poorly. Pointing out its current economic standing does not change that. The fallacy here is that we have to compare california against the best version of itself. Alaska is never going to be a California-level economy because the geography dictates that only a certain kind of person / company will set up there, for example. That doesn't mean the Alaskan government is necessarily bad. Every state has to work within its limits to achieve the best version of itself. Is california the best it could be? I think the answer is obviously no.

dyauspitr3• 2 years ago

Compared to whom? What is this hypothetical well run state. Because it’s hard to talk shit against the state that has the 8th largest economy on the world nation state economy ranking.

dredmorbius5• 2 years ago

Thankfully there are no such similarly single-party states elsewhere in the Union dominated by another party, and if they were, their executives would similarly veto the most inane legislation passed.

</s>

rootusrootus6• 2 years ago

The subtle rebranding of Democratic party to democrat party is a pretty strong tell for highly partisan perspective. How does California compare with similarly large Republican-dominated states? Anecdotally, I’ve seen a lot of really bad legislation originating from any legislature that has no meaningful opposition.

anigbrowl3• 2 years ago

It's such a long-running thing that it's hard to gauge whether it's deliberate or just loose usage.

https://en.wikipedia.org/wiki/Democrat_Party_(epithet)

Lonestar14403• 2 years ago

I'm a pretty pedantic person, but even I just use one or the other at random. I don't think it's a good idea to read into things like this.

dehrmann• 2 years ago

Not sure of Newsom is actually wise enough or if his presidential ambitions moderate his policies.

ravenstine• 2 years ago

It could be presidential ambitions, though I suspect his recent policies have been merely a way of not giving conservatives more ammo leading up to the 2024 election. The way he's been behaving recently is in stark contrast to pretty much everything he's done during and before his governorship. I don't think it's because he's suddenly any wiser.

gdiamos7• 2 years ago

If that bill had passed I would have seriously considered moving my AI company out of the state.

elicksaur• 2 years ago

Nothing like this should pass until the legislators can come up with a definition that doesn’t encompass basically every computer program ever written:

(b) “Artificial intelligence model” means a machine-based system that can make predictions, recommendations, or decisions influencing real or virtual environments and can use model inference to formulate options for information or action.

Yes, they limited the scope of law by further defining “covered model”, but the above shouldn’t be the baseline definition of “Artificial intelligence model.”

Text: https://legiscan.com/CA/text/SB1047/id/2919384

StarterPro• 2 years ago

Whaaat? The sleazy Governor sided with the tech companies??

I'll have to go get a thesaurus, shocked won't cover how I'm feeling rn.

blackeyeblitzar• 2 years ago

It is strange to see Newsom make good moves like this but then also do things like veto bipartisan supported reporting and transparency for the state’s homeless programs. What is his political strategy exactly?

scoofy• 2 years ago

Newsom vetoes so many bills that it makes little sense why the legislature should even be taken seriously. Our Dem trifecta state has effectively become captured by the executive.

dyauspitr3• 2 years ago

As opposed to what? The supermajority red states where gerrymandered counties look like corn mazes and the economy is in the shitter?

stuaxo• 2 years ago

This is good - they were trying to legislate against future competitors.

reducesuffering• 2 years ago

taps the sign

"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." - Geoffrey Hinton, Yoshua Bengio, Sam Altman, Bill Gates, Vitalik Buterin, Ilya Sutskever, Demis Hassabis

"Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity. There are other threats that I think are more certain to happen but are unlikely to destroy every human in the universe in the way that SMI could." - Sam Altman

"I actually think the risk is more than 50%, of the existential threat." - Geoffrey Hinton

"Currently, we don't have a solution for steering or controlling a potentially superintelligent AI, and preventing it from going rogue." - OpenAI

"while we are racing towards AGI or even ASI, nobody currently knows how such an AGI or ASI could be made to behave morally, or at least behave as intended by its developers and not turn against humans." - Yoshua Bengio

"very soon they're going to be, they may very well be more intelligent than us and far more intelligent than us. And at that point, we will be receding into the background in some sense. We will have handed the baton over to our successors, for better or for worse.

But it's happening over a period of a few years. It's like a tidal wave that is washing over us at unprecedented and unimagined speeds. And to me, it's quite terrifying because it suggests that everything that I used to believe was the case is being overturned." - Douglas Hofstadter

The Social Dilemna was discussed here with much praise about how profit incentive caused mass societal issues in social media. I'm astounded it's fallen on deaf ears when the same people also made the AI Dilemna describing the parallels coming with AGI:

https://www.youtube.com/watch?v=xoVJKj8lcNQ

dyauspitr3• 2 years ago

Newsom has been on fire lately.

sandspar• 2 years ago

Newsom wants to run for president in 4 years; AI companies will be rich in 4 years; Newsom will need donations from rich companies in 4 years.

LarsDu88• 2 years ago

Terrible piece of legislation. Glad the governor took it down. This is what regulatory capture looks like. Someone commoditized your product, so you make it illegal for them to continue making your stuff free.

Might as well make Linux illegal so everyone is forced to use Microsoft and Apple.

xpe2• 2 years ago

I disagree.

On this topic, I’m seeing too many ideological and uninformed claims.

It is hard for many aspiring AI startup founders to rationally and neutrally assess the AI landscape, pros and cons.

weebull• 2 years ago

I suspect this was vetoed more for reasons of not wanting to handicap California in the "AI race" than anything else.

EasyMark2• 2 years ago

It’s also what makes companies realize there are 49 other states, and nearly a couple hundred companies. California has a rare zeitgeist of tech and universities, but nothing that can’t be reproduced elsewhere with enough dollars and promises

indigo0086• 2 years ago

Logical Fallacies built into the article headline.

m3kw9• 2 years ago

All he needed to see is how Europe is doing with these regulations

sgt• 2 years ago

What is currently happening (or what is the impact) of those regulations in EU?

renewiltord• 2 years ago

It’s making it nice for Americans to vacation there.

tsunamifury• 2 years ago

Scott Weiner is a total fraud. He passes hot concept bills then cuts out loopholes for his “friends”.

He should be ignored at least and voted out.

He’s a total POS.

water9• 2 years ago

I’m so sick of people Restricting freedoms and access to knowledge in the name safety. Tyranny always comes in the form of it’s for your own good/safety

dandanua2• 2 years ago

Sure, why we just not let everyone to build nukes and use them on anyone they don't like? Knowledge is the power. The BEST power you can get.

anon2919• 2 years ago

You cannot seriously compare nuclear materials delivery / handling to the creation of model weights and computation

cultureswitch5• 2 years ago

Big difference between allowing everyone build nukes and allowing everyone to share the information necessary to build nukes.

richrichie2• 2 years ago

I am disappointed that there are no climate change regulations on AI models. Large scale ML businesses are massive carbon emitters, not counting the whimsical training of NNs by every other IT person. This needs to be regulated.

anon2919• 2 years ago

California already has cap and trade. There doesn't seem to be a need for further regulation. If there's a problem with emissions, adjust the pricing. That's the purpose of cap and trade.

richrichie2• 2 years ago

That’s just bad faith reshuffling of “ownership” and does not address the massive carbon emissions produced by the large scale NN industry. It is going to get worse.

Not a good look for a group of people otherwise obsessed with climate change.

cultureswitch5• 2 years ago

The cap and trade principle is sound however legislators have allowed naked scams to legally claim that they are offsetting carbon output.

Such as owning a piece of forest land and pinky-swear you're not going to exploit it. This offsets nothing. There is no additional carbon being absorbed out of the atmosphere. Yet this counts as emission reduction.

az226• 2 years ago

Based.

pmcf• 2 years ago

Not today, regulatory capture. Not today.

xyst• 2 years ago

I don’t understand why it was vetoed or why this was even proposed. But leaving comment here to analyze later.

dazzaji• 2 years ago

Among the good reasons for SB-1047 to have been vetoed are that it would have regulated the wrong thing. Here’s a great statement of this basic flaw: https://law.mit.edu/pub/regulatesystemsnotmodels

Not speaking for MIT here, but that bill needs a veto and a deep redraft.

gerash• 2 years ago

I'm trying to parse this proposed law.

What does a "full shutdown" mean in the context of an LLM? Stopping the servers from serving requests? It sounds silly idk.

unit149• 2 years ago

Much like UAW, a union for industrial machinists and academics, this bill has united VCs and members of the agrarian farming community. Establishing an entity under the guise of the Board of Frontier Models parallels efforts at Jekyll Island under Wilsonian idealism. Technological Keynesianism is on the horizon. These are birth pangs - its first gasps.

cultureswitch5• 2 years ago

Sounds interesting, could you translate to English?

badsandwitch• 2 years ago

The race for true AI is on and the fruits are the economic marginalization of humanity. No game theoretic actor in the running will shy away from the race. Anyone who claims they will for the 'good of humanity' is lying or was never a contender.

This is information technology we are talking about, it's virtually the exact opposite of nuclear weapons. Refining uranium vs. manufacturing multi purpose silicon and guzzling electricity. Achieving deterrence vs. gaining immeasurable wealth, power and freedom from labor.

This race may even be the answer to the Fermi paradox - that there are few individual winners and that they pull up the ladders behind them.

This is not the kind of race any legislation will have meaningful effect on.

The race is on and you better commit to a faction that may deliver.

h0l0cube2• 2 years ago

This race may even be the answer to the Fermi paradox

The mostly unchallenged popular notion that fleshy human intelligence will still be running the show 100s – let alone 1000s – of years from now is very naive. We're nearing the end of the human supremacy, though most of us won't live to see that end.

kjkjadksj• 2 years ago

To be fair fleshy human intelligence has hardly been running the show any more than a bear eating a salmon out of a river thus far. We’d like to consider we can control the world yet any data scientist will tell you what we actually control and understand is very little, or at best a sweeping oversimplification of this complex world.

concordDance3• 2 years ago

The race is on and you better commit to a faction that may deliver.

How does that help?

The giant does not care whether the ant he steps on worships him or not. Regardless of the autonomy or not of the AI, why should the controllers help you?

xpe2• 2 years ago

Anyone who claims they will for the 'good of humanity' is lying or was never a contender.

An overreach.

Some people and organizations are more aligned with humanity’s well being and survival than others.

cultureswitch5• 2 years ago

I agree with the analysis though LLMs might not be the specific tech that makes it.

I think the truly humanity-respecting solution is to figure out and implement a stable and fair-er 50% post-labor society before we actually get there. UBI (not the unmanaged flash layer) sounds like one part of the solution.

Trying to preserve employment is demonstrably wrong when already today there are too few unskilled jobs that make economic sense for the large amount of humans unable to learn more complex tasks.

GistNoesis• 2 years ago

So I've this code, it's called ShoggothDb, it's less than a megabyte of definitions. The principle is easy, it's fully deterministic.

Code as Data, Data as Code.

When you start the program, it joins the swarm : it starts by grabbing a torrent, and train a model on it in a distributed fashion, and publish the results as a torrent. Then with the trained model, it generates new data, (think of it like alpha-go playing new games to collect more data).

See it as a tower of knowledge building itself, following some rough initial plans.

Of course, at anytime you can fork the tower, and continue building with different plans, provided that you can convince other people from the swarm to contribute to the new tower rather than the old.

Everything is immutable, but there is a versioning protocol built-in that allow the swarm to coordinate and automatically jumps to a next fork when the byzantine resistant quorum it follows vote for doing so (which allow your swarm to be compliant of the law and remove data if it was flagged as inappropriate). This allow some form of external control, but you can also let the quorum vote on subsequent modifications based on a model built on its data (aka free-running mode).

It's using torrent because easier to bootstrap but because the whole computation deterministic, the underlying protocol is just files on disks and any way of sharing them is valid. So you can grab a piece to work on via http, or ftp, or carrier pigeon for all I care. As long as the digital signatures are conforming to the rules, brick by brick the tower will get built.

To contribute, you can either help with file distribution by sharing the torrent, so it's as safe as your p2p client. If you want to commit some computing resources, like your gpu to building some of the tower, it's only requiring you to trust that there is no bug in ShoggothDb, because the computation you'll perform are composed of safe blocks, by construction they are safe for your computer to run. (Except if you want to run unsafe blocks, at this point no guarantee can be made).

The incentives for helping building the tower can be set in the initial definition file, and range from mere access to the built tower to tokens for honest participation for the more materialistically inclined.

Is it OK to release with the new law ? Is this comment OK, because ShoggothDB5-o can built its source from the specs in this comment ?

dgellow• 2 years ago

There is no new law

alkonaut3• 2 years ago

The immediate danger of large AI models isn't that they'll turn the earth to paperclips it's that we'll create fraud as a service and have a society where nothing can be trusted. I'd be all for a law (however clumsy) that made image, audio or video content produced by models with over X parameters to be marked with metadata saying it's AI generated. Creating models that don't tag their output as such would be banned. So far nothing strange about the law. The obvious problem with the law is that you need to require even screenshotting an image AI and reposting it online without the made-with-ai metadata to be outlawed. And that would be an absolute mess to enforce, at least for images.

But most importantly: whatever we do in this space has to be made on the assumption that we can't really influence what "bad actors" do. Yes being responsible means leaving money on the table. So money has to be left on the table, for - erm - less responsible nations to pick up. That's just a fact.

worldsayshi• 2 years ago

Any law that tries to categorize non-trustworthy content seems doomed to fail. We need to find better ways to communicate trustworthiness, not the other way around. (And I'm not sure adding more laws can help here.)

alkonaut3• 2 years ago

No I don't think technical means will work fully either. But the thing about these regulations is that you can basically cover the 99% case by just thinking about the 5 largest players in the field, be it regulation for social media, AI or whatever. It doesn't matter that the law has loopholes or that some players aren't affected at all. Regulation that helps somewhat in a large majority of cases is massive.

arder• 2 years ago

I think the most acheivable way of having some verification of AI images is simply for the AI generators to store finger prints of every image they generate. That way if you ever want to know you can go back to Meta or whoever and say "Hey, here's this image, do you think it came from you". There's already technology for that sort of thing in the world (content ID from youtube, CSAM detection etc.).

It's obviously not perfect, but could help and doesn't have the enormous side effects of trying to lock down all image generation.

Someone• 2 years ago

That way if you ever want to know you can go back to Meta or whoever and say "Hey, here's this image, do you think it came from you".

Firstly, if you want to know an image isn’t generated, you’d have to go to every ‘whoever’ in the world, including companies that no longer exist.

Secondly, if you ask evil.com that question, you would have to trust them to answer honestly for both all images they generated and images they didn’t generate (claiming real pictures were generated by you can probably be career-ending for a politician)

This is worse than https://www.cs.utexas.edu/~EWD/ewd02xx/EWD249.PDF: “Program testing can be used to show the presence of bugs, but never to show their absence!”. You can neither show an image is real nor that it is fake.

kortex• 2 years ago

What's to stop someone from downloading an open source model, running it themselves, and either just not sharing the hashes, subtly corrupting the hash algo so that it gives a false negative, etc?

Also you need perceptual hashing (since one bitflip of the generated media alters the whole hash) which is squishy and not perfectly reliable to begin with.

anon2919• 2 years ago

Creating models that don't tag their output as such would be banned.

This is just silly. Anyone would be able to disable this tagging in an open model.

diggan4• 2 years ago

> Creating models that don't tag their output as such would be banned.
This is just silly. Anyone would be able to disable this tagging in an open model.

And we'd end up with people who thinks that any text that isn't tagged as "#MadeByLLM" as made by a human, which obviously wouldn't be great.

jerjerjer• 2 years ago

Anyone would be able to disable this tagging in an open model.

Metadata (I assume it's file metadata and not a watermark) can be removed from a final product (image, video, text) so open and closed models are equally affected.

drcode2• 2 years ago

turning the earth into paperclips is not gonna happen immediately, so we can safely ignore that risk

woah• 2 years ago

The immediate danger of large AI models isn't that they'll turn the earth to paperclips it's that we'll create fraud as a service and have a society where nothing can be trusted. I'd be all for a law (however clumsy) that made image, audio or video content produced by models with over X parameters to be marked with metadata saying it's AI generated.

Just make a law that makes it so that AI content has to be tagged if it is being used for fraud

dandanua2• 2 years ago

"By focusing only on the most expensive and large-scale models, SB 1047 establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology. Smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047 - at the potential expense of curtailing the very innovation that fuels advancement in favor of the public good."

The amount of idiots who can't read and cheer the veto as a win against "the regulatory capture" is astounding.

OJFord2• 2 years ago

gov.ca.gov

Ah, I think now I know why Canada's government website is canada.ca (which I remember thinking was a bit odd or more like a tourism site when looking a while ago, vs. say gov.uk or gov.au).

whalesalad• 2 years ago

unfortunately the us owns the entire gov tld

OJFord2• 2 years ago

Yes but other countries (off the top of my head: UK, Aus, India) use gov.[ccTLD]

My point was that that's confusing with gov.ca if the US is using ca.gov and gov.ca.gov for California, and that perhaps that's why Canada does not do that.

skywhopper• 2 years ago

Sad. The real threat of AI is not that it will become an unstoppable superintelligence without appropriate regulation (if we reach that point, which we are nowhere close to, and probably not even on the right track) the superintelligence, by definition, will be able to evade any regulation or control we attempt.

Rather, the threat of AI is that we will dedicate so many resources—money, power, human effort—to chasing the ludicrous fantasies of professional snake-oil salesmen, while ignoring the need to address actual problems with real, known solutions that are easily within each given a fraction of the resources currently being consumed by the dumpster-fire pit of “AI”.

Unfortunately the Governor of California is a huge part of the problem here, misdirecting scarce state resources into sure-to-fail “public partnerships” with VC-funded scams, forcing public servants to add one more set of time-wasting nonsense to the pile of bullshit they have to navigate around just to do their actual job.

tim333• 2 years ago

I'm not sure AI risks are well enough understood to have good regulations for. With most risky industries you can actually quantify the risk a bit. Regarding:

we cannot afford to wait for a major catastrophe to occur before taking action to protect the public

Maybe but you can wait for minor problems or big near misses before legislating it all up.

amai• 2 years ago

What are the differences to EU AI Act?

malwrar• 2 years ago

This veto document is shockingly lucid, I'm quite impressed with it despite my belief that regulation as a strategy for managing critical risks of AI is misguided.

tl;dr gavin newsom thinks that a signable bill needs "safety protocols, proactive guardrails, and severe consequences" based on some general framework guided by "empirical trajectory analysis", and also is mindful of the promise/threat/gravity of all the machine learning occurring in CA specifically. He also affirms a general appetite for CA to take on a leadership role wrt regulating AI. My general read is that he wants to preserve public attention on the need for AI regulation and not squander it on SB 1047 specifically. Or who knows I'm not a politician lol. Really strong document tho,

Interesting segment:

By focusing only on the most expensive and large-scale models, SB 1047 establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology. Smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047 - at the potential expense of curtailing the very innovation that fuels advancement in favor of the public good. Adaptability is critical as we race to regulate a technology still in its infancy. This will require a delicate balance. While well-intentioned, SB 1047 does not take into account whether an Al system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data. Instead, the bill applies stringent standards to even the most basic functions - so long as a large system deploys it. I do not believe this is the best approach to protecting the public from real threats posed by the technology.

This is an incisive critique of the fundamental initial goal of SB 1047. Based on the fact that the bill explicitly seeks to cover models whose training cost was >=$100m & expensive fine-tunes, my initial guess about this bill was that it was designed by someone software engineering-minded scared of e.g. open-weight releases a la facebook/mistral/etc teaching someone how to build a nuke or something. LLMs probably replaced the ubiquitous robot lady pictures you see in every AI-focused article as public enemy number one, and the bill feels focused on some of the technical specifics of this advent and its widespread use. This focus blinds the bill from addressing the general danger of machine learning however, which naturally confounds regulation for precisely the reason plain-spoken in the above four sentences. Incredible technical communication here.

londons_explore• 2 years ago

While I agree with this decision, I don't want any governance decisions to be made by one bloke.

Why do we have such a system? Why isn't it a vote of many governors? Preferably a secret vote so voters can't be forced to vote along party lines...

raluk• 2 years ago

California will not abandon its responsibility.

karaterobot• 2 years ago

Safety protocols must be adopted. Proactive guardrails should be implemented, and severe consequences for bad actors must be clear and enforceable.

Here, here. But this vague, passive voiced, hand-wavey statement that isn't even a promise does not exactly inspire me with a ton of confidence. Considering he turned this bill down to protect business interests, I wonder what acceptable legislation would look like, from his perspective. Looking forward to hearing about about it very soon, and I'm confident it'll be specific, actionable, responsible, and effective.

xixixao• 2 years ago

I find the statement quite well measured. He’s not giving a solution, that’s not easy, but is specifically calling out evidence-based measures. The statement calls out both the need to regulate, and the need to innovate. The issue is not black and white and neither is the statement.

lostdog2• 2 years ago

Well, you already can't nuke someone. You can't make and release a biological weapon. It's probably illegal to turn the whole world into paperclips.

There are already laws against doing harm to others. Sure, we need to fill some gaps (like preventing harmful deep fakes), but most things are pretty illegal already.

notepad0x90• 2 years ago

Gonna swim against the current on this one.

This is why we can't have nice things, too many tech people support Newsom on vetoing this. The nightmare of corporate surveillance and erosion of privacy we have to endure every day is a result of such sentiment and short sighted attempt at self-preservation.

"It's vague" yeah, that's the point, the industry is allowed leeway to come up with standards of what is and isn't safe. They can establish a neutral committee to continually assess the boundaries of what is and isn't safe, as technology evolves. Do you expect legislators to define specifics and keep themselves updated with the latest happening in tech? Would it be better if the government established departments that police AI usage? This was the sweetest deal the industry could have gotten.

humansareok1• 2 years ago

The dismal level of discourse about this bill shows that Humanity is utterly ill equipped to deal with the problems AI poses for our society.

diggan4• 2 years ago

For very interested readers, here is a meta-collection of articles from left, center and right about the story: https://ground.news/article/newsom-vetoes-bill-for-stricter-...

And a short bias comparison:

The left discusses Newsom's veto as a victory for Silicon Valley, concentrating on the economic implications and backing from tech giants.
The center highlights the broader societal ramifications, addressing criticisms from various sectors, such as Hollywood and AI safety advocates.
The right emphasizes Newsom's concerns about hindering innovation and his commitment to future collaborative regulatory efforts, showcasing a balanced approach.

pc864• 2 years ago

Ground.News (no affiliation) is great for anyone interested in getting actual news and comparing biases. I particularly like that if you have an account they will show you news stories you're missing based on your typical reading patterns.

I do wish the partisan categorization was a bit more nuanced/intentional. It basically boils down to:

- Major national news outlet? Left.

- Local affiliate of major national news outlet? Center.

- Blog you've never heard of? Right.

There are certainly exceptions but that heuristic will be right 90% of the time.

diggan4• 2 years ago

It is truly great, and cheap too (30 USD/year or something). Not affiliated either, just a happy user.

Yeah, it could be a bit better. As a non-American, the biases are also very off from how left/center/right looks in my own country, but at least it tries to cover different angles which I tried to do manually before.

They can also be a bit slow at linking different stories together, sometimes it takes multiple days for same headlines to be merged into one story.

bcrosby952• 2 years ago

It's funny that it has Fox news as center to me. I watched them back when Obama was president a couple times and some of the shows would play Nazi videos while talking about him. Nevermind birtherism.

I haven't watched them in over a decade, but I assume they haven't gotten any better.

Sohcahtoa82• 2 years ago

- Blog you've never heard of? Right.

That checks out.

There's a certain sect of the far right that is easily convinced by one guy saying "The media is lying to you! This is what's really happening!" followed by the most insane shit you've read in your life.

They love to rant about the Deep State.

qskousen• 2 years ago

I've found The Tangle (https://www.readtangle.com/ - no affiliation) to be a pretty balanced daily politics newsletter. They mentioned the Newsom veto today, and may address it later this week, though I don't know for sure.

tr3ntg• 2 years ago

This is a little annoying. He vetoes the bill, agrees with the intention, paints the "solution" as improper, suggests there is some other solution that is better, doesn't entirely describe that solution with any detail, and encourages future legislation that is "more right."

I'm exhausted already.

I can't think of a less efficient way to go about this.

hot_gril2• 2 years ago

The "this bill doesn't go far enough" thing is normally what politicians say when they don't want it to go in that direction at all. Anyway, I'm glad he vetoed.

blueyes• 2 years ago

Remember when one of the bill's co-sponsors announced that he was co-founding an AI regulation company called Gray Swan?

https://www.piratewires.com/p/sb-1047-dan-hendrycks-conflict...

This bill was tainted from day 1, and Wiener was bamboozled into pushing it.

bradhilton• 2 years ago

I'm glad he vetoed the bill, but his rationale is worrisome. Even if he's just trying to placate SB 1047 proponents, they will try to exact concessions from him in future sessions. I'll take this brief reprieve, but it's still a large concern for me.

RIMR• 2 years ago

What specifically do you find worrisome about his rationale? It mostly seems like he's asking for evidence-based policy that views AI as a potential risk regardless of the funding or size of the model, because this doesn't actually correlate with any actual evidence of risk.

I can't tell what direction your disagreement goes. Are you worried that he still feels that AI needs to be regulated at all, or do you think that AI needs to be regulated regardless of empirical evidence of harm?

karlzt• 2 years ago

Here is the text of the PDF:

"OFFICE OF THE GOVERNOR

SEP 29 2024

To the Members of the California State Senate:

I am returning Senate Bill 1047 without my signature.

This bill would require developers of large artificial intelligence (Al) models, and those providing the computing power to train such models, to put certain safeguards and policies in place to prevent catastrophic harm . The bill would also establish the Board of Frontier Models - a state entity - to oversee the development of these models. California is home to 32 of the world's 50 leading Al companies , pioneers in one of the most significant technological advances in modern history. We lead in this space because of our research and education institutions, our diverse and motivated workforce, and our free-spirited cultivation of intellectual freedom. As stewards and innovators of the future, I take seriously the responsibility to regulate this industry. This year, the Legislature sent me several thoughtful proposals to regulate Al companies in response to current, rapidly evolving risks - including threats to our democratic process, the spread of misinformation and deepfakes, risks to online privacy, threats to critical infrastructure, and disruptions in the workforce. These bills, and actions by my Administration, are guided by principles of accountability, fairness , and transparency of Al systems and deployment of Al technology in California.

SB 1047 magnified the conversation about threats that could emerge from the deployment of Al. Key to the debate is whether the threshold for regulation should be based on the cost and number of computations needed to develop an Al model, or whether we should evaluate the system's actual risks regardless of these factors. This global discussion is occurring as the capabilities of Al continue to scale at an impressive pace. At the same time, the strategies and solutions for addressing the risk of catastrophic harm are rapidly evolving.

By focusing only on the most expensive and large-scale models, SB 1047 establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology. Smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047 - at the potential expense of curtailing the very innovation that fuels advancement in favor of the public good.

Adaptability is critical as we race to regulate a technology still in its infancy. This will require a delicate balance. While well-intentioned, SB 1047 does not take into account whether an Al system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data. Instead, the bill applies stringent standards to even the most basic functions - so long as a large system deploys it. I do not believe this is the best approach to protecting the public from real threats posed by the technology.

Let me be clear - I agree with the author - we cannot afford to wait for a major catastrophe to occur before taking action to protect the public. California will not abandon its responsibility. Safety protocols must be adopted. Proactive guardrails should be implemented, and severe consequences for bad actors must be clear and enforceable. I do not agree, however, that to keep the public safe, we must settle for a solution that is not informed by an empirical trajectory analysis of Al systems and capabilities. Ultimately, any framework for effectively regulating Al needs to keep pace with the technology itself.

To those who say there's no problem here to solve, or that California does not have a role in regulating potential national security implications of this technology, I disagree. A California-only approach may well be warranted - especially absent federal action by Congress - but it must be based on empirical evidence and science. The U.S. Al Safety Institute, under the National Institute of Science and Technology, is developing guidance on national security risks, informed by evidence-based approaches, to guard against demonstrable risks to public safety. Under an Executive Order I issued in September 2023, agencies within my Administration are performing risk analyses of the potential threats and vulnerabilities to California's critical infrastructure using Al. These are just a few examples of the many endeavors underway, led by experts, to inform policymakers on Al risk management practices that are rooted in science and fact. And endeavors like these have led to the introduction of over a dozen bills regulating specific, known risks posed by AI, that I have signed in the last 30 days.

I am committed to working with the Legislature, federal partners, technology experts, ethicists, and academia, to find the appropriate path forward, including legislation and regulation. Given the stakes - protecting against actual threats without unnecessarily thwarting the promise of this technology to advance the public good - we must get this right.

For these reasons, I cannot sign this bill.

Sincerely, Gavin Newsom.".

curious_cat_1632• 2 years ago

For those supporting this legislation: Would you like to share the specific harms to the public that this bill sought to address and prevent?

drcode2• 2 years ago

we'll probably create artificial superintelligence in the next few years

when that happens, it likely will not go well for humans

the specific harm is "human extinction"

curious_cat_1632• 2 years ago

Thanks for sharing your thoughts. I have a somewhat more optimistic take:

We will create some form of artificial intelligence (AI) in the next decade. However, the recent (i.e. over last five years) advances in natural language processing are not the ones that automatically lead to that. I think we need more fundamental breakthroughs than attention-based neural networks and all the tricks that have gone into say a Llama3. [1]

Even if we did develop an AI more aggressively, say by 2030, I am not sure if that automatically leads to extinction. Here, my optimism is guided by the last few times we had major technological innovations (i.e. Steam Engine, Rise of the Semiconductor based circuits, etc.).

To be sure, these technologies were disruptive, probably net-negative in terms of pure climate impact and led to near-term job losses and generally helped increase the power of people on the top of the capitalist food-chain. However, they also helped us accelerate into a world where we can produce more food than we need for feeding everybody on the planet in a given year. And generally, helped us see more of the world by connecting the planet both physically and virtually.

I am not a blind optimist though. I think we need regulation for harm prevention. The problem with that approach is that we need to understand what harms are being caused first and in some instances that might be too late. But, I don't think we start by taking a blanket position (as a society/polity) on how technological innovation should occur -- because a) we are not there yet and b) that's just a poor way to do science.

[1] https://arxiv.org/pdf/2407.21783

lasermike026• 2 years ago

They misspelled AI with Al. This Al guy sounds very dangerous.

creativenolo• 2 years ago

I wish Hooli wouldn’t

lvspiff• 2 years ago

Just give us the Asimov three laws to start rather than these nebulous "open model" language crap. Those should be the basic premise for any regulations that go into a bill in this area (but I'm sure the military contractors would never go for that with the whole "no harm to humans" thing - they want their terminator).

Gavin Newsom vetoes SB 1047

Related Stories