California's AI safety bill has passed. What does it mean for open source AI?

It’s finally happened - California’s AI safety bill (SB-1047) was passed by the State Legislature. The bill now sits on Governor Gavin Newsom’s desk and is one rubber stamp away from becoming law.

So, I guess that’s it? Time to pack up and move onto something else… I mean, for months the AI chattering classes have been telling us how disastrous this bill will be, how it will restrain innovation and kill off open source AI.

Well, not so fast. You see, this story is as old as the hills. Every new and disruptive technology faces similar scrutiny. The legislators have heard all the arguments on both sides a thousand times over, and their job is to strike a balance of protecting society without dampening innovation.

So, now that the act is a formality away from becoming law, let’s take a step back and look at what this bill really means for AI. In this article I’ll offer me layman’s interpretation of what the act covers (and doesn’t cover), and what the real world implications for AI will be, particularly open source AI.

What is SB-1047

To give the bill it’s full name, SB-1047 is the “Safe and Secure Innovation for Frontier Artificial Intelligence Models Act”. The purpose of the bill is to ensure safe and secure innovation for frontier artificial intelligence models in California.

The legislation is of global significance because AI is a highly disruptive emerging technology and the laws are playing catch up. Regulators around the World are watching California carefully, and this bill will likely set in motion a precedence that other states and countries will follow.

And, if you take the view as I do, that AI represents a new industrial age, then the US getting this wrong could hand its technological lead to its competitors, with geopolitical consequences.

With the stakes so high, it’s natural that those in the AI industry are anxious of premature-regulation. Of course, the devil is in the details, so let’s dive into it and explore some of the nuance that is often missed in discussion.

Breaking down the bill

First of all, this bill sets out a couple of crucial definitions that define the scope and purpose of the bill.

”Covered models”

The bill defines specific thresholds for what models are covered:

• An AI model trained using computing power greater than 10^26 floating point operations (FLOPS), with a cost exceeding $100 million.
• OR, a model created by “fine-tuning” a covered model using computing power greater than 10^25 FLOPS, with a cost exceeding $10 million.

10^26 is a number with more zeros than my brain can handle, but $100 million I can just about cope with. It’s a lot of money.

”Critical harm”

The bill provides a specific definition of “critical harm”, as any of the following harms “materially” enabled by a covered model.

• The creation or use of chemical, biological, radiological, or nuclear weapons resulting in mass casualties.
• Mass casualties or at least $500 million of damage resulting from cyberattacks on critical infrastructure.
• Mass casualties or at least $500 million of damage resulting from an AI model acting with limited human oversight that would constitute a crime if committed by a human.
• Other grave harms to public safety and security of “comparable severity” to the above.

The bill also explicitly excludes:

• Harms caused by information that is otherwise reasonably publicly accessible.
• Harms caused by a covered model combined with other software if the covered model didn’t materially contribute to the harm.

The thresholds for both definitions get reviewed after 1 January 2027, and monetary amounts adjusted annually for inflation.

These two thresholds quite clearly show that this is a bill designed to capture very large, very powerful AI models, and is specifically focused on severe, large scale harms like terrorist events or cyberattacks on critical infrastructure.

Requirements for developers

The bill outlines several requirements fort developers of covered models:

• Developers must register their covered models with the California Department of Technology.
• Comprehensive evaluations must be conducted to identify potential critical harms.
• Appropriate measures must be implemented to prevent unauthorised access, misuse, or unsafe modifications of the model.
• Any critical harms or near misses must be promptly reported to the authorities.
• Developers must maintain the ability to cease operation of the model if necessary to prevent critical harm.

As well as creating lots of paperwork for everyone involved, these requirements aim to ensure responsible development and deployment of frontier models, while maintaining transparency and accountability.

What SB-1047 is not?

I suspect most people would agree that ensuring terrorists can’t use AI to learn how to build and use biological weapons is, on balance, a good thing. But, I also think people have more immediate concerns about AI. Things like:

• Mass unemployment and re-skilling in the face of automation.
• The environment and the huge amounts of energy consumed by training AI models.
• Copyright issues and the industrial scraping of internet data for training AI models.
• Easy access to tools that can create highly realistic deep-fakes and deceiving images.

Guess what? SB-1047 has nothing to say about any of that. The bill is as minimal in its scope as you could expect. I mean, if you don’t draw the red line at chemical and nuclear weapons, then where do you?

Impacts on open source AI

Whilst SB-1047 does not specifically prohibit open weight models, it creates a regulatory environment that could discourage developers of covered models from releasing them openly. For example:

“the developer shall […] prevent unauthorized access to, misuse of, or unsafe post-training modifications of the covered model”

the developer shall […] implement appropriate safeguards to prevent the covered model and covered model derivatives from causing or materially enabling a critical harm

a developer shall not […] make a covered model or a covered model derivative available for commercial or public, or foreseeably public, use, if there is an unreasonable risk that the covered model or covered model derivative will cause or materially enable a critical harm

Taken at face value, these provisions seem to be incompatible with a fully open release. Developer’s of closed models can design safeguards into their own systems and infrastructure, but once a model is out in the wild the developers lose all practical control - and they suddenly become liable for the actions of others.

Safeguarding AI is hard

You might be thinking, why can’t the models just be released without the capability of teaching us how to do things like build biological weapons in the first place?

It seems a reasonable question. After all, a model just reflects its training data, so just don’t train it on the bad stuff, right? Unfortunately, it’s not as simple as that.

• Large language models can develop emergent capabilities they weren’t explicitly trained for. They can combine information in novel and unpredictable ways, leading to unexpected capabilities.
• Much information that could be used for harmful purposes, also has legitimate uses. For example, knowledge about chemistry or biology that could be used for important medical breakthroughs, could also be used to create weapons.
• The sheer vastness of possible inputs and outputs makes it practically impossible to anticipate all potential harmful uses.
• Users will always try (and often succeed) in circumventing trained in safety measures through clever prompting techniques.
• Some potential harms, such as cyberattacks, don’t even require any specific “dangerous” training data to be a threat.

There is a whole area of research around “model alignment”. Ensuring AI systems behave in ways that are aligned with human values and intentions is complex and not fully solved.

The bill recognises these complexities and so focuses on risk assessments, monitoring and auditing, and the ability to “shut down” the model. All of which becomes very difficult once that model is released openly and out in the wild.

Is this regulatory capture?

It’s worth noting that as it stands today, there probably isn’t any open-weight model that would fall within the act’s definition of a “covered model”. Even Llama 3.1 405B (the largest open weight model to date) probably falls short.

But, it’s not far off. And there must now be question marks over whether Meta will ever release an even larger model openly.

Time will tell, and there’s certainly enough “unreasonable risk” and “materially contribute” type language in the bill to give Meta’s lawyers something to chew on.

But, if this does discourage future frontier models being released openly, then that’s a scenario that clearly benefits the developers of closed, proprietary models, and all but closes the door to new entrants to the market. That would point towards a future where access to the very best and most powerful AI models is exclusively controlled by a handful of all-powerful tech giants.

Winners and losers

At first glance, SB-1047 might seem like a win for those advocating for stricter AI regulation. However, a closer look reveals a more nuanced picture.

The bill’s light-touch approach and narrow focus means that many in the AI industry, particularly the big AI labs, may be breathing a sigh of relief. Earlier in the week, Anthropic’s CEO Dario Amodei said of the bill, “we believe its benefits likely outweigh its costs” and Elon Musk tweeted that “California should probably pass the SB-1047 AI safety bill.” - statements that suggest to me that they were happy with what was on the table.

On the other hand, those hoping for more comprehensive regulation may feel the bill does not go far enough. The focus on preventing catastrophic harms, while important, leaves many day-to-day AI ethics issues unaddressed.

The most significant impact is the bill’s potential to discourage the release of fully open, frontier-level models. While not explicitly prohibited, the bill’s provisions will make it challenging to release covered models openly. Whilst I don’t believe this in any way “kills open source AI”, as it only targets frontier models, this does potentially take us to a future where the most advanced AI capabilities are concentrated in the hands of a few large corporations.

In essence, SB-1047 strikes a delicate balance. And perhaps the legislators wouldn’t be doing their job if both sides of the debate weren’t at least a little bit unhappy.