Does California's AI bill go too far or fall short? It depends who you ask.

California State Sen. Scott Wiener speaks at an event last year. Wiener is the driving force behind a bill regulating AI in the California legislature.

California State Sen. Scott Wiener speaks at an event last year. Wiener is the driving force behind a bill regulating AI in the California legislature. Kimberly White via Getty Images

 

Connecting state and local government leaders

The legislation requires developers of large AI systems to test if they can be used in various extreme scenarios. It has support as well as plenty of detractors in the state’s large tech community.

A bill regulating artificial intelligence safety in California pitting powerful politicians and experienced academics against some of the world’s biggest technology companies appears to be moving towards final passage. But Gov. Gavin Newsom, who has warned previously against overregulating AI, has given no public indication of whether he will sign it.

The legislation could be a model for how other states and the federal government handle the tension between their desire to regulate AI with businesses' wish to be left alone to innovate—and make money—especially when those businesses are among the most powerful in the country. Colorado passed sweeping legislation of its own in May and already looks set to amend it further amid criticism from tech companies and business groups.

The sprawling legislation, sponsored by State Sen. Scott Wiener, whose district includes San Francisco, would require the developers of the biggest AI systems—those which cost over $100 million to train—to test whether they could be used for attacks on critical infrastructure, cyberattacks or terrorism, or to make weapons.

It also would establish CalCompute, a public “cloud” of computers to help host and build AI tools, offer cloud computing services, promote equitable technology development and research “the safe and secure deployment of large-scale artificial intelligence models,” the bill says. Wiener’s legislation also would offer new protections to whistleblowers at companies building AI tools, including contractors.

The latter provision comes on the heels of Daniel Kokotajlo, a former employee of OpenAI, alleging the company was being too reckless in creating its ChatGPT generative AI chatbot and violating its safety protocols. He was subject to the company’s extremely strict off-boarding protocols.

The bill has already passed the California Senate and is moving through committees in the California Assembly.

“With Congress not moving forward, and with the future of the Biden administration's executive order in doubt, California has an indispensable role to play in ensuring that we develop this extremely powerful technology with basic safety guardrails so that we can allow society to experience AI's significant, massive benefits in a safe way,” Wiener said in a speech on the Senate floor in May.

The bill has received bipartisan support and comes as California has looked to take a leadership role on AI among state governments, including by experimenting with generative AI in government operations.

But even leading AI researchers who support the legislation say it could have gone further. In a letter to state leaders earlier this month, eminent AI researchers Geoffrey Hinton of the University of Toronto, Yoshua Bengio of Universite de Montreal and Stuart Russell of the University of California at Berkeley, as well as Lawrence Lessig of Harvard Law School, warned of the “severe risks posed by the next generation of AI if it is developed without sufficient care and oversight.” 

“It doesn’t have a licensing regime, it doesn’t require companies to receive permission from a government agency before training or deploying a model, it relies on company self-assessments of risk, and it doesn’t even hold companies strictly liable in the event that a catastrophe does occur,” they wrote. “Relative to the scale of risks we are facing, this is a remarkably light-touch piece of legislation.”

Others are not so sure. Fei-Fei Li, the co-director of Stanford’s Human-Centered AI Institute who is regarded as the “Godmother of AI,” said in a commentary that while the bill is “well-meaning,” its restrictions on open source development could harm innovation in California and beyond. In response, Wiener said in a written statement that the bill does not ban open sourcing, but allows the Attorney General to initiate enforcement proceedings in limited circumstances.

The California Chamber of Commerce has come out against the bill, as have companies including Facebook parent company Meta, venture capital firm Andressen Horowitz, and a coalition of think tanks, business and political leaders, including the conservative American Legislative Exchange Council. 

The latter coalition said having developers guarantee that their AI models cannot be used for harmful purposes even before they begin training them is an “unreasonable and impractical standard.” They also argue that complying with various safety standards “would be expensive and time consuming for many AI companies,” and so might force them to leave the state.

The bill also appears to be a flashpoint in California’s political future. Rep. Nancy Pelosi, the House Speaker Emerita and a powerful player in California—and national—Democratic politics released a mid-August statement, saying Wiener’s bill “is well-intentioned but ill informed” and “would stifle innovation and will harm the U.S. AI ecosystem.” Wiener is regarded as a contender for Pelosi’s San Francisco-area House seat when she retires.

Pelosi echoed many of the concerns raised by Rep. Zoe Lofgren, ranking member on the House Science Committee and another California Democrat.

In a written response to those critiques, Wiener said he rejects “the false claim that in order to innovate, we must leave safety solely in the hands of technology companies and venture capitalists.”

“While a large majority of people innovating in the AI space are highly ethical people who want to do right by society, we’ve also learned the hard way over the years that pure industry self-regulation doesn’t work out well for society,” Wiener continued.

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.