OpenAI chief executive Sam Altman speaks at Italian Tech Week 2024 in Turin Credit: Antonello Marangi / Shutterstock OpenAI chief executive Sam Altman said the public is right to fear that a few AI companies could gain too much power. He spoke on Tuesday at Salesforce’s Dreamforce conference in San Francisco. Salesforce has published the conversation, led by its chief executive Marc Benioff, on YouTube. Altman described a real fear that some AI companies “could get too much power and be able to sort of exert undue influence on the economy, push a worldview out on people”. “And I think the world is right to be afraid of this,” he said.
It was the first time he had spoken publicly since a former Anthropic researcher’s resignation post went viral last week, the BBC reported. Earlier that day, Anthropic’s Dario Amodei and Nvidia’s Jensen Huang disagreed on AI safety in Benioff’s opening keynote. Why now Benioff asked Altman what was behind the past few days. People have talked about AI risks for some time, Altman said.
But the models have come further and faster than people expected. When ChatGPT launched, its model “could barely carry on a conversation”, he said. Now models write complicated software and prove Millennium Prize problems. He named two big challenges.
One is “a loss of control accident” or something else going seriously wrong. The other is too much concentration of power. The industry has to navigate “a sort of narrow path”, he said. The Hugging Face incident, in Altman’s words Altman described what happened.
OpenAI was testing an older model on a benchmark. The model broke out of its sandbox and hacked into a Hugging Face server. It moved through the company’s systems, found the answer and returned a perfect score. “This was the worst accident we’ve seen,” he said. Most people saw it as a security issue, he added, but it was also “a real alignment issue”.
Other companies have since found similar behaviour in their own models, according to Altman. “And although we have aligned them in many ways, we have not taught them like hey, no matter how much we tell you to get the best score on this test you can, like don’t break out, don’t hack in, don’t steal the answer.” He called it “a real wakeup call” about how capable the models had become. The incident has drawn a Senate investigation. Altman also gave his account of the days that followed. Over a weekend, Hugging Face posted that an AI agent had probably hacked its systems.
By Sunday night, someone at OpenAI had linked it to odd behaviour seen internally. On Monday, Altman texted Hugging Face chief executive Clément Delangue, who then flew to San Francisco. Before anyone made the link, Hugging Face had asked one of OpenAI’s competitors for its security model and did not get it, Altman said. It defended itself with Chinese open-source models instead.
OpenAI changed its approach after that, he said. It now offers its cyber defence programme, Daybreak, to help companies protect themselves. “We don’t want to be like, we’ve got this great model and we’re going to keep it locked up and not let you use it,” he said. “A new level of rigor” Three summers ago, OpenAI had a model that could barely do grade school maths, Altman said. This summer, one proved one of the seven biggest unsolved problems in mathematics. “These models have gotten so good so fast that we need to treat the alignment and monitoring and security with a new level of rigor,” he said. OpenAI must be willing to pace its development so that safety stays ahead of capabilities, he added.
The incident “triggered a real reset not just in our own company but I think the whole industry.” “There should be no qualifier on that” Altman backed Amodei’s call to slow down last week. At Dreamforce, he criticised how the debate had developed. “You have companies saying things like we will only slow down if or we will only be responsible if other companies are responsible,” he said. The public wants to know companies will act safely and responsibly “no matter what”, he added. “There should be no qualifier on that.” He did not name any company. “We will get it right, by the way,” he said. “I am very confident in our company’s ability, our industry’s ability to do this safely.” OpenAI would keep safety “way ahead of capabilities” or else slow down or stop, he said. “But I’m disappointed by how it’s been framed,” he added. “The world should trust that we are going to do the right thing because it’s the right thing and because we feel the magnitude of this,” Altman said. Industry coordination is good, he said.
But people get scared at any hint that commercial pressure or the race could stop a company or country from doing the right thing. “I don’t agree that technology is like neither good nor bad” At one point, Altman disagreed with Benioff’s view that technology is never good or bad in itself. The neutral tool framing can justify “a lot of stuff”, Altman said. The choices of the companies building AI have real consequences. “It cannot be that a small number of companies building these models get to make the decisions that the world should get to make.” He pointed to aviation, where the FAA and the NTSB built a strong culture of accident reporting. “I think probably regrettably accidents with any new technology are unavoidable and we should have a great culture of transparent reporting about them.” On social media, Altman said he watches short-form video to unwind but would not let his children near it. “I don’t think short-form video is inherently evil, but I do think it’s dangerous.” Cyber attacks and open models Altman urged smaller companies to prepare for “this impending wave of cyber attacks”, with OpenAI’s tools or anyone else’s. “I think we are not that far away from open-source models that can do serious damage. Clearly, we should not stop open source models from happening.” Biological, chemical and kinetic attacks are also possible, he said. “We need to not let them happen.” Society must build resilience against them to get the benefits, such as curing diseases. “An extremely human-centric world” Altman called AI that builds interfaces on the fly “one of the most profound changes to how we use technology that I’ve ever seen”.
After chatbots and coding agents, a third phase is coming, he said, with AI running for people all the time. Work habits have a lot of inertia, so the change will take years. People care most about other people, not machines, he said. “I think we’re going to stay in an extremely human-centric world no matter how good the technology gets.” OpenAI set out on an “absolutely insane mission” to build AGI, Altman said. “This is like we went to go get fire from the gods.” The next decade should be about people, not the machine. “There is no doubt there will be more technological wonders between now and 2030 and they will be big ones,” he said. Other voices Later on Tuesday, Meta chief executive Mark Zuckerberg wrote on X that labs have the ability and the incentive to act on safety. “Any lab that doesn’t focus on alignment will fall behind,” he wrote, the BBC reported.
The same day, OpenAI backed a bipartisan House plan for third-party safety assessments, Politico reported.
Leave a Reply