Feed: Slashdot
Feed Link: https://slashdot.org/
---

Title: AI Insiders Issue New Warnings - Including Former Anthropic Engineer
Jacob Coxon

Link: https://slashdot.org/story/26/09/17/2042242/a...

Neither OpenAI or Anthropic is acting responsibly, warned AI researcher Jacob
Coxon when resigning last week from Anthropic. But just hours earlier, OpenAI
VP of Research Aidan Clark had posted "For the first time, I am asking myself
if things are moving too fast. I'm honestly not sure, but I am sure that it
would be good for us to have an answer to 'What would a successful pace look
like?'" CNN noted Wednesday they're just some of the many AI insiders who are
now concerned about the speed of research. Coxon even wrote that "The people
building AI earnestly believe that it could kill us all by the end of the
decade. This is not a marketing stunt. If anything, many executives and
senior researchers will couch their phrasing in the press to sound sensible -
but I hear the same people express fear privately." In 10 days since, Coxon's
post has been viewed more than 170 million times, warning that OpenAI and
Anthropic are "racing straight to self-improving superintelligence and
gambling with our lives." It's part of what CNN now calls "pressure on AI
companies and governments to do something about the pace of development and
safety," where "much of that pressure is coming from staffers inside the
companies." One staffer at a top AI company told CNN the fears of how AI
could hurt humanity keeps them up at night. Another researcher who recently
left a different AI company said it's a common subject of conversation at
parties and social events in Silicon Valley. "You can't spend more than a few
hours in this community without realizing that a very substantial number of
people are really pretty worried about these sorts of outcomes," said the
researcher. "A majority would say there's some chance of it killing
everyone...." Dozens of Coxon's colleagues in the AI industry publicly
supported his statements, with some making even more dire predictions...
OpenAI CEO Sam Altman and SpaceXAI CEO Elon Musk agreed to Amodei's proposal
to embed independent watchdogs at the AI companies... "I would burn my equity
to the ground in a heartbeat for a 1% higher chance we make it out of this
situation alive. I expect a great many of my colleagues across the industry
would as well," wrote Drake Thomas, who works on AI safety at Anthropic. "I
promise you, we are actually just f**king scared, it's not galaxy brained
marketing..." AI staffers told CNN that they fear that as AI gets better at
training and improving itself, their leverage goes down, prompting today's
urgency. "As we get into this recursive self-improvement loop, I think that
might substantially reduce staff's bargaining power, because frankly, you'll
be able to replace many of the staff with models that can do as good a job,"
the researcher who recently left a top AI company said. Coxon posted Tuesday
on X that Anthropic "largely initiated" the race to recursively self-
improving AI, justifying it with "a belief in its inevitability." And then
OpenAI "had to shed a bunch of dead weight like Sora," as he sees it,
"because Anthropic was going for the jugular." (In fact, his specific
disagreement with Anthropic's leadership was whether China and the U.S. could
ever negotiate an alternative to their current race towards self-improving
AI...) In an informal "Ask Me Anything", Coxon responded to a question about
when we'd see a Terminator-like malevolent AI by saying that "Skynet could go
live in the 2030s if we aren't careful. ai-2027.com is a modern skynet story
written a year ago and it's on track so far." Yet while AI development risks
an end to humankind, "I do think that if we go slower we can take risk to
0%... But this requires radical action." He acknowledged there was still a
possibility that the steady increases to model intelligence could suddenly
plateau, but "They haven't so far, and it's just a few more steps up the
ladder to hit the finish line." To avoid stifling innovation, he recommends
"prioritizing applications that actually improve people's lives [like
healthcare discoveries], rather than immediately going for raw economic value
or intelligence." But isn't mass unemployment a more pressing threat? "Things
are coming so fast that unemployment would be a brief preliminary to deadly
superintelligence." To people who feel disempowered, Coxon offered his
solidarity. "I also feel disempowered. Part of resigning was a feeling of
hopelessness about the future. I would say - keep your eyes open as things
get crazier and advocate for increased transparency into AI companies." When
asked if he'd start his own company now, Coxon said he had "No idea what I'm
doing next."

Read more of this story at Slashdot.

---
VRSS v2.1.180528