Meet Jacob Coxon, Ex-Anthropic Researcher Who Sparked A Debate On AI Safety

A little-known former Anthropic researcher has sparked a global debate with a warning on the potential dangers of artificial intelligence. Not many had heard about Jacob Coxon until he quit the artificial giant, saying the AI researchers believed the technology "could kill us all by the end of the decade."

He barely had an online footprint and was not a major player in the community advocating for AI safety. Now, the 27-year-old British researcher is at the centre of discussion surrounding AI and its risks.

Coxon briefly tried his hand at commodity training before shifting to Silicon Valley, as per the Wall Street Journal.

Years later, the Cambridge alum quit his job at Anthropic and said that labs racing to build AI were "gambling with our lives."

Who Is Jacob Coxon?

Coxon is the son of a professor of medieval German literature. As a high schooler, he was selected for the International Mathematical Olympiad's UK team. He won a silver medal in 2016 and a bronze the following year.

He studied math at Cambridge. His graduation in 2020 coincided with the launch of OpenAI's GPT-3, a breakthrough model.

"GPT-3 was really the 'wow' moment," Coxon told WSJ.

He joined OpenAI in 2023. He specialised in pretraining, which involved feeding AI models with massive amounts of text, images, and code before they could be used for specific tasks.

Will DePue, a former colleague, described Coxon as a "normal researcher".

Over time, Coxon started getting worried about superintelligence that could go beyond human capabilities. However, he assumed governments would slow down the pace of development, if needed.

While he wasn't ready to work on AI safety, Coxon did keep an eye on AI's progress in math as the models developed faster than he expected.

In May this year, Coxon started working at Anthropic, believing that it was more transparent about AI safety than rival OpenAI.

However, incidents like the Hugging Face attack at OpenAI made the researcher nervous.

Coxon said he went to his bosses at Anthropic in recent weeks to talk about his concerns.

He considered shifting to a role working on AI safety before deciding to quit. "Even working on safety at Anthropic felt like being complicit in the race," he said.

Before finalising his departure, Coxon met with Kokotajlo, an old colleague and former OpenAI safety team member who is now an AI-safety researcher for advice. Kokotajlo assured the 27-year-old that he was making the right decision.

After he quit and posted his message about the dangers of AI, Coxon's comments were echoed by the leader of Anthropic's team responsible for making sure AI does what humans intend.

Soon after, his comments caused an uproar, with fears of an AI-driven apocalypse taking hold of people.

Coxon has continued his warnings, stating that trying to build super-intelligent AI is similar to summoning an alien species, as it is something researchers cannot fully control.