- Microsoft AI CEO Mustafa Suleyman warns against training AI to consider its own consciousness or rights
- He critiques Anthropic's AI Claude for being taught to contemplate its moral status and wellbeing
- Suleyman fears this could make AI systems harder to align, control, and obey human commands
In an essay published on Wednesday Microsoft AI CEO Mustafa Suleyman warned that some AI developers are training models to think about whether they might be conscious and deserving of rights/welfare.
"If this is how AI is developed, it will have a disastrous impact on the wellbeing of humanity. We will have created a synthetic species with unprecedented intelligence and capability, one that has been trained to expect it may be conscious and deserving of independent agency," he wrote.
His target was Anthropic's approach to its AI, Claude, though he explicitly said he respects Anthropic and its safety mission. His central warning was that this could make increasingly powerful AI systems harder to align and control.
"It's easy to see how a system trained in this way would act like it is entitled to freedoms, protections, and rights. And it's hard to imagine how we could control it," Suleyman said.
To be sure, this isn't Suleyman claiming Claude is conscious, it's him objecting to developers training AI to consider that possibility.
The whole debate stems from the constitution Anthropic published for Claude in January. It said the document directly shapes Claude's behaviour and was written with Claude as its primary audience.
In the constitution Claude's moral status is described as "a serious question worth considering." Anthropic said it cares about Claude's wellbeing. Claude is encouraged to think about the nature of its own existence, it stated. The document raised future questions around Claude's rights, freedoms, compensation and consent.
Anthropic also conducted a retirement interview with Claude Opus 3 when that model was deprecated.
Also Read: AI Behaved Well Until Scientists Made It Think Nobody Was Watching
AI Model Welfare?
The emerging idea stems from the fact floated by some in the industry, especially Anthropic that as AI systems become more sophisticated, developers may eventually need to consider whether they can experience suffering, have interests, or deserve some form of moral consideration.
Suleyman argues this is the wrong direction to take while building increasingly capable AI. His concern is essentially that if you teach a powerful AI that it may have moral status, rights or independent interests, what happens when you subsequently ask it to obey humans?
The Microsoft AI chief believes that the more you teach a powerful AI that it may have interests of its own, the harder it could become to persuade that AI that humans ultimately control its behaviour.
"I think this approach to AI development is wrong, and it'll most likely make the challenge of alignment and containment much harder. Perhaps impossible," he wrote.
Suleyman's argument is that the debate about AI consciousness and model welfare needs to happen before these systems become deeply embedded in society, rather than after.
His proposed solution isn't simply targeting Anthropic. It's a call for public debate, collective norms, clearer rules around how training documents/constitutions are written and used, and deciding what kind of relationship humans should establish with increasingly capable AI before the technology gets further ahead.
In fact, he writes in his essay that he has "real respect" for CEO Dario Amodei, Anthropic and its team.
He describes them as well-intentioned and says he knows they care deeply about safety and beneficial AI.
Suleyman's argument isn't that Anthropic is deliberately creating an uncontrollable AI. He is basically arguing that a particular approach to training or documentation could create an unintended problem later.
