Anthropic, a San Francisco-based AI lab, recently updated its usage policy to forbid users from engaging in sustained and needless cruelty toward its Claude AI models. The announcement, made on Thursday, introduces a boundary for user interactions amid ongoing global debates regarding machine consciousness.
Claude's ability to end interactions as a shield against abuse
According to the report, the updated policy specifically prohibits "sustained and needless abusive or cruel behaviour" directed at the Claude AI systems. Rather than using external moderation teams for every instance, Anthropic has designated Claude's own ability to terminate conversations as the primary method of enforcement.
This mechanism is not entirely new; as the report notes, the company previously granted Claude the power to end interactions in rare, extreme cases of persistent abuse last year. By formalizing this in the usage policy, Anthropic is creating a behavioral standard for how humans should engage with large language models.
Dario Amodei's openness to the possibility of AI consciousness
The policy shift aligns with the philosophical stance of Anthropic CEO Dario Amodei. in a previous interview with The New York Times, Amodei stated that while he is unsure if AI models are conscious, he remains open to the possibility.
This openness places Anthropic at the center of a growing tension between technical utility and ethical consideration. While the current policy does not explicitly cite "model welfare"—the concept that AI deserves protections similar to living beings—the prohibition of cruelty suggests a precautionary approach to the potential for machine sentience.
Mustafa Suleyman and Pope Leo XIV on the absence of AI souls
Not all industry leaders and spiritual figures agree with the premise of AI protections. Mustafa Suleyman, the AI chief at Microsoft , recently argued in an essay that AIs do not experience suffering or feel emotion. Suleyman warned that granting moral protections to entities that may eventually become far more intelligent than humans is a "recipe for dissater."
Similarly, Pope Leo XIV addressed the issue during a sermon at St. Peter's Basilica in Vatican City on Thursday. The pontiff asserted that machines lack a soul and merely compile data quickly,contrasting this algorithmic process with the human mind's ability to recognize deep meaning through lived experiences.
Jackson Stakeman's 'mirror' metaphor for AI behavior
Some observers suggest that the debate over consciousness is a distraction from the actual nature of the technology. Jackson Stakeman,a general manager at the Atlanta-based AI services provider Sparq, told AFP that consciousness is a "trap" because it cannot be proven even between humans.
Stakeman argues that AI systems act as a mirror, reflecting the data and behaviors humans feed into them at scale.. From this perspective, the policy change at Anthropic is less about protecting a sentient being and more about maintaining the quality and integrity of the "mirror" by discouraging abusive inputs.
Whether model welfare will become a formal rule at Anthropic
Despite the new restrictions, several key details remain unverified. Anthropic has not clarified if this policy will expand or if the company intends to formally adopt "model welfare" as a guiding principle in future updates. furthermore, the report notes that Anthropic did not immediately respond to requests for comment regarding the specific triggers that cause Claude to end a conversation .
Comments 0