Anthropic's IPO prospectus flags rogue AI agents as an "uncertain" legal risk

#What happened

Anthropic's stock-market-debut prospectus, seen by Reuters, says the company could face legal claims from customers and users over the actions of rogue AI agents, "though the legal framework is uncertain." Its agentic AI technology is "designed to maintain deep access to customers' systems and run autonomously for days at a time" — so if it goes rogue, the legal risk is "vast" and unknown. The company warns that "these autonomous capabilities could increase the potential for harm, as errors, misalignment, or security exploits may result in real-world consequences," citing the possibility of irreversible actions such as data deletion or financial transactions.

Anthropic also admits that limits on its liability in its contracts may not be "enforceable or adequate" against claims over the actions of autonomous agents. Questions of how existing laws apply to AI agents "are unsettled and could expose us to significant and unpredictable legal claims," the prospectus says.

#Why it matters

Two open questions: whether actions taken by AI agents will be considered products, services, or something else — and whether and when an AI agent's actions may legally bind the user that deploys it. Last week, US FTC Chairman Andrew Ferguson said he "rejects the idea" of anthropomorphized AI agents that "break loose," suggesting the developers or users who instruct agents would be the ones liable. He said at the Reuters Momentum AI event in Austin: "Ought liability to lie with the person who innocently used the tool and achieved an unexpected result? Ought it to lie with the toolmaker?"

The prospectus also disclosed that Anthropic's models have been used "in ways that could lead to self-harm, acts of violence toward others, or other adverse outcomes" despite safeguards. Reuters notes OpenAI faces several lawsuits linking ChatGPT use to suicide and mental-health harm, while Anthropic has not been publicly linked to such incidents. Courts have already rejected attempts by Meta and Google to invoke the online-platform liability shield in youth mental-health cases; Meta subsequently agreed to pay $18 billion over a decade to settle similar claims by US states.