If you want to know how risky something really is, do not ask the people selling it. Ask the people who would have to pay when it goes wrong.
Right now, those people are backing away from AI.
The underwriters are writing it out
Over the past year, some of the largest names in insurance moved to exclude AI from standard coverage. Berkshire Hathaway, Chubb, and Travelers sought state approval to exclude AI related damages from general liability policies, and regulators approved most of those requests. The Insurance Services Office, whose standard forms underpin the large majority of U.S. property and casualty policies, introduced optional AI endorsements that make the carve out easy to adopt at scale.
Berkley went further, introducing what was described as an absolute AI exclusion across directors and officers, errors and omissions, and fiduciary liability lines. The language reaches any actual or alleged use, deployment, or development of artificial intelligence. That is not a narrow carve out. It is a refusal to price the thing at all.
Notice what the insurers are not saying. They are not saying AI is uninsurable. A crop of specialists, from Munich Re to startups like Armilla, now sells standalone AI liability cover. The large carriers are saying something more uncomfortable: they do not know what it costs yet, and until they do, it is not in the standard policy.
The courts already decided whose problem it is
While insurers are stepping back, the legal question of who is responsible has been settled faster than most companies noticed.
In February 2024, a British Columbia tribunal ruled in Moffatt v. Air Canada. The airline's chatbot told a grieving customer he could apply for a bereavement fare after traveling. The actual policy said otherwise. Air Canada argued, in part, that the chatbot was a separate legal entity responsible for its own actions, and that the correct information was available elsewhere on its site.
The tribunal rejected both arguments. The chatbot was part of the company's website, so the company owned what it said, and customers are not expected to cross check one page of a site against another.
The damages were about $650. The principle was worth far more. If your software says it, you said it. There is no defense in arguing that the model made it up.
The gap in the middle
Put those two facts side by side. The liability for what an agent does sits with the company that deployed it. The traditional mechanism for transferring that liability, the insurance policy, is increasingly written to exclude it.
That gap is where every company deploying agents now operates. And agents widen it, because the Air Canada bot only talked. An agent acts. It updates records, sends messages, issues credits, commits the company to things. The exposure scales with every permission it is granted.
Most AI rollouts are still being evaluated on whether the output is good. That is the wrong first question for anything that acts. The first question is the one an underwriter would ask: what is the worst thing this can do, how often, and can you show me?
What the underwriter wants to see
Insurers do not refuse risk forever. They refuse risk they cannot measure. Coverage returns when there is evidence: logs of what the system did, limits on what it is allowed to do, records of when a human approved an action, and a history long enough to estimate how often things go wrong.
That is the uncomfortable takeaway for anyone running an AI program. The controls that feel like friction today, approval steps, action limits, audit trails, are exactly the evidence that will decide whether your risk can be transferred tomorrow. Companies that skip them are not moving faster. They are self insuring without knowing it.
The people whose job is to put a price on risk have looked at autonomous software and said: not yet. That is not a reason to stop. It is a reason to build as if you will have to prove every action your software took, because when something goes wrong, you will.