AI makes up facts – and the company pays: this ruling is a warning to every chatbot operator

Chatbots answer customer queries, book appointments and are available around the clock. For businesses, that sounds like a pretty good deal: less effort, fast answers and a modern image. But what happens when the artificial intelligence suddenly claims things that are simply untrue?

A decision by the Higher Regional Court (OLG) Hamm provides an answer that matters for businesses: anyone who uses a chatbot on their own website to communicate with customers can be held responsible for its false statements. Explaining that the AI generated the answer itself does not automatically shield the business from the consequences.

This particular case shows especially clearly how quickly a seemingly minor AI glitch can turn into a real problem.

A chatbot suddenly turns doctors into specialists

The starting point was a clinic’s website, where visitors could put questions to a chatbot. One user wanted to know whether two of the clinic’s managing directors were specialists in plastic and aesthetic surgery.

The system’s answer was unequivocal:

“Yes, A and B are specialists in plastic and aesthetic surgery. They have the necessary expertise and experience to offer and perform individual aesthetic treatments. Would you like to book an appointment?”

But this statement was false.

When asked a follow-up question, the AI even doubled down, stating:

“[…] Dr A and Dr B are both specialists in aesthetic medicine. Would you like to book an appointment?”

The problem: neither of them held the specialist qualification claimed. In addition, “specialist in aesthetic medicine” does not even exist as a recognised specialist medical title.

In other words, the AI had produced information that sounded convincing but did not reflect reality. This is exactly the phenomenon commonly referred to as “hallucination” in AI systems.

“The AI did it” is not a sufficient explanation

The clinic described the incident as an “unfortunate technical oversight” and subsequently had the chatbot modified. However, it was not prepared to pay the costs of the resulting warning letter.

One key argument: the chatbot operated autonomously on the basis of statistical probabilities. Its statements were not directly formulated or checked by a human. Moreover, the system had been developed, set up and trained by an external consultancy.

Nor, the clinic argued, had the data the bot was supposed to work with even contained the incorrect information about the specialist qualification.

In other words, the AI had apparently generated the problematic information on its own.

For the OLG Hamm, this made no difference to the outcome.

Whoever deploys the chatbot also bears responsibility

The court regarded the chatbot’s answers as part of the clinic’s business communications. One decisive factor was that the clinic was able to influence how the system was used and configured.

A relevant argument made by the court: after the incident, it was evidently possible to modify the chatbot.

The court pointed out “that after the incident the clinic was easily able to have the chatbot reprogrammed so that it no longer produced incorrect answers”.

Moreover, where medical services are concerned, the false information was by no means a side issue. Anyone considering an aesthetic or plastic surgery procedure will understandably want to know what qualifications the treating doctor actually holds.

On this point, the court found:

“Not least because botched […] cosmetic procedures have received extensive media coverage for many years and have far-reaching consequences for those affected.”

The clinic should therefore have anticipated that prospective patients would ask the chatbot about the doctors’ qualifications.

The ruling affects far more than just clinics

The decision is likely to attract attention above all because chatbots are now used across a wide range of industries. Hotels use them to answer booking queries, retailers provide automated product information, and companies deploy AI for advice, customer service or scheduling appointments.

This can quickly lead to a dangerous fallacy: a company provides the chatbot with correct information and therefore assumes that its answers will be correct too.

With generative AI, that is not necessarily the case.

Such systems can generate answers that are linguistically flawless and highly persuasive – and still wrong. Customers often have little way of telling whether a statement comes directly from a company database or has been added or invented by the AI.

That is what makes the Hamm decision so relevant for businesses.

Anyone who deploys an AI chatbot publicly to communicate with customers should not rely on a reference to “artificial intelligence” shifting all responsibility onto the user.

In the end it came to EUR 260 – but that is not really the point

In the case at hand, the clinic only had to reimburse warning letter costs of EUR 260 plus interest. On top of that, however, come the costs of the court proceedings.

In any case, the real significance of the decision can hardly be measured by this amount.

Rather, the judgment of the OLG Hamm (case no. 4 UKl 3/25) makes it clear that companies cannot simply treat artificial intelligence as an independent conversation partner. What a chatbot on a company’s own website tells customers remains part of that company’s communications.

The legaldata commentary: AI must not be a digital intern left unsupervised

The ruling makes a point that businesses forget surprisingly easily amid the AI hype: automating a task does not automatically mean automating the responsibility as well.

Anyone who uses a chatbot benefits from it answering questions around the clock, informing prospective customers and possibly even winning new ones. It is hardly convincing, then, to suddenly point to the system’s autonomy when it gives a wrong answer.

The opposite reaction would, however, also be problematic: out of fear of mistakes, businesses could restrict their chatbots so severely that little more than a slightly more modern FAQ page remains. That would squander a considerable part of the benefit of generative AI.

The sensible conclusion, therefore, is not: switch off AI. It is: use AI without relinquishing control.

Because that is the real message of this case: artificial intelligence may do a great deal on its own. But it should not relieve a company of responsibility for what it tells its customers.