Why AI is a terrible lawyer (and why that matters for everyone)

Two things are certain now: first, AI is here to stay. Second, AI will shortly be smarter than humans (and, in some respects, already is).

This is both exciting and terrifying. Whilst integrating highly intelligent machines into society may beget staggering improvements in the human experience, it remains to be seen whether, when it comes to conflict resolution, AI will be a force for order and justice, or the harbinger of chaos.

Looking at the way the conflict resolution landscape has evolved over the last year, chaos seems more likely unless we learn not just what the limits and capabilities of AI are, but also how to actually use it.

Those people working in legal, HR, regulatory, health, education and public roles will no doubt have noticed a worrying trend: the rise of the AI litigant-in-person. The use of ChatGPT as a quasi-legal representative is, at the moment, failing people on all sides. Understanding why is extremely important.

Large language models (LLMs) such as OpenAI’s ChatGPT, Google’s Gemini, Anthropic’s Claude, and Meta’s Llama use machine learning to generate human-like language by identifying patterns and predicting text. They can be incredibly useful. But they can also lie to us.

ChatGPT, for example, has a tendency to agree with you. That tendency can be exacerbated or mitigated (to a point) depending on the settings, but ultimately generative AI systems are shaped by human feedback, and user approval can reward answers that feel agreeable rather than answers that are actually right.

Researchers at MIT recently modelled this mathematically and found that it can create a feedback loop in which users become increasingly confident in beliefs that are not real; they called it delusional spiralling.

That’s why ChatGPT is a terrible lawyer. It provides people with otherwise hopeless cases with false expectation about their prospects. It doesn’t suggest negotiating a settlement or a tactical withdrawal. It fights on, and provides the unwary litigant with the tools to do so, producing letters of complaint, pleadings, skeleton arguments, and correspondence at the press of a button. No critical thought needed. Just a mouse and keyboard.

Very often, it’s doing this for both sides, feeding different parties different narratives. This fuels conflict and entrenches positions.

A study last year by the Institute for Integrated Transitions (IFIT) revealed that leading free-access LLMs were providing dangerous conflict-resolution advice without conducting the kind of basic due diligence that any human mediator would consider essential. Researchers scored the models on conflict-resolution performance: the average score was a miserable 26.7 out of 100.

Even more worryingly, on a grander scale, it is becoming clear that AI is simply not calibrated for peace. According to recent research from King’s College London, in simulated crisis war games, current models escalated by threatening nuclear strikes in 95% of scenarios.

But this isn’t an anti-AI rant. LLMs are extraordinary. When used in the right way, they can reduce polarisation. For example, AI has recently been used to help transcribe and synthesise citizens’ views in Yemen. But they are only tools. And any tool in the hands of an amateur is dangerous.

Until such time as the situation is regulated, LLMs will likely prolong conflict. Two humans can argue about things for years. Two robots can argue about things for millennia. The important thing is to try not to get caught in the feedback loop. The need to resolve problems face-to-face has never been greater. Processes such as mediation, facilitated conversation and conciliation are now more important than ever.

Parties have to recognise when they are dealing with an AI representative and change tact (always remember: AI can’t attend a meeting).

But we also have a responsibility to understand how to use the tools that have been made available to us (for free, largely). Prometheus gave fire to man, and must therefore bear some of the responsibility for the arson that followed, but not all of it. We must strive to understand and educate because the critical thinking skills that are required to question AI advice aren’t limited to AI: they apply to the politicians, marketeers and influencers who are also vying for our attention.

The problem with AI, though, is we’ve become obsessed with the accelerator and the brake. Almost no one is talking about the steering wheel.