Hallucinations

In August twenty twenty-five, a King's Counsel — about as senior as a barrister gets — handed a judge in the Victorian Supreme Court submissions containing quotes from a parliamentary speech that was never given, and citations to judgments that had never been written. A murder trial. He'd used AI, and he hadn't checked it. That's the top of the legal profession — the person in the room with the most to lose from getting it wrong. This didn't happen because he was careless. It happened because the answer looked right.

The word for this is hallucination, which is a poor name, because it sounds like a glitch. It isn't. These tools work by predicting what a good answer looks like. And a court case that doesn't exist has the same shape as one that does — same format, same court, same year. There's nothing in the machinery that flags the edge of what it knows. So it fills the gap and hands it over in the same steady voice it uses for everything else.

A research team in Paris keeps a database of court filings worldwide caught containing AI-invented citations. In mid twenty twenty-five it held around two hundred. Twelve months later, more than fifteen hundred — and climbing by several a day. The penalties climbed too. The first sanctioned case, in twenty twenty-three, drew a five thousand dollar fine; by twenty twenty-six courts were handing out fifteen thousand per lawyer and suspending practising certificates. In Australia, one solicitor lost the right to run his own practice for two years.

You're not filing court documents. The pattern still applies, though, anywhere the detail is specific and checking it is dull. A statistic in a board paper. A quote attributed to a real person. A date, a price, a policy name. And it gets worse the more obscure your question is — ask about something niche and you're more likely to get a confident invention than an admission of not knowing.

Three things help. Give it the material rather than asking it to recall — paste in the report, upload the document, point it at the page. Working from something in front of it is a different task, and a far more reliable one. Then ask for the source, and open the source. Don't just request a link — click it. A made-up citation often arrives with a made-up link attached. And decide before you start which parts of the answer you couldn't afford to be wrong about, because those are the ones worth your time.

One last thing. The tone never shifts. Correct, guessing or inventing, it all arrives in the same measured voice. Every instinct you have for catching a person who's bluffing is useless here, because those instincts read hesitation, and there isn't any.

One thing to do this week. Next time an answer comes back with a number or a source in it, open the source before you use it. Either it checks out, and you've learned something useful about where this tool is strong — or it doesn't, and you'll never read a confident answer the same way again.