A few years ago it became common to read about lawyers who had filed briefs containing case citations that did not exist [1]. The cases had plausible names, plausible reporters and page numbers, and quotations that read like judicial prose [1]. They had been produced by a chatbot asked to find supporting authority, and the attorneys who filed them had not checked them against a legal database [1]. What strikes me about these episodes is how ordinary the failure looked. The fabricated citations arrived in the same register as accurate ones, and that register was persuasive enough to carry them past people trained to be skeptical.
That is the problem this post is about. Generative AI systems present their output with a uniform fluency that bears no reliable relationship to whether the output is correct [2]. In most professional settings, people have learned to read confidence as a signal. A colleague who hedges is telling you something, and a colleague who speaks flatly is usually staking a reputation on the claim. Language models break that link, and organizations that adopt them without replacing the lost signal are taking on a risk they may not see until it costs them. The remedy, I will argue, is a discipline of evaluating claims by their support rather than their tone, and I want to propose a simple framework for doing that.
Fluent text is evidence of fluency and nothing else
A large language model generates text by predicting, token by token, what is likely to come next given its input and the patterns it learned in training [2]. Most of the writing it learned from is declarative and assured, so assured prose is its natural output. Legal memos, medical references, and analyst reports are written in a confident register, and the model reproduces the register along with the content. The confidence is therefore a feature of style. It indicates that the text resembles what usually appears in that context, which is a different question from whether its specific claims are true, and a fabricated citation is as easy for the model to produce as a real one because both have the shape of a citation.
Researchers describe the gap between expressed and actual accuracy as a problem of calibration. A well-calibrated forecaster who assigns a 70 percent probability to a set of events should see about 70 percent of them occur, and there is evidence that language models, particularly after the fine-tuning used to make them helpful conversational assistants, are often less well calibrated than their underlying probabilities would suggest [2]. Asking a model to state its confidence does not reliably fix this, since the stated confidence is itself generated text [3]. Some systems add retrieval, drawing answers from documents supplied at the time of the query, and that helps by giving a reader something to check. Even so, retrieval does not guarantee that the model has read the document correctly or quoted it faithfully [4].
The same failure carries different costs in each field
The mechanism is the same everywhere, but the damage depends on what a wrong answer sets in motion. In law, the risk is an invented authority, a misstated holding, or a miscalculated filing deadline, any of which can lead to sanctions, a lost motion, or a malpractice claim. A confident medical summary might omit a contraindication or state a dose in the wrong unit, and there the error lands on a patient rather than a document. Finance has its own version. A model summarizing a quarterly filing can transpose a figure or attribute a number to the wrong period, and an analyst under time pressure may carry that figure into a model that drives an investment decision.
Education and ordinary business show a quieter form of the problem. A student who asks a tutoring tool to explain a historical event receives an answer whose errors are invisible precisely because the student does not yet know the subject, and the confident tone teaches trust in the source along with the false fact. For a business example, consider a hypothetical case at Northwind Labs, an invented company I will use for illustration. A procurement manager asks an AI assistant to summarize a supplier agreement and is told the contract can be terminated on 30 days' notice. The agreement actually requires 90 days and written notice delivered to a specified address, and Northwind discovers this only after it has signed with a new supplier and is paying both.
What these cases share is that the error could not be seen in the output. Nothing in the tone of the answer distinguished the wrong termination clause from a right one, and nothing in the prose of the fabricated brief distinguished it from a real one. The cost of each mistake was set by what happened downstream. The person deciding whether to act, however, had no signal from the text itself about how much checking it deserved.
Confidence persuades because reviewing is harder than it looks
It would be convenient to say that the answer is human review, and review is part of the answer, but it has weaknesses of its own. Psychologists have long observed that people tend to judge easily processed information as more likely to be true, a pattern often called the fluency effect [5]. Studies of automation bias find that people working with automated aids tend to accept their outputs and to miss errors the aid did not flag [6]. Machine-generated text is about as easy to process as text gets, and it comes from a tool the reviewer has chosen to use. Both tendencies therefore push in the same direction.
The structure of the work makes this worse. When a person writes a memo, the act of writing forces them to find each source and decide what it says. When a person reviews a generated memo, they start from a finished product that already looks complete, and the task quietly becomes reading for problems rather than rebuilding the argument. Errors of omission are especially hard to catch this way, because a missing contraindication or an unmentioned clause leaves no trace on the page. As generated drafts multiply faster than anyone can read them closely, the reviewer's attention is spread across more material that all sounds equally sure.
A framework for evaluating a conclusion before acting on it
The practical question is what to put in place of the signal that confidence used to provide. My answer is to treat every AI-generated conclusion as a set of claims that must each earn trust, and to scale the effort to the consequence of being wrong. The framework I propose has five steps, applied in order.
Decompose the output into the specific factual claims it depends on, such as a citation, a figure, a dose, a date, or a contractual term, and separate them from the reasoning that connects them.
Trace each claim to a source that exists outside the model, and open that source rather than accepting a reference to it.
Check the claim against the source directly, confirming that the source says what the output says it says, in the same terms and for the same period, jurisdiction, or population.
Assess what an error would cost and whether the action it supports can be reversed, and set the depth of checking accordingly.
Assign a named person who is accountable for the decision, and record which claims were verified and which were accepted on judgment.
The first step does most of the work, because a conclusion that cannot be broken into checkable claims cannot be evaluated at all, only believed or disbelieved. The second and third steps address the failure in the legal cases directly: a citation is not support until someone has read the cited text. The fourth step keeps the framework usable. A draft email to a colleague does not need the scrutiny that a dosing recommendation or a court filing needs, and a process that pretends otherwise will be quietly ignored. The fifth step closes the loop, because accountability restores the old signal; when a person puts their name to a conclusion, confidence once again means that someone has staked something on it.
Verification establishes support, not truth
I want to be careful about what this framework achieves. Checking a claim against a source confirms that the source supports the claim, which is narrower than confirming that the claim is true. Sources can be outdated or simply wrong, readers can misread them, and a contract summary that matches the contract says nothing about whether that contract was the right one to rely on. Verification also cannot catch what a generated answer leaves out. The reviewer still needs enough domain knowledge to notice what ought to be there and is not.
There is a design lesson here for the people who build these systems. Tools that show their sources, quote the passage they relied on, and make it easy to open the original reduce the cost of the second and third steps considerably, and that changes how often people actually perform them. Tools that present a polished answer with no visible provenance push the cost back onto the user, who will often decline to pay it. If I were evaluating an AI product for a high-consequence workflow, the first thing I would ask is how easy it makes checking its work.
The open question, and one I do not think is settled, is whether models can learn to express uncertainty in a way that tracks their actual reliability, and whether people would heed it if they did. Better calibration would help, but a machine's hedge may come to be read as boilerplate, much as people skim past the disclaimers attached to so many other products. Until that is resolved, the responsibility sits with the organizations that deploy these systems and the individuals who act on them. Trust in an output should be earned by its sources, checked by a person, and owned by someone who will answer for it, because the machine will sound equally certain whether or not it is right.
References
Mata v. Avianca, Inc., No. 22-cv-1461 (PKC) (S.D.N.Y. June 22, 2023) (opinion and order on sanctions). Secondary sources report it at 678 F. Supp. 3d 443, but the fetched page does not show that citation, so check it before you use it. This source documents one case, not a trend. See UNSUPPORTED. https://www.courtlistener.com/opinion/9885417/mata-v-avianca-inc/
OpenAI, GPT-4 Technical Report 10, arXiv:2303.08774v6 (Mar. 4, 2024). This is evidence about one model only. The appended system card (§ 2.13) adds that errors can be more convincing "due to authoritative tone". https://arxiv.org/pdf/2303.08774
Katherine Tian et al., Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback, in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing 5433 (2023). This paper is peer reviewed. On three benchmarks it found that asking RLHF models for a stated confidence improved calibration. The… https://aclanthology.org/2023.emnlp-main.330/
Nelson F. Liu, Tianyi Zhang & Percy Liang, Evaluating Verifiability in Generative Search Engines, arXiv:2304.09848v2 (Oct. 23, 2023). The study covers four 2023 commercial generative search engines. It shows that cited sources often fail to support what the system says, which is evidence for the claim but not a measurement of "reading correctly" as such. https://arxiv.org/abs/2304.09848
Rolf Reber & Norbert Schwarz, Effects of Perceptual Fluency on Judgments of Truth, 8 Consciousness & Cognition 338 (1999). This is a primary experimental study and supports the core claim. The page says "perceptual fluency". The label "fluency effect" does not appear in the fetched text. https://philpapers.org/rec/REBEOP
Linda J. Skitka, Automation: Decision Aid or Decision Maker? 33 (NASA Ames Coop. Research Agreement NCC 2-986, Final Report) (undated). The report also documents commission errors, where people follow wrong automated directives, which covers "accept their outputs". The peer-reviewed version is Skitka, Mosier & Burdick, "Does automation bias decision-making?", 51 Int'l J. Hum.-Computer Stud. 991… https://ntrs.nasa.gov/api/citations/19980048379/downloads/19980048379.pdf
