Suppose a production database at Northwind Labs, a hypothetical software company I will use as an example throughout, goes down for 40 minutes late on a Tuesday night. The next morning an engineer pastes the logs, the alert timeline, and the deployment history into a language model and asks for a draft of the incident review. The draft that comes back is excellent on its face. It gives a clear sequence of events, a plausible root cause in connection-pool exhaustion, and a set of remediation steps written in the measured tone of a senior engineer. The on-call engineer who was awake that night, however, knows the real trigger was a configuration change someone made by hand in a cloud console, a change that never appeared in any record the model was given. The draft was coherent and internally consistent, it was also wrong, and nothing in its prose signaled the difference.
That gap between a convincing answer and an understood situation is the subject of this essay. My argument is that generating persuasive reasoning and understanding are different capacities, and that the difference has practical consequences for anyone building, buying, or governing these systems. The distinction is easy to state and hard to keep hold of. Fluent, well-organized language is exactly the signal people have always used to judge whether someone knows what they are talking about, and that signal is now cheap to produce.
Fluency reflects training rather than understanding
A large language model is trained to predict likely continuations of text, and at sufficient scale that objective yields systems that write, summarize, and explain with real skill [1]. What the objective rewards is plausibility relative to the training data and the prompt. Correctness often coincides with plausibility, because much of what people write down is true, and that coincidence is why these systems are useful most of the time. The coincidence is statistical rather than guaranteed, though, and the model has no dependable internal signal that separates the cases where plausible and true diverge. Fabricated citations and invented case law in otherwise polished documents are the visible edge of this property [2].
Philosophers have argued about this distinction since long before current systems existed. John Searle's Chinese Room argument held that manipulating symbols according to rules, however successfully, does not by itself amount to understanding what the symbols mean [3]. The argument remains contested, and I do not think it settles the question for modern models [3]. It does name the right worry: an outside observer judging only by outputs cannot tell competent symbol manipulation from comprehension, and the Northwind incident review is a practical instance of that problem.
Reasoning traces show the work without guaranteeing the grasp
A natural response is that newer systems do more than produce answers, because they reason step by step and can explain themselves. The research here is encouraging. In ReAct: Synergizing Reasoning and Acting in Language Models, Yao and colleagues prompted models to interleave reasoning steps with actions such as search queries, so that each step could be checked against an observation from outside the model [4]. On knowledge-intensive tasks, grounding reasoning in retrieved evidence in this way reduced some of the errors that come from reasoning in isolation [4].
The nature of that improvement deserves attention. What ReAct-style designs add is contact with external evidence, which makes individual claims checkable; they do not add a grasp of which evidence matters or what is missing from it. Had the Northwind model been given a search tool, it would have searched the logs it could reach, and the console change would still have been invisible. There is also an open research question about whether the reasoning a model displays faithfully reflects the computation that produced its answer, with some studies suggesting it often does not [5]. A visible chain of reasoning is a useful artifact for review, but it is better read as a draft of an argument than as a window into comprehension.
Understanding depends on what never made it into the data
The engineer who caught the error did not out-reason the model. She knew something the model could not have known, because she had been in the incident channel that night and had seen a colleague mention adjusting a timeout "just to get things moving." Much of what we call understanding is of this kind: knowledge acquired by being present, by having made similar mistakes, and by knowing the habits of the people and systems involved. Michael Polanyi called a version of this tacit knowledge, the part of what we know that we struggle to put into words [6]. Knowledge that never gets written down cannot be learned from text, however much text there is.
Contextual awareness adds a further layer. Someone who understands a situation can tell when the question being asked is the wrong one, or when the materials provided are incomplete in a way that matters. Asked for a root-cause analysis, the model produced one, because that is what the request called for. An experienced engineer, reading the same logs, would more likely have said that nothing in them explained the timing and asked what else had changed. Part of understanding is a sense of the shape of what you have not been told, and current systems are poor at noticing absences in their input [7].
Accountability cannot rest with a system that bears no consequences
These limits matter more as systems take on longer sequences of work. Frameworks such as AutoGen, described by Wu and colleagues in AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation, let several model-backed agents converse with one another, call tools, and optionally include a human participant in the exchange. Anthropic's Building Effective Agents draws a useful line between workflows, in which models and tools follow predefined code paths, and agents, in which the model directs its own process and tool use. The same piece advises starting with the simplest arrangement that works and adding autonomy only when it earns its cost. That advice reads to me as an engineering principle with an epistemic core: every step a system takes on its own is a step at which no one with understanding is looking.
Accountability is where the difference between answering and understanding becomes institutional. When the Northwind engineer signs the incident review, she accepts that she can be asked why she believes it, that her credibility depends on it, and that she will have to revise it publicly if it proves wrong. A model can be asked why, but it will simply generate a new answer, and nothing about its situation changes if the first one was false. The NIST AI Risk Management Framework 1.0 lists accountability and transparency among the characteristics of trustworthy AI, and its Govern function is concerned with assigning roles and responsibilities around AI systems [8]. The framework places accountability with the organizations and people who design, deploy, and use these systems, which seems to me the only place it can coherently sit [8].
For teams designing these systems, a few practical commitments follow from this argument. None of them depends on a better model; each concerns where people sit in the process and what they are asked to check. Taken together, they aim to keep understanding located with someone who can be held to it, and they can be applied to any workflow in which a model drafts something consequential:
Every consequential output has a named person who reviews it and is answerable for it.
The reviewer is someone positioned to know what the inputs left out, not only someone able to judge the prose.
Autonomous steps are added only where an error would be caught before it causes harm.
Model reasoning is retained as a record that a reviewer can test, not accepted as a justification.
None of these commitments is exotic, and each will slow some work down. The cost is the point, since it marks where an organization has decided that a fluent answer is not enough. At Northwind, the second commitment is the one that would have mattered most. The engineer who caught the error was valuable because she had been in the incident channel that night, and a reviewer who had seen only the logs would have had every reason to approve the draft. The third commitment applies Anthropic's advice about autonomy to the same problem, by asking before each new automated step whether anyone would notice if that step went wrong.
The myth of knowing applies to people as much as to machines
It would be convenient to treat all of this as a defect peculiar to machines, but I do not think that is honest. What I think of as the myth of knowing is the habit, in organizations and in individuals, of mistaking confident articulation for understanding. Executives approve strategies because the memo is persuasive, investors fund companies because the founder tells a coherent story, and engineers accept explanations because they fit what was expected. Language models did not create this habit. They have made confident articulation abundant and inexpensive, which exposes how heavily we already relied on it as a proxy.
Seen this way, the useful response is less a matter of distrusting AI than of becoming more deliberate about what counts as knowing. A model's output is best treated as a well-formed hypothesis, one whose value depends on whether someone with experience and stakes in the outcome tests it against what they know. The same discipline would improve how we read human experts, including ourselves. If the Northwind team took one lesson from its incident, I would hope it was a review process that asks "what would have to be true for this to be wrong?" of every draft, regardless of who or what wrote it.
What I cannot answer is whether this boundary is permanent. It is possible that systems with persistent memory, continuous contact with the world, and some form of consequence for error would come to have something we would fairly call understanding. It is equally possible that understanding, as people experience it, depends on having a life in which being wrong matters, and that no amount of capability supplies that. For now, the practical conclusion does not depend on which view turns out to be right: until a system can be answerable for what it says, someone else has to be.
References
Tom B. Brown et al., Language Models are Few-Shot Learners, arXiv:2005.14165 (2020), https://doi.org/10.48550/arXiv.2005.14165. https://arxiv.org/abs/2005.14165
Mata v. Avianca, Inc., 678 F. Supp. 3d 443, 448 (S.D.N.Y. 2023). https://www.law.berkeley.edu/wp-content/uploads/archive/2025/12/Mata-v-Avianca-Inc.pdf
David Cole, The Chinese Room Argument, Stanford Encyclopedia of Philosophy (rev. Oct. 23, 2024), https://plato.stanford.edu/entries/chinese-room/; see also John R. Searle, Minds, Brains and Programs, Behav. & Brain Sci. (1980). https://plato.stanford.edu/entries/chinese-room/
Shunyu Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models, arXiv:2210.03629. https://ar5iv.labs.arxiv.org/html/2210.03629
Miles Turpin, Julian Michael, Ethan Perez & Samuel R. Bowman, Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting, arXiv:2305.04388 (2023). https://arxiv.org/abs/2305.04388
Michael Polanyi, The Tacit Dimension 4 (1966) (as quoted in David Pimm, Commentary on Knowing More Than We Can Tell, in Theories of Mathematics Education 613 (Bharath Sriraman & Lyn English eds., 2010)). https://link.springer.com/chapter/10.1007/978-3-642-00742-2_57
Harvey Yiyun Fu et al., AbsenceBench: Language Models Can't Tell What's Missing, arXiv:2506.11440 (2025). https://arxiv.org/abs/2506.11440
Nat'l Inst. of Standards & Tech., NIST AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0) 12 (Jan. 2023), https://doi.org/10.6028/NIST.AI.100-1. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf
