Consider a workflow that many companies are now building. A language model reads incoming supplier invoices, compares each one against the purchase order and the contract, and proposes whether to pay, hold, or dispute it. An accounts payable analyst reviews each proposal and clicks Approve before anything happens. On a slide, this looks responsible, because a person signs off on every outcome. In practice, if the analyst sees two hundred proposals a shift, cannot see what the model actually read, and is measured on throughput, the click records a decision that nobody really made [1].
My argument in this post is that a human approval step creates accountability only when the system gives that person the information, time, authority, and independence to disagree. Without those conditions, the approval step does something worse than nothing. It moves responsibility onto the reviewer while leaving the substance of the decision with the model. To keep the discussion concrete, I will use a hypothetical company, Northwind Labs, which is invented for illustration, and follow its invoice workflow through each part of the argument.
An approval button records a decision without making one
The first problem is behavioral, and it gets worse as the model gets better. People who supervise automated systems tend to accept the system's recommendations, particularly when those recommendations have been reliable for a long stretch [1]. Suppose the Northwind model has been right on nearly every invoice for several weeks. The analyst learns, quite reasonably, that approving is almost always correct and always fast, while rejecting takes time, requires a written justification, and slows the queue. Under those incentives, agreement becomes the default, and attention drifts toward the cases that are easy to judge.
The second problem concerns where blame lands when something goes wrong. If Northwind pays a duplicate invoice or a fraudulent one, the audit trail will show that a named analyst approved it. The organization can point to that approval as evidence of human oversight. Yet the analyst may have had no realistic opportunity to catch the error, because the screen showed a summary rather than the documents, or because the volume left a few seconds per case. Responsibility has been assigned to a person who never held the control that responsibility implies, and that gap is what a good design has to close.
Informed judgment depends on conditions the workflow supplies
If the human step is going to mean anything, the workflow has to create the conditions under which a person can exercise real judgment. Each of those conditions answers one of the failures described above: a reviewer who cannot see the evidence, who has too little time, who is penalized for disagreeing, or who is steered toward the model's answer before thinking about the case. I find it useful to state the conditions as requirements that a design either meets or does not.
The reviewer can see what the system saw, including the source documents, the rules it applied, and the specific fields that drove its recommendation.
The reviewer has enough time for the stakes of the decision, which means volume is set by what careful review requires rather than the reverse.
The reviewer has the authority to reject or modify the output, and doing so carries no penalty in how their performance is measured.
The reviewer understands the case well enough to evaluate it, including the kinds of errors the model is known to make.
The reviewer forms at least part of their view before seeing the model's framing of the case.
The last condition is the hardest to meet and the easiest to ignore. When a review screen opens with the model's proposed answer and a fluent rationale, the reviewer begins anchored to that answer [1]. Northwind could address this for a random sample of invoices by showing the analyst the invoice and the purchase order first, asking them to note any discrepancies, and only then revealing the model's proposal. That design is slower, which is precisely why it is informative: it measures whether the human contributes anything the model did not. Each of these conditions has a cost, and an oversight step that costs nothing in time or attention is usually not doing much overseeing.
Traceability makes a decision reviewable after the fact
Judgment in the moment is only half of oversight; the other half is being able to reconstruct a decision later. For each invoice, Northwind should record the model version, the instructions it was given, the exact versions of the documents it retrieved, the result of every rule check, the proposal it produced, the identity of the reviewer, which documents the reviewer opened, how long they spent, and what they finally decided. The distinction that matters here is between a record and a reconstruction. A record is captured at the time of the decision. Reconstruction, by contrast, is assembled afterward from whatever survives, and it tends to flatter the process that produced it.
Some of those fields matter more than they first appear. The time a reviewer spent and the documents they opened reveal whether review actually happened [1]. If the logs show approvals arriving four seconds after the proposal, with the contract never opened, Northwind learns that its oversight is nominal before an auditor or a court points it out. Traceability is also what makes the system correctable. When an error surfaces, a complete record lets the team determine whether the model misread a document, a rule was missing, a contract in the repository was out of date, or the reviewer missed something visible on the screen, and each of those causes calls for a different remedy.
Verification has to be independent of what it checks
A common response to concerns about model error is to add a second model that reviews the first. This can catch some mistakes, but the two models often share training data, architecture, and blind spots, so their errors tend to be correlated [2]. A check that fails for the same reasons as the thing it checks adds less assurance than it seems to. Independence means the verification step can fail only for different reasons, which usually requires a different method rather than a second instance of the same one.
Deterministic rule checks are the clearest example of an independent method. Northwind can confirm that the invoice total equals the sum of the purchase order lines, that the vendor's bank account matches the vendor master record, and that the payment terms fall within the contract. These checks are inexpensive to run and give the same answer every time, which makes them easy to audit. They also have a firm limit: verification against a rule shows that an output meets a stated requirement, not that the output is correct. A fraudulent invoice whose numbers are internally consistent can pass every rule. For that reason, rule checks should be paired with sampled independent review, in which a separate team re-examines a random set of decisions from scratch and compares its conclusions with what the system and the approver decided. The disagreement rate from that exercise is one of the few direct measurements of whether the whole process, including the human step, is working.
Escalation turns uncertainty into a route instead of a guess
A reviewer facing an ambiguous case without an escalation path has two poor options. They can approve and hope, or they can reject and absorb the delay and the questions that follow. An escalation protocol gives uncertainty somewhere to go. At Northwind, sensible triggers would include any failed rule check, amounts above a set threshold, a vendor with no prior payment history, a mismatch between the model's stated rationale and the documents, and the analyst's own doubt, recorded with a sentence of explanation. I would be cautious about relying on a model's self-reported confidence as a trigger, since those scores are often poorly calibrated and work better as one signal among several [3].
A trigger is only the first part of a protocol. The design also has to specify who receives the escalation, what information travels with it, how quickly a response is due, and what happens if no one responds. That last point is where an organization's real priorities show. If an escalated invoice at Northwind is paid automatically once its deadline passes, the protocol is a delay rather than a safeguard; if it defaults to a held state with the vendor notified, the protocol protects the company at the cost of some friction. Override and escalation rates deserve regular attention as well. An approval step that rejects almost nothing for months might reflect an excellent model or a disengaged reviewer, and only the traceability described above can tell those apart.
Accountability belongs to whoever designs the conditions of review
Regulators are beginning to frame oversight in similar terms [4]. The European Union's AI Act, for example, requires that high-risk systems be designed so that the people overseeing them can understand the system's capabilities and limitations and can decide not to use its output [5]. Whatever the legal particulars turn out to mean in practice, the direction is clear enough. Oversight will increasingly be judged by what the human could actually do, rather than by whether a human appeared somewhere in the process.
The practical consequence is that accountability for an automated decision sits largely with the people who designed the review, set its staffing, and chose its defaults. Before approving any workflow with a human sign-off, I would want the design team to show what a reviewer would need to catch the most likely error and where the screen provides it. The team should also be able to state how long a careful review takes and whether staffing allows that time, how a reviewer who rejects often is treated, and whether any decision can be reconstructed from records six months later. A design that cannot answer those points has a human in the loop in name only.
The harder question is what happens as models improve. Better models make errors rarer and subtler, and sustained vigilance for rare events is something people do poorly [1][6]. It may be that the human role shifts from approving individual outputs toward designing, auditing, and owning the system that produces them, with case-by-case review reserved for decisions that are high in stakes or hard to reverse. I do not know where that line should fall, and I suspect it depends heavily on the domain. What seems clear is that placing a person at the end of an automated process does not settle the question of accountability, and that organizations would do better to begin with it.
References
Raja Parasuraman & Dietrich H. Manzey, Complacency and Bias in Human Use of Automation: An Attentional Integration, 52 Hum. Factors 381, 390 (2010). https://depositonce.tu-berlin.de/bitstreams/cafd2873-814b-4c59-bab1-addd42e249d2/download
Elliot Kim, Avi Garg, Kenny Peng & Nikhil Garg, Correlated Errors in Large Language Models (Int'l Conf. on Mach. Learning 2025), arXiv:2506.07962. https://arxiv.org/abs/2506.07962
Miao Xiong et al., Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (v2 Mar. 17, 2024), arXiv:2306.13063. https://arxiv.org/abs/2306.13063
Regulation (EU) 2024/1689, supra, recital 73. https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=OJ%3AL_202401689
Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act), art. 14(4)(a), 2024 O.J. (L 2024/1689). https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-14
Joel S. Warm, Raja Parasuraman & Gerald Matthews, Vigilance Requires Hard Mental Work and Is Stressful, 50 Hum. Factors 433 (2008). https://journals.sagepub.com/doi/10.1518/001872008X312152
