Consider an accounts payable clerk at a mid-sized manufacturer who has to decide whether to pay an invoice that does not match its purchase order. She opens the invoice in one system, the purchase order in another, the receiving record in a warehouse application, and an email thread with the vendor in a fourth window. Each of those systems works as designed, and each offers dozens of features: search, filters, approval buttons, export to spreadsheet. None of them knows that she is trying to make a decision, what evidence that decision depends on, or which company policy governs it. The judgment happens in her head, and when she clicks approve, almost none of her reasoning is recorded anywhere.
That gap is where I think the next generation of software will be built. For several decades, we have organized applications around features: screens, forms, functions, and reports that a person operates in order to get work done. The systems now becoming possible, which combine language models with structured data, business rules, and workflow engines, can be organized around something else. They can be built around the decisions an organization makes, gathering the evidence for each one, applying the relevant policy, recommending an action, and keeping a record of why. The change has less to do with adding intelligence to existing products than with changing what a product is designed to accomplish.
Features describe capability, decisions describe purpose
A feature is defined by what it lets a user do: create a record, run a report, route a document for signature. This way of thinking has served the industry well, because features are easy to specify, build, test, and sell. Product roadmaps are lists of them, and procurement teams compare vendors by counting them. The cost of this approach is that the purpose of the software sits outside it. A feature-rich system can make every step available while leaving the hardest part of the work, deciding what to do, entirely to the person in front of the screen.
A decision is a different unit of design. It has inputs (the facts that bear on it), a governing policy (the rules or thresholds that constrain it), a set of possible outcomes, an owner who is accountable for it, and consequences that can be observed afterward. When software is designed around a decision, the designer stops asking which buttons the user needs and starts asking what evidence the decision requires, which parts of it a rule can settle, which require judgment, and who must sign off when the two disagree. Features do not disappear in this model. They become the means by which a decision is investigated and carried out.
A decision-centered system is assembled from evidence, policy, and judgment
To make this concrete, imagine a hypothetical company, Northwind Labs, which I will use purely as an illustration. Northwind receives an invoice for $48,200 against a purchase order for $45,000. Its policy allows automatic payment when the invoice, purchase order, and receiving record agree within 2 percent, and it requires a manager's review for any larger variance or for any vendor whose bank details changed in the past 30 days. In a feature-centered system, the clerk discovers all of this herself, one window at a time. A decision-centered system assembles the case before she sees it.
A system built that way would produce, for each invoice, a decision record with a small number of distinct parts:
The evidence gathered, including the invoice, the purchase order, the receiving record, the vendor's payment history, and any related correspondence, each linked to its source.
The deterministic checks applied, such as the three-way match and the tolerance threshold, with the result of each.
The recommendation, for example to hold the invoice for manager review because the variance of 7.1 percent exceeds the 2 percent tolerance, along with anything the model contributed, such as its reading of a vendor email explaining a price increase.
The person who decided, what they decided, and when.
The outcome observed later, such as whether a disputed charge was eventually credited.
The boundary between the second and third items is where much of the design effort belongs. A tolerance check is deterministic: given the same numbers, it produces the same answer every time, and it can be tested exhaustively. The model's reading of an email is probabilistic, which means it may be right most of the time while still varying from run to run and occasionally being confidently wrong. A well-built decision system keeps these apart, uses rules wherever a rule is adequate, and reserves the model for work that genuinely requires interpreting language or weighing ambiguous evidence. Coordinating the steps, deciding which check runs when and who is notified, is orchestration, and it is a separate job from the judgment itself.
Explanations have to come from the record, not from the model's account of itself
The promise that intelligent systems will explain their reasoning deserves more caution than it usually receives. A language model can produce a fluent account of why it recommended an action, but research on model interpretability suggests that such self-reported explanations do not always reflect the process that actually produced the output [1]. An explanation that sounds plausible and is unfaithful is worse than none, because it invites trust it has not earned. For that reason, I think the explanation a decision system offers should be built mainly from what the system can show: the evidence it retrieved, the rules it applied and their results, the policy version in force, and the specific passage of a document it relied on.
Verification calls for the same discipline. When Northwind's system confirms that an invoice passes the three-way match, it has shown that the documents agree with one another and with the policy. It has not shown that the goods arrived in good condition or that the vendor is honest. Verification checks requirements, not truth, so a decision record should make clear which conclusions rest on rules, which rest on a model's interpretation, and which rest on a person's judgment. Anyone reading the record later, whether an auditor, a regulator, or a colleague picking up the case a year on, needs to know which kind of claim they are looking at.
Developers will design decision boundaries as much as interfaces
For engineers, the change shows up mainly in the artifacts they own. Policies become versioned objects that the system enforces and records, so that a decision made in March can be evaluated against the rules that applied in March. Escalation paths become part of the specification: the conditions under which a recommendation goes to a person, which person receives it, and how long the case can wait before it is routed elsewhere. Evaluation becomes a continuing activity instead of a release gate, because a probabilistic component can drift as the data it encounters changes.
Testing changes along with it. A team can replay a year of historical invoices through a new version of the system and compare its recommendations with what experienced staff actually decided, paying particular attention to the cases where the two disagree. Those disagreements reveal where the policy is unclear, where the model is weak, or where the people were inconsistent, and each finding points to a different repair. The work rewards a mix of software engineering, domain knowledge, and something close to institutional design. A developer who understands why a 2 percent tolerance exists will build a better system than one who only knows how to implement it.
Organizations will need to decide who owns a recommendation
The harder questions are organizational. When software recommends an action and a person approves it, responsibility for the outcome is shared in a way that most companies have not thought through. A manager who approves 40 recommendations an hour may be performing a formality, and studies of automation bias suggest that people tend to defer to automated suggestions even when they have reason to doubt them [citation needed]. Some jurisdictions already restrict decisions made solely by automated means when those decisions significantly affect individuals, a matter the European Union's General Data Protection Regulation addresses in Article 22 [2]. An organization adopting decision-centered software therefore has to define, for each decision, what the system may do on its own, what it may only recommend, and what a reviewer is expected to check before approving.
The shape of work will shift as well. When routine cases are resolved by rule and assembled evidence, the people who once handled them will spend more of their time on exceptions, disputes, and revisions to policy itself. That can be more interesting work, and it can make expertise more visible, since a decision record shows how a skilled reviewer reasons. It also carries a risk I do not think anyone has resolved. Judgment is learned partly through repetition on ordinary cases, and a workforce that sees only the exceptions may have fewer chances to develop it, so organizations that want capable reviewers in 10 years will need to plan how those reviewers are trained when the system handles most of the practice.
The first step is naming the decisions software already touches
For anyone building or buying software now, the most practical step is modest. Before adding a model to a product, list the decisions the product already touches, who makes each one, what evidence they consult, and which policy governs it. A team that does this will often find that some decisions are made inconsistently, that the governing policy was never written down, or that the evidence lives in places the software cannot reach. Those findings are worth more than any model, because a system cannot support a decision that no one has defined.
The larger question is how we will judge such software once it exists. Features can be counted and demonstrated in a sales meeting, while the quality of decisions shows up slowly, in outcomes observed months later and in errors that never happened. I suspect the organizations that do well with this generation of software will be the ones willing to measure what their decisions actually produced and to treat the decision record as an asset worth keeping. Whether most of them are prepared to look that closely at their own judgment is, to my mind, the open question that matters most.
References
Miles Turpin, Julian Michael, Ethan Perez & Samuel R. Bowman, Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (arXiv:2305.04388v2, Dec. 9, 2023), https://arxiv.org/abs/2305.04388. https://arxiv.org/abs/2305.04388
Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the Protection of Natural Persons with Regard to the Processing of Personal Data and on the Free Movement of Such Data (General Data Protection Regulation), art. 22, 2016 O.J. (L 119) 1, 46. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32016R0679
