Designing agentic AI people can trust
For a few years, the AI in our products mostly suggested things. It autocompleted, summarised, recommended. The worst it could do was waste your time. Agentic AI is different: it doesn’t just surface information, it takes action. It reconciles the exception, drafts the configuration, resolves the break. And the moment software starts doing rather than advising, the central design problem changes. It stops being about the interface and becomes about trust.
This matters most in exactly the places agentic AI is most useful: high-stakes, regulated, expensive-to-get-wrong work. In financial operations, an AI that posts the wrong entry to the books is an incident, not a bug report. So the interesting question isn’t “can the AI do this?” Increasingly, it can. The question is “under what conditions will a professional let it?”
Trust is structural
It’s tempting to treat trust as something you earn with a polished UI: a confident tone, a reassuring animation. That’s backwards. In high-stakes work, trust is structural. People extend it when they can see what the system is about to do, stop it before it does, and account for it afterwards. Get that structure right and the personality barely matters. Get it wrong and no amount of friendly copy will save you.
The model I keep coming back to is staged autonomy. The AI earns scope in three steps:
- It recommends. The AI proposes an action and explains its reasoning; a person decides. Nothing happens without a human.
- It acts with approval. The AI prepares the action in full, the actual change ready to go, and a person approves or rejects it. The human is still the gate, but the work is done.
- It acts autonomously. Only for the narrow, well-understood cases where the cost of being wrong is genuinely low, and always with the ability to intervene.
Most agentic features should live in the first two stages far longer than the technology strictly requires. That’s how you build the evidence, and the comfort, that earns the third.
”AI suggests, humans approve” is a feature
There’s a worry that keeping a human in the loop defeats the point: if a person still has to approve everything, where’s the saving? But that misreads where the value is. The slow, expensive part of most expert work is the investigation, the gathering, the drafting, the cross-checking, not clicking approve. Let the AI do that and present a finished, explained, reversible proposal, and you’ve removed almost all of the cost while keeping the one thing that makes the whole thing adoptable: a person who remains accountable.
So I treat “AI suggests, humans approve” as the design that makes adoption possible, not a limitation to be engineered away. The aim is to remove the human from the labour and leave them in charge of the judgement.
What this asks of the design
Once you accept that the approval moment is the product, the design priorities reorder themselves.
Design the moment of approval, not the moment of automation. The screen that matters is the one where a person decides, not the one where the AI works. It has to show what will change, why the AI proposed it, how confident it is, and what happens if it’s wrong, clearly enough to be judged in seconds, not studied for minutes.
Make confidence legible and honest. A system that’s equally assertive whether it’s certain or guessing teaches people to ignore it. Surfacing how sure the AI is, and being visibly more cautious when it isn’t, is what lets someone calibrate how much to lean on it.
Make everything reversible, and say so. People approve more readily when they know they can undo. Designing the exit ramp is as important as designing the action.
Treat the audit trail as a first-class surface. In regulated work, “who decided this, and on what basis?” is the thing that lets the capability exist at all, not an afterthought. A complete, readable record of what the AI did, what a human approved, and why, is part of the user experience, not a compliance tax bolted on at the end.
The unglamorous centre
None of this is about making AI feel magical. It’s about making AI legible, governable, and accountable enough that a cautious professional will hand it real work. The teams that win the next few years of enterprise AI won’t be the ones whose models are marginally cleverer. They’ll be the ones who designed the trust structure around the model: the approvals, the transparency, the reversibility, the record.
Which, when I think about it, is the same job design has always had in complex software: making the hard thing safe to rely on.