I’m doing another very specific call for examples for my next book, The System Says No: How to Use Rules, Data and AI Without Surrendering Human Judgement.
I’ve now got plenty of examples of rules, policies, data and processes producing questionable outcomes. What I’m short of are two rather different kinds of story.
First: professional judgement where scoring or criteria weren’t enough. I’m looking for real examples from assessment, recruitment, education, healthcare, audit, regulation, inspection, professional accreditation or similar settings where there was a rubric, score, threshold or set of criteria, but applying it mechanically would not have produced the right decision. Perhaps an experienced person had to interpret conflicting evidence, take context into account, moderate a judgement or challenge what the score appeared to say.
I’m particularly interested in what made that judgement defensible rather than arbitrary: moderation, second opinions, calibration, an audit trail, further evidence, an appeal process, etc.
Second: examples of organisations deliberately designing human judgement into a system. For example, a process with a genuine human override, appeal or escalation route; someone successfully challenging a system and getting it changed; a decision being reviewed because the outcome didn’t make sense; or an organisation deliberately retaining human authority over an automated or AI-assisted decision.
The second category can include examples where the organisation got this right. I definitely don’t need every story to be another tale of organisational madness.
I’d be especially interested in an AI example where a system recommended, scored, screened or generated something and a person spotted that it was missing context or reaching the wrong conclusion, and had both the judgement and the authority to do something about it.
These can be small everyday examples rather than dramatic ones. I’m interested in what happened, why the normal system wasn’t sufficient, who made the judgement, what safeguards existed, and what happened as a result.
Everything can be anonymised, and I wouldn’t use anyone’s story without agreeing that with them first. Please comment, DM me, or point me towards someone I should speak to.
Thank you - the examples people have shared so far are making this a much better book.
