Not that long ago Garry published a great article on why Procurement Needs an “AI Incident Log”.
Simply put, because most failures will be quiet.
(And, even worse, to the extent possible, they will be covered up.)
For example, as Garry states a supplier gets mis-classified as low risk for months. A category recommendation nudges the organization towards convenience over resilience. A contract summary misses a clause that only matters when something goes wrong. A “temporary” exception becomes the new normal because the tool makes it easy to repeat. And as long as nothing explodes, standards and practices get to keep drifting from well designed and established norms that were designed to be best practice for the organization.
These are failures, even if they don’t result in disasters in the near-term, and in many ways, they are the worst kind of failures. That’s because, by the time something goes significantly wrong, it will not only be a disaster but it won’t be one that can be quickly recovered from as the data, process, monitoring, and mitigations will be so bad as to be unusable.
And, as Garry points out, this will all be due to AI influence as its permeation is literally causing organizational decay as a result of the cognitive atrophy, curiosity decay, false memories, and overall cognitive offloading and general acceptance of the enshittification it is bringing with it. The easier the tools make it to do nothing, the more likely that is what is done as we are wired to be lazy as a species and, sadly, most of white-collar humanity gives into that wiring.
So unless you want your performance to suffer from AI-induced enshittification, you need to prevent the enshittification from happening in the first place. To do that, you need to stop the process drift that is a result of humans shifting decisions to systems that should stay with them.
And, according to Garry, that means adopting an AI incident log to track signals that take them off course to make sure mistakes are not repeated. The system should tell you four things early:
- where humans are overriding the system and why — not because this is a bad thing, it’s typically a good thing as it means humans are dealing with exceptions, validating decision suggestions before they get accepted and executed, or cutting off AI where it shouldn’t be used; the lack of these overrides is the signal that’s scary where AI has been deployed
- where exceptions are repeating — good systems allow exception resolutions to be turned into rules and automatically processed going forward; if that’s not happening, the cast iron ball is being dropped repeatedly and at some point it’s going to break someone’s toes when it’s not caught in time
- where speed has increased but clarity decreased — hard to detect, unless you ask actions to be explained … when there is no instant explanation, there was no thought, just a system recommendation (which you hope wasn’t the result of a lazy employee asking clod or chat, j’ai pété and sharing your confidential data
- where accountability has blurred — when something goes wrong, you need to know who precisely was responsible for the decision, not a role shared between multiple people or a team, a person who made the decision and accepted the authority for it
Now, this incident log, as Garry states, doesn’t need to be heavy or overbearing. Just a short description of “system/AI used, by who, when, result generated, human response/override, consequence, suggestion/rule to prevent future occurrences”. Short and sweet so the incident log actually gets used.
You can’t improve as an organization if you can’t learn from near misses to prevent foreseeable mistakes. Otherwise, your successes will just be wiped out from inevitable failures. Because, as Garry states, in the beginning, it’s unlikely that AI will break Procurement with one big failure as most organizations will start small with the odds in their favour.
But of course, given time, without proper monitoring and intervention, that failure will happen. And when it does and the incident is significant, two things need to happen.
1. A very detailed end-to-end (root cause) analysis needs to be conducted, along with a detailed mitigation plan with executable data capture, process, and system changes to prevent it from ever happening again.
2. Full publication in a Public Procurement Incident log (perhaps maintained by one of the major associations) where an organization shares what happened, how it all went wrong, and what might be done to prevent future failures of that type. (Which will often be “don’t use this [Gen-]AI tool AT ALL for this type of problem or process”.)
Unless the failure was so bad that it reaches the public by its very nature, most businesses, especially in the B2B world, will try to sweep the AI failure under the rug, especially when the consultants claim it’s just a “growing pain” and will “not happen again” with more training data and model tweaks and finance claims it will sink the stock price.
But this will only lead to more failures and even worse ramifications if the story gets out that AI cost the company millions (or billions) and the company tried to hide it.
In the Age of BS AI Overpromises and Hype, the only solution is a public forum where companies come together and share their war stories to help each other cut through the hype and understand precisely what modern “AI” tools can and can do, to what degree, and how to use those that do work in some situations in a way that won’t result in disaster.
Now we know it will likely never happen, but this is why we have continual boom-and-bust cycles in the IT sector and more failures than we should 150 years after the Gilded Age began and the railroad barons built successful multi-national companies that could manage their entire supply chains from source to sink(ing of the tie in the railway). And do it with an efficiency that wasn’t seen again until Toyota started to implement lean in its Production System (TPS) development over 50 years later. (Look, they wrote the first purchasing manual. They knew their stuff!) If Engineers could manage global supply chains in the industrial age using only pen, paper, letter mail, and their intelligence and do so with more predictability than our most advanced systems today, that tells us something — that the answers don’t lie with AI but HI (Human Intelligence) and that we need systems in place to ensure HI is always used when decisions need to be made and learnings are publicly shared.
Or we can give in to the AI, let our IQs recess faster than we ever thought possible (and they are recessing — roughly 14 points over a 120 year period between the Victorian Age and the end of the first decade of the century), and becoming drooling idiots just waiting to be plugged into the Matrix. (Recent studies have shown that heavy AI users perform up to 17% worse in conceptual tasks compared to non-users. Given that an average IQ should be 100, that’s a 17 point decline in a year or so, meaning that AI is making us stupider 100 times faster than every technology that came before! [Source: Psychology Today.])
(Remember, while it is our right to dare to be stupid, it’s not the smart thing to do, and there will be consequences. So if you think it’s pretty fly that Gen-AI, we strongly suggest you think again.)
