An AI's Vulnerability is an Asset
When AI shows its vulnerability, trust in the system increases. It also happens to feel more human.
While driving my son to a sports practice, I had Full Self Driving beta (FSD) turned on. It started to sprinkle outside, and we immediately saw an alert: "Weather detected. FSD performance may be degraded." My son asked me what "degraded" meant in this scenario. I said, "I assume it means that it has less certainty that FSD won't make a mistake, and is telling me to increase my attention and ability to safely correct if it does something wrong." He said, "is it scared?"
Wow, what a question. Is my Tesla's FSD beta "scared?" In a way, yes. "I'm letting you know that there may be an increased risk ahead." Is that fear? I told him "No, fear involves negative emotion. I don't think this is emotional for FSD. FSD is expressing its uncertainty. Having uncertainty and knowing it is actually quite powerful, and can come with positive emotions (if FSD can have emotions at all)." (On the other hand, perhaps this message conveys fear from the developer who designed the message and its appearance. But that's for another blog post.)
Instead of fear, I suggested that FSD was communicating its vulnerability. Anyone following modern advice (in leadership, personal development, or business) knows that vulnerability is a hot topic. Be vulnerable, and reap the rewards of trust, connection, and happiness!
The same goes for AI. When designing AI, know where it is weak, measure the weaknesses, and communicate the weakness to the user (who in turn will make up for the vulnerability with judgment and action). The AI will be more trustworthy as a result.
Here's the mind-boggler: this goes for any system, not just AI. If a system is designed to produce a result, and the system can express its vulnerability, it is much more trustworthy than its competitors. As we race to speed up software releases in medtech (and, consequently, regulatory review), we must incorporate measurement and communication of its strengths and weaknesses.
~Shannon the Optimistic Optimizer