The Risk Ecologist: Why MedTech Software Keeps Failing the Same Way

This content was created with AI assistance from the full episode transcript.

Robert Charette has spent 50 years documenting why large-scale IT systems fail. Not medical device systems specifically — he is the first to say that medical devices are not his specialty area. What he has studied is DOD procurement disasters, national EHR rollouts, government payroll systems, space shuttle software safety, and Fortune 100 enterprise failures. The patterns he has documented across those domains are now showing up in medtech on a monthly basis, which is exactly why he was a guest on Inside MedTech Innovation.

This episode covers the difference between a failure and a blunder, why compliant software still causes recalls, why all risk pools at the interfaces, and why AI amplifies expertise rather than replacing it.

Who Is Robert Charette?

Charette is a Contributing Editor at IEEE Spectrum, where he has written about software failure for 15 years. He was a member of the NASA commission that reviewed shuttle software safety following Challenger. He has worked with the Department of Defense, foreign governments, and large commercial organizations on software risk management. He pioneered the application of formal risk management to the software field alongside Barry Boehm and Peter Neumann.

He calls himself a risk ecologist: "I take a look at the technical, the financial, the social, the political, all these different things. Because it is an ecology, and you can't just poke at one."

He is not a medical device safety expert. He makes this clear and Shannon booked him because of it, not despite it. When someone who has watched the same failure patterns play out across every major IT domain for 50 years looks at medtech software, the perspective is worth hearing.

Failures vs. Blunders: A Distinction That Reframes Recalls

The most important distinction in this episode is one Charette draws between failures and blunders.

A failure is when you do everything you should and things still go wrong. A blunder is when you just decide not to do things.

These are not the same problem. They don't have the same fix. And the difference matters enormously for how you think about software recalls.

If most recalls are failures, the implication is that the work was done correctly and something unforeseeable happened. The response is better luck, more resources, perhaps more testing. If most recalls are blunders, the implication is that known steps were skipped, that the decision not to do them was made somewhere in the organization, and that the fix is cultural and structural, not technical.

Charette's read, based on 50 years of case studies: most are blunders.

The Compliance-Performance Gap

Shannon has been pushing a specific thesis on this show for two years: the medtech industry moved from no product security to compliance, and now needs to move from compliance to performance. Charette arrived at the same conclusion from a different direction.

"There's a lot of incentives to be compliant, but not necessarily to be performance-enhanced."

He illustrated it this way: map the recall data against the compliance standards that were supposedly being followed. The result is not reassuring. Software that passed every required review is still causing harm. The standards were met. The software was not safe. Those are different things.

Where recalls originate matters here. "If you take a look at where the areas that most of the recalls are coming from, it's the design area — which is basically saying you're not building the right thing." Verification asks whether the software does what it's supposed to do. Validation asks whether you were building the right thing in the first place. Most regulatory frameworks emphasize the former. Most failures live in the latter.

Why the Same Failures Keep Repeating

Charette's 2005 article for IEEE Spectrum argued that software failures are, for the most part, predictable and avoidable. He could publish it again today with minimal revisions. That is not a statement about the article — it is a statement about the industry.

"How many ERP systems need to fail because you didn't test your assumptions to create yet another one where you don't test your assumptions? I'm not sure that's information asymmetry. I think that's just being dumb."

The Phoenix payroll system in Canada failed in ways nearly identical to the Victorian government payroll system failures from 1995, which were nearly identical to failures documented before that. Government officials call it "teething problems" and treat it as normal. Charette's question: what did you do to backstop the teething problems? Usually: nothing.

His phrase for it: "It's this deja vu to the nth power problem."

The excuse organizations use is always the same — we're different, we have better technology, we're going to be more careful. The first thing they do when a budget problem hits: cut testing. The testing that was already reduced to get the proposal approved.

All Risk Pools at the Interfaces

One of Charette's maxims: all risk pools at the interfaces.

This is not a metaphor. It is a design principle. When you modularize software and make the interfaces between components explicit — who owns each interface, who controls it, who is responsible when something stops crossing that boundary — you create the conditions for real verification. When you don't, verification becomes impossible in practice.

"You need to structure your software so that it will make it more allowed to be verified and validated." If V&V is treated as something you do after the software is built, on a system that was not designed to be testable, you will find errors after patients are harmed — not before.

He connected this directly to Change Healthcare. The question he asks for any system that sits at the center of critical infrastructure: have you mapped every interface, identified who owns it, and asked what happens when something stops crossing that boundary?

That is the whole trick.

Human Behavior Is the Missing Variable

Every framework for improving software quality fails at the same point: human behavior.

Charette has entered organizations to implement risk management more times than he can count. His first step is always a cultural assessment. "I always did a big cultural assessment to try to find out what the lowest level — what the programmers wanted from it — and then what the senior managers wanted from it. And it was totally different."

The process has to satisfy both or it will be ignored. "If you don't satisfy what their human needs are — it doesn't matter. It's all garbage at the end of the day."

The specific behavior that undermines risk management in engineering organizations: raising a risk gets interpreted as not knowing your job. "Once you bring up a risk, especially in an engineering organization, that gets interpreted as you're not doing your job." If surfacing problems signals incompetence rather than professional responsibility, problems stay hidden.

Charette's solution, demonstrated at Rockwell Collins — one of the only companies he has met that changed direction while still successful — was to change the dashboard convention from green-to-red to red-to-green. Under the NASA Apollo model: you do not move from red to yellow until you have proved something worked. You do not move to green by assumption. The default is red, and you earn your way out.

Most organizations run the opposite convention: green by default, with risks flagged on top. The result is a board that hears "we're green" and stops asking questions.

AI Amplifies Expertise. It Does Not Replace It.

Charette's view on AI in software development is specific and does not match either dominant position in the debate.

"AI is really good for experts. It's not really good for people who aren't experts."

His illustration: a friend who designs games for Sony used Anthropic to generate very sophisticated code, tested it himself, and said "we're all going to be out of business." Charette's response: who else on your team could have framed that question? And who else could have evaluated whether the result was correct? The junior developer on the team could not have done either.

"Who, other than yourself, could have framed the question for Anthropic to solve? And who, other than yourself, could test whether the answer was right?"

The downstream risk is not displacement — it is skill atrophy. If junior engineers stop learning because AI handles the output, the expertise required to use AI well disappears in the next generation. No one will be left who can frame the question or evaluate the result.

On AI ethics as a separate debate, he is dismissive: "I refuse to write about AI and ethics, because it's the same story we've been writing for a long time." The root causes of AI risk are the same three things that cause all IT risk: lack of information, lack of control, lack of time. AI accelerates them. It does not create new ones.

What This Means for MedTech

Charette is not in medtech. He has never claimed to be. What he has is 50 years of case studies from every domain where large-scale software systems have failed, and a clear-eyed read on why the same patterns keep showing up.

The medtech industry moved from no software regulation to compliance. That was progress. Compliance is not the same as building software that works. The next step — from compliance to performance — requires confronting the gap between what the documentation says and what the software does. It requires designing for verification from the first architecture decision, not adding it at the end. It requires a risk culture where raising a problem is treated as professional responsibility, not incompetence.

The failures Charette has been documenting for 50 years are predictable and avoidable. The fact that they keep happening is not a technology problem.

Listen to the full episode: https://creators.spotify.com/pod/profile/shannon-lantzy

Inside MedTech Innovation is hosted by Shannon Lantzy. This post was created with AI assistance from the full episode transcript.