The 18-Year Journey: How One Algorithm Transformed Energy Markets

A conversation with Dr. Richard O'Neill on regulatory innovation, market optimization, and the challenges of implementing change in critical infrastructure

When Dr. Richard O'Neill received a call from the CEO of PJM, the electricity market operator serving 13 states, he expected a routine update on their software testing. What he heard instead changed his understanding of how quickly proven innovations could impact entire markets.

"We're doing the testing," the CEO told him, "and in six months we saved half a billion dollars." [00:00:00]

This moment, nearly two decades after Dr. O'Neill first began working on market optimization algorithms at the Federal Energy Regulatory Commission (FERC), validated years of research and development. But it also highlighted one of the most persistent challenges in regulatory innovation: the gap between proof of concept and widespread adoption.

The Foundation: Understanding Energy Markets

To understand the significance of Dr. O'Neill's work, it helps to first grasp how electricity markets operate. Unlike typical commodity markets, electricity cannot be easily stored and must be produced in real-time to match demand. This creates unique optimization challenges that require sophisticated mathematical solutions. [00:03:00]

Independent System Operators (ISOs) serve as market makers, running daily auctions where electricity generators bid to supply power to consumers. These auctions are essentially optimization problems—determining the most cost-effective way to meet electricity demand while maintaining system reliability. [00:04:00]

During his early years at FERC, Dr. O'Neill observed that ISOs were using outdated software to run these critical markets. The inefficiencies were measurable and the potential for improvement was clear, but convincing market operators to adopt new technology would prove more challenging than developing the algorithms themselves.

The Conference That Started Everything

In 1999, Dr. O'Neill organized a conference at Rutgers University, bringing together ISO representatives and software developers. The goal was straightforward: demonstrate that newer optimization software could create lower costs for consumers and higher efficiency for markets. [00:04:00]

The software developers presented their tools, showing ISOs what was possible with modern optimization techniques. The technology existed, the benefits were demonstrable, and the path forward seemed clear. Yet it would take seven years before the first ISO was ready to begin testing. [00:05:00]

This delay illustrates a fundamental characteristic of innovation in critical infrastructure: the higher the stakes, the greater the resistance to change. Electricity markets serve essential functions that affect millions of people daily. Grid failures can cascade into widespread blackouts, economic disruption, and public safety issues. In this environment, "if it's not broken, don't fix it" becomes more than a saying—it's a survival strategy.

The Beta Test That Changed Everything

When PJM finally decided to test the new optimization software, they followed best practices for critical system updates: running the new and old software in parallel. This approach allowed them to compare results without risking system reliability. [00:06:00]

The parallel testing revealed savings that exceeded Dr. O'Neill's expectations. The half-billion dollars saved in six months represented a significant improvement in market efficiency, translating directly to cost savings for electricity consumers across Pennsylvania, Maryland, and 11 other states. [00:05:00]

But perhaps more importantly, the beta test provided something that academic papers and conference presentations could not: real-world proof of performance under actual market conditions. This wasn't theoretical optimization—it was demonstrated improvement in a functioning market serving millions of customers.

The Adoption Timeline: 18 Years and Counting

Despite PJM's successful results, adoption across other ISOs proceeded slowly. Some ISOs claimed their existing software was equally effective. Others prioritized different initiatives. The last ISO didn't adopt the optimization software until 2017—18 years after the initial Rutgers conference. [00:11:00]

This timeline might seem frustratingly slow, but Dr. O'Neill approaches it with the patience of someone who has worked in regulatory environments for four decades. "Every chance I get I essentially bring up the subject," he explains. "They don't turn me down. They just don't do anything. And I will just continue to pester them in some way or another until they see the light." [00:13:00]

The gradual adoption pattern reflects institutional realities that extend beyond technical merit. ISO staff must be trained on new systems. Existing workflows must be modified. Risk assessments must be conducted. In an industry where simple mistakes can cause widespread blackouts, caution is not just preferred—it's essential.

Current Work: The Next Generation

Today, as artificial intelligence applications drive unprecedented growth in electricity demand, Dr. O'Neill is working on the next iteration of market optimization algorithms. The challenge is evolving: while the original algorithms focused primarily on efficiency, current work addresses more complex market dynamics including price signals that stimulate investment in new generation capacity. [00:16:00]

The scale of the challenge is significant. According to recent estimates, AI data centers currently require about 92 gigawatts of electricity, while the entire US electricity system operates at approximately 130 gigawatts. Meeting projected AI demand could require doubling electricity generation capacity within the next decade. [00:15:00]

Dr. O'Neill's current algorithms aim to create better price signals that tell entrepreneurs where to locate new generation investments and provide more accurate estimates of potential returns. Early testing in the Midwest ISO showed an average 10% increase in prices—not an increase in costs to consumers, but better price signals that reflect true market conditions and encourage appropriate investment. [00:27:00]

The Implementation Challenge Continues

Despite having completed successful beta testing for his next-generation algorithms, Dr. O'Neill faces familiar adoption challenges. The person who championed the testing at the Midwest ISO moved to a different organization, effectively resetting the implementation timeline. [00:27:00]

This highlights one of the persistent challenges in regulatory innovation: the importance of individual champions within organizations. Technical merit alone is insufficient; successful implementation requires people within institutions who understand the technology, believe in its benefits, and have the authority to drive change. [00:21:00]

When asked what unlimited resources could accomplish in accelerating adoption, Dr. O'Neill points to structural challenges that go beyond funding. ISOs are designed to be independent market operators who don't profit from increased market efficiency. Their incentives align with reliability and regulatory compliance rather than innovation adoption. [00:23:00]

Additionally, regulatory jurisdiction creates barriers to comprehensive market reform. While FERC regulates wholesale electricity markets, retail consumers are regulated at the state level. This division of authority can complicate implementation of market-wide innovations that require participation from multiple stakeholder groups. [00:29:00]

Lessons for Other Industries

Dr. O'Neill's experience offers insights relevant to innovation in any regulated industry. Healthcare, financial services, telecommunications, and other sectors face similar challenges when implementing new technologies in critical systems.

First, technical merit is necessary but not sufficient for adoption. Organizations operating critical infrastructure have legitimate reasons for resistance to change, even when improvements are proven.

Second, implementation timelines in regulated industries should be measured in years or decades, not months. The complexity of these systems and the consequences of failure create natural friction that cannot be easily eliminated.

Third, individual champions within organizations play crucial roles in driving adoption. Technology transfer in regulated industries is ultimately a human process that requires relationship building, education, and persistence.

Finally, parallel testing and gradual implementation strategies can help bridge the gap between innovation and adoption. These approaches allow organizations to verify benefits while managing risks.

Looking Forward

As energy markets face increasing pressure to support AI growth, the lessons from Dr. O'Neill's work become more relevant. The algorithms that now generate billions in annual cost savings took 18 years to fully implement. Current energy challenges may not allow for similar timelines.

This creates a tension between the need for rapid innovation and the realities of implementing change in critical infrastructure. Resolving this tension will require not just better technology, but better approaches to technology transfer in regulated industries. [00:08:00]

Dr. O'Neill continues working on algorithm improvements while engaging with ISOs, software vendors, and policymakers. His approach remains consistent: develop proven solutions, demonstrate their benefits, and persistently advocate for adoption. [00:31:00]

The story of market optimization algorithms in electricity markets provides a case study in both the potential and the challenges of regulatory innovation. The benefits are measurable and significant—billions in annual cost savings that flow to consumers. But realizing these benefits requires navigating complex institutional environments where technical merit must be balanced against operational realities.

For innovators working in regulated industries, Dr. O'Neill's experience offers both encouragement and perspective. Significant improvements are possible, but they require patience, persistence, and understanding of the systems you're trying to change.

Dr. Richard O'Neill spent 40 years at FERC and ARPA-E working on energy market optimization. His algorithms now generate billions in annual cost savings for electricity consumers across the United States. This blog post is based on his conversation with Shannon Lantzy on the Inside MedTech Innovation podcast.

Listen to the full episode: https://open.spotify.com/show/0idCTXcel0SvjHLalRoxIl?si=3f02ec24c9594d7c

Connect with Dr. O'Neill on LinkedIn: LinkedIn | Website

Connect with Shannon Lantzy on LinkedIn:  LinkedIn | Website

This content was repurposed from the original podcast discussion by a genAI prompt.