Topics
Latest
AI
Amazon
Apps
Biotech & Health
Climate
Cloud Computing
Commerce
Crypto
Enterprise
EVs
Fintech
Fundraising
Gadgets
Gaming
Government & Policy
Hardware
Layoffs
Media & Entertainment
Meta
Microsoft
Privacy
Robotics
Security
Social
Space
Startups
TikTok
Transportation
Venture
More from TechCrunch
Staff
Events
Startup Battlefield
StrictlyVC
Newsletters
Podcasts
Videos
Partner Content
TechCrunch Brand Studio
Crunchboard
Contact Us
One fallen power line exposed a growing AI data center problem. Here’s how to fix it.
A power line went down outside of Washington, DC, this week. Normally, the grid would only need a few seconds to recover from such an event. But this one took more than 10 minutes because more than 3 gigawatts of data centers stopped drawing power nearly simultaneously.
The event caused voltage across the PJM grid to spike from Northern Virginia to Chicago, according to data collected by Ting Labs, a startup that runs an IoT sensor network out of people’s electrical sockets.
The event didn’t cause a blackout, but it did cause lights across the region to flicker. The incident demonstrated the effect that data centers can have on the grid — an outcome that experts believe will become more frequent.
Northern Virginia, which is in PJM’s territory, is home to the highest concentration of data centers in the world.
“It’s the canary in the coal mine,” Ricardo de Azevedo, CTO at ON.Energy, told TechCrunch. These sorts of events involving large loads like data centers are “happening more and more,” he added.
The event echoes one that happened two years ago, also on PJM’s grid, and it could foreshadow larger events if data centers aren’t built to more elegantly handle disruptions to power supplies. The PJM Interconnection manages grids from New Jersey to Illinois and serves 67 million customers, making it the largest grid operator in the United States.
When the power line went down this week, it triggered data centers to switch to backup power, and about 3.1 gigawatts of load vanished in about 30 seconds, according to PJM data. The grid appeared to recover somewhat, but a short time later additional loads dropped off. At its peak, PJM’s grid had an extra 3.49 gigawatts of electricity on it. It took another 11 minutes before it stabilized. The disconnected data centers represented around 3% of total demand on PJM at the time, according to Reuters.
A few percent may not sound like much, but the electrical grid needs to operate in a state of near-perfect balance, with supply and demand closely matched. If they don’t, voltages can sag or spike. The grid and devices connected to it can tolerate small fluctuations, but if those fluctuations grow too large, they’ll trigger failsafes within the grid or within individual facilities, causing them to disconnect.
When data centers in Northern Virginia sensed the fluctuation caused by the failed power line, they switched to backup power, which removed their load from the grid. As more data centers made the switch, they removed greater amounts of load from the grid. What started as a relatively small drop in supply became an even larger drop in demand, sending supply surging and causing light bulbs to flicker.
Most data centers make decisions in a split second, and those that disconnected this week appear to be no different. When the voltage dip reached them, they all decided to disconnect within a few seconds of each other, Ali Zain Banatwala, senior market models specialist at the Independent Electricity System Operator, told TechCrunch.
“We need to figure a way for these loads that are located next to each other to sequentially either disconnect or reconnect,” he said. A more orderly process would allow grid operators to develop more robust procedures in advance.
Alternatively, data centers could be built to absorb disruptions and not turn their backs to them. One startup, ON.Energy, has been working on a product to help data centers — and the grid — ride through events like the one that occurred this week.
The company has developed an uninterruptible power supply for an entire data center campus, covering not just servers but also chillers and other equipment. The company essentially hides the data center behind a bank of batteries connected to sophisticated power conversion equipment. All the grid “sees” is one consistent, well-behaved load rather than the peaks and valleys from each individual part of the data center. ON.Energy’s system allows data centers to ramp computing workloads up and down, including AI training, without bothering the grid.
Perhaps more important, it also means that data centers can absorb power fluctuations from the grid. Rather than disconnecting from the grid, ON.Energy’s system can use any extra power to charge its batteries, and if the flow dips, the system can dispatch power to servers. Plus, it can follow the grid’s lead within milliseconds, preventing sags or surges like the ones that caused this week’s problem for PJM.
ON.Energy is currently installing a total of 3 gigawatts worth of its systems at four different data center campuses, de Azevedo said.
Grid managers have also woken up to the problem.
ERCOT, for example, is going to require large loads like data centers to “ride through” disruptions, de Azevedo said.
The clock is ticking, though. The mass disconnection this week was twice as large as a similar event in 2024, when 60 data centers simultaneously disconnected, pulling 1.5 gigawatts of load from the grid. Back then, data centers accounted for about 6% of PJM’s load, according to Synapse Energy Economics. By 2040, they are expected to make up 24%. If the problem isn’t addressed soon, things could get a lot worse.
according%2024-089.pdf)
Topics
When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.
we may earn a small commission
Senior Reporter, Climate
Tim De Chant is a senior climate reporter at TechCrunch. He has written for a wide range of publications, including Wired magazine, the Chicago Tribune, Ars Technica, The Wire China, and NOVA Next, where he was founding editor.
De Chant is also a lecturer in MIT’s Graduate Program in Science Writing, and he was awarded a Knight Science Journalism Fellowship at MIT in 2018, during which time he studied climate technologies and explored new business models for journalism. He received his PhD in environmental science, policy, and management from the University of California, Berkeley, and his BA degree in environmental studies, English, and biology from St. Olaf College.
You can contact or verify outreach from Tim by emailing tim.dechant@techcrunch.com.
Scale faster. Grow your portfolio. Gain practical expertise. No matter your goal, Disrupt can empower you.Save up to $330 today!
Most Popular
US accuses American of allegedly wiping his phone using a ‘duress’ password during border search
Zack Whittaker
US accuses American of allegedly wiping his phone using a ‘duress’ password during border search
US accuses American of allegedly wiping his phone using a ‘duress’ password during border search
US accuses American of allegedly wiping his phone using a ‘duress’ password during border search
Zack Whittaker
Anduril reportedly in talks to raise funding at $100B valuation, more than 3x last year’s mark
Ram Iyer
Anduril reportedly in talks to raise funding at $100B valuation, more than 3x last year’s mark
Anduril reportedly in talks to raise funding at $100B valuation, more than 3x last year’s mark
Anduril reportedly in talks to raise funding at $100B valuation, more than 3x last year’s mark
Ram Iyer
Tesla’s robotaxis are moving in reverse
Sean O'Kane
Tesla’s robotaxis are moving in reverse
Tesla’s robotaxis are moving in reverse
Tesla’s robotaxis are moving in reverse
Sean O'Kane
Jack Dorsey is taking on Slack with Buzz, a group chat platform for teams and their AI agents
Amanda Silberling
Jack Dorsey is taking on Slack with Buzz, a group chat platform for teams and their AI agents
Jack Dorsey is taking on Slack with Buzz, a group chat platform for teams and their AI agents
Jack Dorsey is taking on Slack with Buzz, a group chat platform for teams and their AI agents
Amanda Silberling
Light made a flip phone — it’s colorful and it’s cheap
Amanda Silberling
Light made a flip phone — it’s colorful and it’s cheap
Light made a flip phone — it’s colorful and it’s cheap
Light made a flip phone — it’s colorful and it’s cheap
Amanda Silberling
AI music generator Suno breach affects 55M users, per Have I Been Pwned
Zack Whittaker
AI music generator Suno breach affects 55M users, per Have I Been Pwned
AI music generator Suno breach affects 55M users, per Have I Been Pwned
AI music generator Suno breach affects 55M users, per Have I Been Pwned
Zack Whittaker
Judge pauses $110B Paramount-Warner Bros. merger
Aisha Malik
Judge pauses $110B Paramount-Warner Bros. merger
Judge pauses $110B Paramount-Warner Bros. merger
Judge pauses $110B Paramount-Warner Bros. merger
Aisha Malik