Entso-E final report on Iberian 2025 blackout

https://www.entsoe.eu/publications/blackout/28-april-2025-iberian-blackout/

by Rygian • 4 months ago

223 112 4 months ago

darkwater2• 4 months ago

The fact that there is not a single root cause but several ones makes me instinctively think this is a good report, because it's not what the "bosses" (and even less politicians) like to hear.

ragebol• 4 months ago

Yep, sounds like "This was bound to happen at some point"

cucumber3732842• 4 months ago

Which on some level is exactly "what the bosses and politicians want to hear"

When it's everybody's fault it's nobody's fault.

drob5184• 4 months ago

Frequently, when you see these massive failures, the root cause is an alignment of small weaknesses that all come together on a specific day. See, for instance, the space shuttle O-ring incident, Three-Mile Island, Fukushima, etc. These are complex systems with lots of moving parts and lots of (sometimes independent) people managing them. In a sense, the complexity it the common root cause.

amelius• 4 months ago

It usually starts with a broken coffee machine.

linuxguy2• 4 months ago

It's like the Swiss Cheese model where every system has "holes" or vulnerabilities, several layers, and a major incident only occurs when a hole aligns through all the layers.

https://en.wikipedia.org/wiki/Swiss_cheese_model

roenxi10• 4 months ago

See, for instance, the space shuttle O-ring incident

That wasn't really a result of an alignment of small weaknesses though. One of the reasons that whole thing was of particular interest was Feynman's withering appendix to the report where he pointed out that the management team wasn't listening to the engineering assessments of the safety of the venture and were making judgement calls like claiming that a component that had failed in testing was safe.

If a situation is being managed by people who can't assess technical risk, the failures aren't the result of many small weaknesses aligning. It wasn't an alignment of small failures as much as that a component that was well understood to be a likely point of failure had probably failed. Driven by poor management.

Fukushima

This one too. Wasn't the reactor hit by a wave that was outside design tolerance? My memory was that they were hit by an earthquake that was outside design spec, then a tsunami that was outside design spec. That isn't a number of small weaknesses coming together. If you hit something with forces outside design spec then it might break. Not much of a mystery there. From a similar perspective if you design something for a 1:500 year storm then 1/500th of them might easily fail every year to storms. No small alignment of circumstances needed.

burningChrome• 4 months ago

This is the same thing that happened with the 35W bridge collapse in Minneapolis. The gusset plates after the disaster were examined and found to be only 1/2" thick when the original design called for them to actually be 1" thick. The bridge was a ticking time bomb since the day it was built in 1967.

As the years went on, the bridge's weight capacity was slowly eroded by subsequent construction projects like adding thicker concrete deck overlays, concrete median barriers and additional guard rail and other safety improvements. This was the second issue, lining up with the first issue of thinner gusset plates.

The third issue that lined up with the other two was the day of the bridges failure. There were approximately 300 tons of construction materials and heavy machinery parked on two adjacent closed lanes. Add in the additional weight of cars during rush hour when traffic moved the slowest and the bridge was a part of a bottleneck coming out of the city. That was the last straw and when the gusset plates finally gave way, creating a near instantaneous collapse.

OgsyedIE• 4 months ago

There are ways to aggregate these into a single resilience score for policy makers with only moderate loss of detail but it's unpopular.

red_admiral• 4 months ago

Yes, a lot of modern engineering is good enough that single-cause failures are very rare indeed. That means that failures themselves are rare, but when they do happen, they're most likely to have multiple causes.

How to explain that to non-engineers is another problem.

pseudohadamard4• 4 months ago

I think a better way of explaining it to people is that we've made critical systems so reliable that, in order for them to fail, the failures have to be quite complex.

javier2• 4 months ago

we are not as complicated as the national grid, I have been here for nearly 10 years now, and our outages have gone from single cause, two causes, or now its nearly always 3 things that need to go wrong at the same time.

cameldrv• 4 months ago

This is almost universal in aviation. They always talk about the "accident chain." Essentially everything that can kill you with one mistake is illegal through training and operational requirements and engineering and maintenance regulations.

toomuchtodo4• 4 months ago

They need more battery storage for grid health, both colocated at solar PV generators (to buffer voltage and frequency anomalies) and spread throughout the grid. This replaces inertia and other grid services provided by spinning thermal generators. There was no market mechanism to encourage the deployment of this technology in concert with Spain’s rapid deployment of solar and wind.

z2• 4 months ago

There are non-battery buffers available too--I recently got rooftop residential solar installed, and learned that my area is covered by a grid profile requiring that the solar system stay online through something like 60 +/- 2Hz before shutting down completely, and ramping down production linearly beyond a 1Hz deviation or so. The point is to avoid cascading shutdowns by riding through over/undersupply situations, whereas an older standard for my area would have the all solar systems cut off the moment frequency exceeded 60.5Hz (which would indicate oversupply from power plant generators spinning faster via lower resistance).

In my system's case, switching to this grid profile was just a software toggle.

tuetuopay• 4 months ago

In this very specific case, battery storage would not have helped (in fact, it would have worsened the problem). One of the issues in the failure is renewables, but not because of intermittence. It's because of their ~infinite ramp and them being DC.

Anything that's not a spinning slug of steel produces AC through an inverter: electronics that take some DC, pass it through MOSFETs and coils, and spits out a mathematically pure sine wave on the output. They are perfectly controllable, and have no inertia: tell them tout output a set power and they happily will.

However, this has a few specific issues:

- infinite ramps produce sudden influx of energy or sudden drops in energy, which can trigger oscillations and trip safety of other plants

- the sine wave being electronically generated, physics won't help you to keep it in phase with the network, and more crucially, keep it lagging/ahead of the network

The last point is the most important one, and one that is actually discussed in the report. AC works well because physics is on our side, so spinning slugs or steel will self-correct depending on the power requirements of the grid, and this includes their phase compared to the grid. How out-of-phase you are is what's commonly called the power factor. Spinning slugs have a natural power factor, but inverter don't: you can make any power factor you want.

Here in the spanish blackout, there was an excess of reactive power (that is, a phase shift happening). Spinning slugs will fight this shift of phase to realign with the correct phase. An inverter will happily follow the sine wave measured and contribute to the excess of reactive power. The report outlines this: there was no "market incentive" for inverters to actively correct the grid's power factor (trad: there are no fines).

So really, more storage would not have helped. They would have tripped just like the other generators, and being inverter-based, they would have contributed to the issue. Not because "muh renewable" or "muh battery", but because of an inherent characteristic of how they're connected to the grid.

Can this be fixed? Of course. We've had the technology for years for inverters to better mimic spinning slugs of steel. Will it be? Of course. Spain's TSO will make it a requirement to fix this and energy producers will comply.

A few closing notes:

- this is not an anti-renewables writeup, but an explanation of the tech, and the fact that renewables are part of the issue is a coincidence on the underlying technical details

- inverters are not the reason the grid failed. but they're a part of why it had a runaway behavior

- yes, wind also runs on inverters despite being spinning things. with the wind being so variable, it's much more efficient to have all turbines be not synchronized, convert their AC to DC, aggregate the DC, and convert back to AC when injecting into the grid

cyberax• 4 months ago

One problem that happened here is the _voltage_ spikes as the synchronous generation went away. Voltage _spikes_ on generation going away seem insane, but it's a real phenomenon.

The problem is that the line itself is a giant capacitor. It's charged to the maximum voltage on each cycle. Normally the grid loads immediately pulls that voltage down, and rotating loads are especially useful because they "resist" the rising (or falling) voltage.

So when the rotating loads went away, nothing was preventing the voltage from rising. And it looks like the sections of the grid started working as good old boost converters on a very large scale.

Zopieux• 4 months ago

Nope, they need more inertial storage to smooth things out and buy time / absorb inevitable failure bursts/cascades from inverted production means or safety disconnection events.

wortelefant• 4 months ago

It is very carefully worded, but variable renewables are holding the smoking gun here. This is why spain now requests a better connection to french nuclear now. This reckless overbuild of variable generation is a valuable negative example, wind and solar without adequate hydro or nuclear is dead

TheOtherHobbes• 4 months ago

It's lack of experience managing variability, not variability itself.

Wind and solar are very far from dead, but they do need some adjustments - as the report makes clear.

anthk4• 4 months ago

Spaniard here, I didn't hear about that.

Rygian2• 4 months ago

Your statement is wrong.

The report describes that there was no mechanism to dispatch the reactive power of renewables separately from the active power.

In page 452, item numbered 1 states "RES power plants follow fixed power factor" (RES = Renewable Energy Sources). The source of this finding is in section 4.2.1.

In page 208, footnote 35, the reference is given to Royal Decree 413/2014 of 6 June, which mandates this fixed power factor. The Article 7, section e), states that renewable energy sources must follow the instructions given by the operator to set power factor, and only if the distribution lines support it.

And footnote 36 describes how this worked in practice on the date of the outage: renewables were told, by email on the previous day, which fixed power factor correction to use the following day.

This lack of dynamic dispatch of reactive power was a known problem, already reported in 2022 [1]

[1] https://www.eldiario.es/economia/competencia-reconocio-julio...

algoth1• 4 months ago

As someone who lived through the blackout it was wild. I felt back into the pre-internet, pre-smartphone era. It was pretty cool actually. The rumor mill spread so fast that Within hours the official word on the street was that we were getting hacked by a foreign military and people were joking that we had nothing of interest to be conquered xD

pfortuny• 4 months ago

The hack thing spread wildly, indeed. Weird experience.

nunobrito2• 4 months ago

In Germany a few months prior saw CCC publishing a method for destabilizing energy grids using radio waves a cheap hardware: https://media.ccc.de/v/38c3-blinkencity-radio-controlling-st... and presented an attack vector to which most infrastructure in Europe is exposed.

About 4 hours before the grid collapse on the 28th of April 2025 was recorded the largest purchase of Monero in the past 3 years (to remember: monero is coin of choice for special operations), making it surge +40% in 24 hours. The initial Spanish reports mentioned conflicting power information from dozens of locations at the same time which is consistent with a sequential attack using the blinkencity method so the grid itself is forced to close down.

madaxe_again• 4 months ago

I didn’t even know about it until the next day - totally off grid, and starlink for internet access - and no mobile signal where we live to give it away either.

bluebarbet2• 4 months ago

Might have been less fun if it had been in the depths of winter. The fact that it was a balmy sunny day in springtime made it a pleasantly novel experience, I agree. Of course, the "sunny day" seems to have been correlated.

unmole• 4 months ago

We're talking about Spain. How bad could a winter really be?

NooneAtAll32• 4 months ago

and then people accuse social media of making people paranoid...

you are able to be paranoid on your own just fine

andyferris• 4 months ago

My theory is that social media simply increases the connectivity and reach (of the “rumour mill” or what have you) and thus it amplifies an existing social “failure mode”.

(That and earlier mass media was heavily moderated and regulated, while things like Facebook or twitter/X are basically a free-for-all).

Oarch• 4 months ago

It was fun and exciting at first. However when phone batteries started getting low and the streetlights were still off you could see that changing. Candles and the relaxed Spanish attitude to life helped a lot :)

chr15m2• 4 months ago

Rumour huh.

singhrac• 4 months ago

I think people underestimate how valuable these reports are, so I’m very glad that detailed investigation is done here. Every major grid operator around the world is going to study this and make improvements to make sure this doesn’t happen on their grid.

In a lot of ways it’s like investigations into airplane crashes.

Rygian2• 4 months ago

Obligatory read: https://how.complexsystems.fail/

WJW• 4 months ago

The root cause tree on page 452 gives a good overview of how complex the behavior can be.

The good news is that the grid operators have a good idea of what the problem was/is and it's well understood how to fix it. The downside is that it will require quite a bit of both time and money to reinforce the grid infrastructure.

pseudohadamard4• 4 months ago

  Every major grid operator around the world is going to study this and make improvements to make sure this doesn’t happen on their grid.

You mistyped "Every major grid operator is going to get their lawyers to reword their contracts to make sure they can't get sued when this happens".

pjc50• 4 months ago

Plenty of grids are publicly owned, or regulatory equivalent.

holgerschurig• 4 months ago

Spotted the US-american assuming their law system is used world-wide.

If over here a lady buys a hot coffee in a McDrive, drives away, spills the hot coffee on their legs and makes a car accident due to this ... she won't be able to sue the McDrive. There's no fine-print or "Coffee is hot, you dumb person" writing needed anywhere. She could be lucky if she doesn't get fined for endangering others by her stupid actions.

So, if we have a power outage here, the courts don't suddenly get busy. Because there simply no one is suing.

Fun fact: despite this bad power outage, the power grid systems in Europe are still better (even way better) than in the US. There is a comparable statistics measure called "SAIDI" --- system average interuption duration index. And duration wise, per custom and year, the US power grids are worse than over here than in most of West Europe: (US SAIDI 2020: 1.3 hours, German SAIDI 2020: 0.3 hours). That's a factor of more than 4 on the worse-iness of US power grid!

That could be an indicator that suing at the tiniest chance isn't helpful macro-ecnomical. Or that a general suing culture (with legalese trying to protect one from the economic risks) aren't actually helping improving things in the general sense, although they reduce the risk of getting bankrupt. But society-wise, a sue culture is most probably a negative: you spend energy/time/money on things that aren't necessary in saner law systems.

jacquesm5• 4 months ago

472 pages. That's going to be a nice bit of reading this weekend. It is very nice to see such a comprehensive report as well as the fact that it was made public immediately.

dvh2• 4 months ago

Maybe practical engineering will make a video about it

jacquesm5• 4 months ago

I like reading.

wedg_• 4 months ago

I was supposed to fly home from Santiago de Compostella when the blackout happened. Me and my girlfriend had checked out of our hotel and headed to the bus stop to take the bus to the airport. The blackout had already started but we didn't realise (in hindsight, I do remember the pedestrian crossing not working. But I didn't think much of it). Anyways our flight was cancelled and it was clear we needed somewhere to stay the night.

I immediately rebooked the same hotel, but when we got back there the receptionist had left so you had to check in over the phone instead. Except WhatsApp wasn't working. Then mobile data went down. And before long we were walking through the old town going hostel to hostel looking for a place to sleep, as everything got darker and darker (due to the lack of powered street lighting). The old town in almost pitch black was pretty scary!

We ended up breaking back into the hotel, borrowing a bunch of towels from a laundry cart in the hallway and sleeping in this lockable room we found in the basement.

Besides that somewhat stressful part, it was a really strange but fun experience to see the city without power: no traffic lights, darkened shops with lots of phone lights, cafés still operating just with only outdoor seating and limited menus, the occasional loud generator, and most of all the people seemingly having a great time in spite of it.

I would've loved to have stayed out all night exploring the city, but finding somewhere to sleep that night was a bit more pressing!

AnotherGoodName3• 4 months ago

Can’t read all of this since it’s 424 pages but i want to point out that Australia is beating Europe on grid connected storage. Not on a per capita basis. It’s beating all of Europe combined outright https://www.visualcapitalist.com/top-20-countries-by-battery...

We did have many many problems previously. The state of South Australia went out for a couple of weeks at one point in similar cascading failures. This doesn’t happen anymore. In fact the price of electricity is falling and the grid is more stable now https://www.theguardian.com/australia-news/2026/mar/19/power...

This price drop is inline with the lowered usage of gas turbine peaker plants (isn’t that helpful right now? No need for blockaded gas for electricity).

A lot of people say it can’t be done. That you can’t have free power during the day (power is free on certain plans during daylight due to solar power inputs dropping wholesale prices to negative) and that you can’t build enough storage (still not there but the dent in gas turbine usage is clear).

It’s one of these cases where you’ve been lied to. Australia elected a government that listened to reports battery+solar is great for grid reliability and nuclear was always going to be more expensive.

postexitus• 4 months ago

You need grid connected storage where you have (unpredictable) renewables. That doesn't negate the benefits of Nuclear baseload power. In an ideal mix, you need both, and also Gas for emergencies. One is not better than the other, they have different roles in a balanced grid.

drtgh6• 4 months ago

Plus, related (storage), you do not want to put hydroelectric in water reservoirs targeted to population consumption, as you could find out one summer that the reservoirs are empty, the result of such water being used with the intention of generate electricity, or even used as inertial stabilizer for renewables.

This is the moment were at the news you read "There's a drought because it isn't raining" and similar excuses, when in reality your five years of water's reservoirs become reduced to half -or one third- due they focused the electricity production over the population real water demand.

I mean, hydroelectric needs at least two level’s reservoirs, one to generate electricity (or even exclusive two level's reservoirs with water pumps for this), and the next one, absolutely untouchable by the electric companies, targeted as water storage for the population/agriculture, the classic more than five years reservoir, for real.

adrianN• 4 months ago

Nuclear has a hard time existing in a net with dominant renewables during most of the year. Down-regulating nuclear absolutely kills its profitability. What you want is power plants with low capex that can be profitable with just a few hundred hours at full capacity per year. For example you can burn hydrogen.

gpm• 4 months ago

Nuclear has the same issue as (unpredictable) renewables, it is incapable of cost efficiently following the demand curve. As a result, just like renewables, it requires a form of dispatch-able power to complement it (gas, batteries, etc). Solar and nuclear fill the exact same role in a balanced grid - cheap non-dispatchable power.

Or at least nuclear would if it was cheap, but since its costs haven't fallen the same way that the costs of other energy did... well new nuclear buildout really doesn't have a good role at all right now, it's just throwing away money.

Solar and nuclear complement eachother fine - because their shortfalls (darkness for solar, high demand for nuclear) are mostly uncorrelated... a mix of non-dispatcahble power with uncorrelated shortfalls helps minimize the amount of dispatchable power you need... but batteries have made it cheap enough to transform non-dispatchable power to dispatchable power that nuclears high costs really aren't justifiable.

laurencerowe2• 4 months ago

A case can be made that nuclear could potentially be cheaper than renewables plus batteries in Northern Europe when targeting 100% zero carbon electricity. (It seems unarguable that renewables can get to 80% zero carbon electricity more cheaply).

But they're not really complementary in that one can't fill in for the gaps in the other. So the case for new nuclear gets more and more uneconomic the more cheap renewables we deploy.

Qwertious2• 4 months ago

Renewables are very predictable, they're just intermittent.

preisschild• 4 months ago

Australia elected a government that listened to reports battery+solar is great for grid reliability and nuclear was always going to be more expensive

The report you mean (csiro) was wildly biased though. They based their nuclear power cost estimate on a nuclear reactor that was never deployed anywhere (Nuscale) instead of "normal" nuclear power plants that have been deployed for decades.

mastax• 4 months ago

Was the Nuscale cost estimate somehow worse than AP1000 or EPR(2)? That seems very unlikely to me given the history of those programs.

ViewTrick10022• 4 months ago

The NuScale cost was what the project itself announced. And they hadn’t even started building yet. The latest reports also include large scale nuclear power.

I find it funny when people get outraged because all CSIRO does is use real world construction costs easily proving how unfathomably expensive new built nuclear power is.

laurencerowe2• 4 months ago

The CSIRO report appears to have cost estimates for "normal" nuclear power plants too.

    Large scale nuclear $155-$252/MWh.
    Solar PV and wind with storage $100-150/MWh.

https://www.abc.net.au/news/2024-05-22/nuclear-power-double-...

the_mitsuhiko• 4 months ago

Can’t read all of this since it’s 424 pages but i want to point out that Australia is beating Europe on grid connected storage. Not on a per capita basis. It’s beating all of Europe combined outright

That makes no sense. Those are projections and for battery only. Europe today has around 100GW energy storage, Australia has around 6GW.

AnotherGoodName3• 4 months ago

For the discussion of replacing gas peaker plants pumped hydro isn’t as useful as grid connected battery storage which is the focus of the above discussion.

NooneAtAll32• 4 months ago

If someone wants a "quick and dirty" answers - there's presentation linked https://eepublicdownloads.blob.core.windows.net/public-cdn-c...

page 11 contains "Full root cause tree" - one image with all the high level info

mythern• 4 months ago

That was quite the interesting read!

wedge01• 4 months ago

0.63 Hz and 0.2 Hz grid instability. Oh my.

christkv• 4 months ago

Lol it took about an hour for us to realize there was a blackout as our house had switched to island mode. Ended being the sort of organization point of the area for the neighborhood as we could provide mobile charging and light until the power came back around 22 at night

chr15m2• 4 months ago

That's epic, congratulations on your foresight!

anthk4• 4 months ago

In my case I was near and Bilbao the blackout ended around ~12PM while other towns around -up to 12km far- didn't got the power back until 22PM.

Everyone was outside with pocket radio receivers listening to instructions and having a good time around bars as if it were a European soccer final cup with the local team being playing...

On telecomms, I barely got 2G speeds and no Spanish newspaper (nor media) dared to set a https://lite.cnn.com or https://text.npr.org like page to read the news. I had to literaly use a Gemini (the protocol) client, Lagrange, tunneling the webs thru gemini://gemi.dev with their 'News Waffle' service. And for international news I just resorted to Gopher and magical.fish. The mainstream web barely loaded some of the text (if any), while several IM options ran at crawling network speeds. The lines were clogged and I coudn't even call my relatives.

Entso-E final report on Iberian 2025 blackout

Related Stories