TSMC Arizona outage saw fab halt, Apple wafers scrapped

https://www.culpium.com/p/tsmc-arizona-outage-saw-fab-halt

by speckx • 8 months ago

220 101 8 months ago

taurath• 8 months ago

In September

But first being reported now: It was only speculation on the financial reports before this. How quickly do they normally report disruptions like this?

I wouldn't think it would have to be too quickly since I've heard about fab disruptions from fires and such since the early 2000's. Probably just sometime after quarterly reporting to set the record straight? Why not in the report?

samus• 8 months ago

I also had the impression from the report that shareholders were miffed about this Q3 snag, so they had to publish this even though they were about to treat this as business as usual.

bob10292• 8 months ago

forcing the facility to shut down for at least a few hours
As a result, the company had to scrap thousands of wafers

Anything involving wet chemistry, photoresist, furnaces, etc. is very time-constrained. You can't let wafers sit around indefinitely. Certain process steps must be followed up very quickly to avoid scrap.

This is why you dont see redundant power for manufacturing lines. A 3nm line needs hundreds of megawatts to operate. You cant clear queued lots without a fully functional line. There's not much you could save by keeping part of the line operational.

tantalor2• 8 months ago

Good idea for a Factorio mod?

A new failure mode resets output progress back to zero if you lose power or some other input while crafting.

You could design circuit networks to cut power to non-essential systems so the rest of the factory can keep producing.

ActorNightly• 8 months ago

Someone needs to make the whole chip manufacturing process into a factorio like game and let the gamers optimize it, then build the factories around that.

sidewndr46• 8 months ago

Isn't that what spoilage is?

tetha3• 8 months ago

Some mods in modded minecraft had that and it's a very punishing mechanic unless implemented well.

It eats all of your power and usually also very expensive items very quickly usually. Assume you have like 600RF/tick generated, common with certain generator constellations. 1 tick - 1024 RF and one input consumed, crafting fails due to not enough power. 1 tick wait, 1 tick, 1024 RF and one input consumed, ... This can void 10+ items / second, which can hurt very badly. Even for common items in fact.

It also tends to kick you while you're down, because it only kicks in if everything else is already failing. Then the only thing to continue functioning is the thing voiding your energy and your expensive items. Or even worse, if you did one miscalculation about your power grid, and then all of your resources are gone, often before you can react.

It can be interesting in the right packs, but it is Gregtech level hardness.

rtkwe2• 8 months ago

Power brownouts are pretty rare outside of the very early game. It's too easy and cheap to massively over produce power for that to really harm players outside the early game so I don't think there'd be much interest. Usually brownouts rapidly develop into full blown blackouts and black restarts as your miners reduce output during the brownout often leading to a reduction in incoming fuel leading to even less power being generated in a self consuming cycle.

dylan604• 8 months ago

Mindustry has something similar with pumping various gasses/liquids through plumbing. If you accidentally mix them while building new lines, things stop working when your gases get mixed up forcing you to purge the line.

Kye• 8 months ago

A video showing those steps, for the curious: https://www.youtube.com/watch?v=dX9CGRZwD-w

It's probably not 100% identical to TSMC's process.

sevensor5• 8 months ago

It didn’t happen, but the facilities team at the fab where I worked was seriously considering installing a flywheel to cover power bumps. What I don’t get about this story is how this actually happened. All our process gasses were out in a tank farm and we knew how much pressure we had. We would have stopped the line if there wasn’t enough to proceed. Were they separating air onsite or something?

bobmcnamara• 8 months ago

Yes, Linde has an onsite plant and is building two more.

For some processes, stopping will botch the wafer. In the event of a gas shortage, do plants plan which lines to take down first, and which lines should complete a process step?

jaggederest• 8 months ago

I was very impressed by the modest little fab I worked at having thousands of lead acid batteries for momentary takeover, and 8 five-megawatt locomotive engines for longer term redundancy. Apparently their steady state usage was 25MW, which allowed still having a hot spare and concurrent downtime for two of the locomotive generator units.

j_walter3• 8 months ago

TSMC has backup generators in their AZ fab. You actually have to have backup power or a few hundred millisecond blip could cause days or weeks of tool down time. You should see what happens when you lose the ability to keep a clean room at temp/humidity/airflow...it's weeks or months.

angelgonzales• 8 months ago

This isn’t very big news. Issues occur during bring-up often. Linde’s processes are possibly so power intensive that failing over to generator power is not possible. TSMC is right to put Linde on notice since Linde should have a PFMEA and control plan to eliminate any root causes for downtime. I suspect in the long term TSMC has plans to insource this if the issue persists. Scrap happens sometimes during manufacturing, if the writer only has journalism experience and no manufacturing experience then they may not have a conceptual understanding of acceptable first pass yield. After all, the TSMC logo features failing parts!

FaradayRotation2• 8 months ago

In many ways I agree with you, but the problem statement (constrained/exhausted gas supply from vendor) makes it seems like this was not just line down, but the whole factory stopped for a few hours. Line down is a miserable migrane but still managable... while a whole factory stoppage makes a lobotomy seem like a good idea. It also sounds like there was not enough forewarning to park critical customer wafers in a "safe" stage of the process.

Even so, I also would still call this another monday at a semiconductor factory. Welcome! Here we play a nearly endless game of whack-a-mole. Here's your mallet and your towel. Now whack enough of the moles hard enough until they stop coming back (at least through the same holes). Beware the alpha moles.

By any road, I am surprised to see even this high-level perspective on a quality event disclosed to the mainstream public; I thought this was not standard practice. I enjoyed the read.

itopaloglu832• 8 months ago

Just curious, would a full factory stoppage require recalibration or revalidation of certain equipment? Or is it more like an atmospheric issue that only affects the product.

throwaway2037• 8 months ago

    > This isn’t very big news.

The opening paragraph feels a bit pearl clutching to me.

    > the company had to scrap thousands of wafers that were in production for clients at the site which include Apple, Nvidia, and AMD.

Eh. So what? I am sure scrap thousands of wafers for all kinds of other reasons. I would be better to know the cost per hour of a total plant shutdown. (Of course, I'm sure the author doesn't have this information.)

    > After all, the TSMC logo features failing parts!

Final hat tip here. I never knew that.

nutjob2• 8 months ago

After all, the TSMC logo features failing parts!

I'm not sure about that, I think the blank spaces are just parts that have been picked. The dies have been cut and the good ones are being removed.

prezk• 8 months ago

Normally a wafer would have die-sized spaces for test structures used for optical, electrical, chemical and other tests. Think the TV test card https://en.wikipedia.org/wiki/Test_card

agentifysh3• 8 months ago

seems like what is often downplayed or silent on American media is the cultural mismatch between TSMC taiwanese engineers and their american counterparts

so it always comes to those out of the loop as a bit of a surprise but from what I've read from individual Taiwanese workers and their feedback its clear that there is significant regret from one side.

and it doesn't seem to limited to just TSMC but another large company as of recent that receive icey reception for their large investment in America manufacturing.

i think this is a big reason why lot of these jobs simply wouldn't stay in america as the consumer would not be able to foot the costs added by "cultural premium" faster than what innovation can reduce.

itake5• 8 months ago

Perhaps if the US workers earned OT like TW workers do, the "culture" gap would shrink.

limagnolia• 8 months ago

Is OT "overtime"? How is it legal not to pay overtime in any US factory? Unless they are salaried (exempt)?

j_walter3• 8 months ago

TW workers have a majority of their compensation in bonuses, so the OT portion is quite small and many do not even bother to ask for it. The overall compensation between a TW and US engineer at TSMC is also significant. Not to mention the lowest paid hourly workers...where in TW they make 2-3X minimum wage, but in the US it's like 1.25X.

lazide3• 8 months ago

Hahaha. The work culture between TW and the US is night and day - and it isn’t flattering for the US.

coredog64• 8 months ago

There are like half a dozen semiconductor manufacturers in Phoenix that were here before TSMC arrived. There's a robust pipeline from ASU to these same manufacturers. Can we please just stop with the nonsensical notion that "Americans don't know how to fabricate semiconductors"?

itake5• 8 months ago

its not that the USA can't produce semi-conductors. Its that semi-conductor production, at TSMC's scale (both in terms of number of units, yield rates, and depth) currently requires highly skilled workers to work a lot of their hours to "baby sit" the wafer production.

Maybe there is a world where TSMC can hire enough skilled workers and optimize processes enabling people to go home at 5p, but that is not currently the case.

barkingcat• 8 months ago

American economics doesn't allow fabrication of semiconductors even if there is the know how.

Think about how Intel, who pioneered the know how, can't build cutting edge nodes in the levels that they need to make it profitable.

IBM had to sell their fabs to cater to the whims of "shareholders".

It's the greed of stockholders that you need to blame.

lysace3• 8 months ago

Spell it out. WTF.

tarsinge• 8 months ago

Why do you think it’s cultural and not a skill issue? For example Tim Cook said they produce in China not because of costs but skills https://www.benzinga.com/markets/25/04/44783543/when-tim-coo...

itopaloglu832• 8 months ago

I remember the example of Apple where they spent twice as much the Marshall Plan in China and created entire infrastructure to train people with high skill. So, it’s a form of stop hitting yourself game. You cannot just skip training an entire generation in the US and expect all the high skills to just exist.

behnamoh• 8 months ago

Does it further delay the launch of M5 Max/Ultra?

richisferezs• 8 months ago

guys is it me or sonnet 4.5 just became like 10x worst ?

perihelions• 8 months ago

Some HN threads about the past incidents mentioned in OP,

https://news.ycombinator.com/item?id=17686310 ("Computer Virus Cripples Several Taiwan Semiconductor Plants (bloomberg.com)"—2018, 100 comments)

https://news.ycombinator.com/item?id=19214952 ("TSMC's Photoresist Material Incident: $550M Loss (anandtech.com)"—2019, 15 comments)

Animats• 8 months ago

This was apparently a Linde installation custom built for TSMC in Arizona.[1] Nitrogen, oxygen, and argon are extracted from air on-site and purified. That's Linde's primary business; liquefying and distilling air. This isn't some little local company or a company operating outside their area of expertise.

Those gases are storeable, so it's surprising there wasn't enough tank capacity to deal with outages.

The site plan [2] shows "Gas Plant 1", and future "Gas Plant 2" and "Gas Plant 3". The gas plants are across a small road from the fab and feed the plant directly. Once Gas Plants 2 and 3 were built, there would be redundancy, but at this stage, there isn't a backup. The plan doesn't show a large tank farm, so they can't store gases in bulk.

[1] https://www.aztechcouncil.org/utility-company-makes-progress...

[2] https://semiwiki.com/forum/threads/tsmc-phoenix-arizona-fab-...

NoiseBert692• 8 months ago

Linde is huge. They can produce and offer all gases in all available purity classes.

jack_tripper• 8 months ago

Is this an ad?

astrange3• 8 months ago

Is there some reason you'd want gasses in a lower purity class?

(Well, it's cheaper.)

sevensor5• 8 months ago

Unless things have changed a lot since I fled semiconductor manufacturing, you would still need silane tanks at least. I’m as surprised as you that they don’t have buffer tanks.

cosmic_quanta2• 8 months ago

Why did you use the term "fled"? Any interesting story?

From the outside, I would love to participate in semiconductor manufacturing.

lgats• 8 months ago

Silane, with the chemical formula $SiH_{4}$, is a highly flammable and toxic inorganic gas used in industries like semiconductors, construction, and dentistry. It serves as a crucial precursor for creating high-purity silicon and is a versatile coupling agent that enhances adhesion between materials like glass and polymers. Its reactivity also allows it to be used in coatings, sealants, and various chemical syntheses.

Chemical Formula: $SiH_{4}$ Appearance: A colorless gas with a repulsive odor Flammability: Highly flammable and pyrophoric, meaning it can ignite spontaneously in air Toxicity: Very toxic by inhalation and a strong irritant to the skin, eyes, and mucous membranes

ErroneousBosh• 8 months ago

Those gases are storeable, so it's surprising there wasn't enough tank capacity to deal with outages.

It probably depends on the duration of the outage. I'd expect they have some storage, and if they plan on having the compressor plant down for longer that that can manage they'll bring in tanks.

caycep• 8 months ago

the rebels have hit the tibonna gas supply I see

therealfiona• 8 months ago

Linde shut down the shop that made these machines. So I'm sure they're shaking in their boots about being put in notice...

TSMC Arizona outage saw fab halt, Apple wafers scrapped

Related Stories