Maintaining large-scale AI capacity at Meta

https://engineering.fb.com/2024/06/12/production-engineering/maintaining-large-scale-ai-capacity-meta/

by samber • 2 years ago

106 59 2 years ago

engineering.fb.com

Even if this scale is massive and second-to-none, it’s funny how some issues are the same for all of us. In particular “Bad hosts are very bad” aka “a chain is only as strong as its weakest link” can happen with as little as a few (4) machines, and then ruin your day.

loeg• 2 years ago

This is really unique to the AI training clusters for reasons I'm not super clear on. Most other types of horizontally scaled workloads can sort of tolerate a slightly underperforming host, or hosts going bad every so often, with little P99/P99.9 impact. For some reason, AI training workloads really cannot.

eugenhotaj• 2 years ago

This is because everyone is training with synchronous sgd. all gpus need to synchronize on each gradient step so tail latency will kill you.

chaos_emergent• 2 years ago

When training a large model you're doing a single forward pass + backprop over multiple Infiniband-connected nodes for a single model instance, so if one node goes down it takes a logical unit of nodes down with it. For reference, GPT-4 was rumored to be around 1.7T, and doing some back-of-the-hand math[1], that's like 500-700 H100 GPUs per model instance, which means you need a multiple of that for any training parallelism whatsoever.

[1] back-of-the-hand-math: 1.7T * 4 bytes = 6.8 TB; 3-4x that for activation + gradients = 27.2 TB; 27.2TB / (80GB / H100) = 349 H100s; 1.5-2x conservative multiplier accounting for not fully using node resources + memory overhead in the machine = ~500-700 H100s.

truly insane numbers.

realreality2• 2 years ago

I’m old enough to remember when companies were eager to claim that their data centers (or some aspect) were finally “carbon neutral”.

Now, with the enormous data center growth for AI purposes, companies don’t even bother pretending that any of this is sustainable.

At best, they might delude themselves into believing that a glorified text autocomplete program will magically solve the world’s problems, including the unsustainability of the machines running the program.

123yawaworht456• 2 years ago

we could exist without any of the modern conveniences. let's tear down the electric grid and return to monke.

realreality2• 2 years ago

That’s right (even though you’re probably being facetious).

AYBABTME• 2 years ago

We're way past that. Global warming requires (or will, soon enough) heat pumps for survival in many regions of the world. Plenty of regions require large amount of electricity for life critical functions. Degrowth isn't the answer.

dorkwood• 2 years ago

Could you really survive without trucks delivering food to stores on your city? I couldn't.

bloodyplonker22• 2 years ago

It's just that investors are now completely over the whole "DEI and ESG" alphabet investing type of phase after seeing that it has not helped companies produce returns at all.

Spivak• 2 years ago

ESGs are still going strong, the whole point is potentially accepting lower returns in exchange for voting with your wallet on what companies you support. Investors, the people who collect money for a living, have never been the target for ESGs.

lumost• 2 years ago

So I was pondering, NVidia quarterly datacenter revenue is around 18.4 billion. Meaning that the raw cost input to the AI industry is somewhere around 14-24 billion dollars per year post depreciation. This is against known revenues of ~3.8 Billion at OpenAI and ~800 Million at Anthropic. Based on reported revenue at cohere of 20 MM - I think it's a fair assumption that the only other material revenue in the industry is in the applications side either at megacaps or smaller startups targeting various back office tasks.

One could make a bearish claim on NVidia, that their revenue/valuation is unsustainable unless the AI industry grows 100x over the next few years.

zitterbewegung• 2 years ago

Random commenter on HN just tried to do a bearish analysis that Morningstar or any other analysis group you can obtain. News at 11.

How do we know that AI is going to be the only thing that GPGPUs are the end game?

bamboozled• 2 years ago

and that it grows using current day approaches and technologies...

m3kw9• 2 years ago

Regardless NVDA gonna pop Monday on this news

kortilla• 2 years ago

What news?

candiddevmike2• 2 years ago

It is unsustainable, but no one wants to see the music stop. AI hype is floating a lot of tech stocks right now, and NVIDIA is at the center of it.

mlinhares• 2 years ago

If they can keep the narrative, like Elon has kept the Tesla narrative, they'll have made so much money that it doesn't matter if it is sustainable or not.

hectormalot2• 2 years ago

Interesting way to look at it! At least looks like companies are investing with some expected future return that is not yet there today.

Other thoughts:

1. Current revenues might be a bit higher than you calculated. E.g. I’m not sure if copilot and azure OpenAI service are fully included in those revenues, and those might be relevant figures at this scale.

2. The 10-100x growth might actually materialize. Corporates are much slower to adopt and scale a technology than people might expect. As a result, many big potential users are only at the very beginning of adopting AI. (I am assuming there are valuable applications for them to use AI for)

chasd002• 2 years ago

Outside of your usual customer service chat bots I haven’t seen very many real genai applications in enterprises. I led a team that put one in prod that was pretty basic RAG over a knowledge base to answer questions for about 100 specialists internal to my employer. The knowledge base had to be so refined and the system prompt so tuned that any additional content added would blow it up.

There was another bot that popped up in our firm meant to answer questions about corporate policy. I’m guessing that team did something similar but over our policy docs. It vanished about 3 months of being online. It probably gave a bad answer and someone called it out to legal.

I’m curious if anyone else has seen a real application in an enterprise worthy of the hype.

ipaddr• 2 years ago

You are only calculating some sources. I'm sure other cloud venders like aws are part of the global datacenter spend. As well as many smaller private or government entities. OpenAI will continue to spend more taking a bigger percentage if supply remains flat.

tikkun• 2 years ago

Yes. Also, how much can Nvidia revenue go up from here? There are only a small handful of companies that want more than 100k H100s - if there are ten, that’s $30b of GPUs, which Nvidia would sell in less than 6 months, and then who keeps buying?

Or are we expecting a few companies to be buying 1 million plus H100s each next year

soloist11• 2 years ago

What's the failure rate of GPUs over a 10 year period?

mgraczyk2• 2 years ago

One huge oversight in this analysis is the assumption that LLMs will not get better. Imagine there is a 10% chance that LLM capabilities increase so significantly in the next 5 years that a significant portion (say 1%) of the economy will be automated by them. That's enough to make Nvidia a multi trillion dollar company.

__loam2• 2 years ago

That's a pretty big assumption. We can also assume they do not get much better and extrapolate from that, and I think that it's responsible to analyze both cases and try to ascertain which is more likely.

56% of the American economy is also based on intellectual property so it's also a big claim that the existing status quo will have nothing to say about large tech firms trying to displace that, if it is even good enough to do that.

ganeshkrishnan• 2 years ago

That's enough to make Nvidia a multi trillion dollar company.

There are tons of chip companies breathing down it's neck. AMD dropped the ball but Huawei Ascend chips are already making a dent and NVIDIA had to drop prices in China. Rest of the industry will start eating into NVIDIA as the market cap grows.

therobots927• 2 years ago

Seeing this sentiment more and more. And I share it. All bubbles pop sometime.

mrb• 2 years ago

AI generating material new revenue at megacaps is likely. If AI currently generates only 5% of Alphabet's revenue, and if 100% of the cost of this revenue is $ spent on Nvidia's hardware, then this represents 15% of Nvidia's revenue.

Seen another way, in this example, Nvidia would only need ~7 Alphabets to sustain current revenue.

modeless• 2 years ago

People are going to want to use GPT-4o style real time conversational voice AI a lot. I can see people having hour-long conversations on a regular basis within a few years. That application alone is going to need mountains of GPUs, more than are needed for training. And it will be extended to real time video input and output within a few years with another increase in GPU requirements.

nmfisher• 2 years ago

If it does play out like this, I feel that inferencing will be taken over by lower-power ASICs (e.g. Groq) or even commodity CPU hardware. GPUs are an expensive solution (both power consumption and capex) to a straightforward problem. There are already real-time TTS models running on ARM, and while the fidelity isn't on par with large GPU models yet, a year or two of hardware improvements and software optimizations will probably close that gap.

The same may not apply to audio2audio models, though, and NVIDIA will probably keep a firm grip on training, too.

notarealllama• 2 years ago

Maintenance trains, interesting takeaway from this. There's a trolley problem joke in there somewhere.

nomilk• 2 years ago

Meta is going hard into AI (both hardware and software), which is great to see. Something that's not super obvious is what specific features of existing apps require AI, that is, how will Meta get return on investment?

Two uses I can think of are i) text and image content moderation on fb and instagram (won't need as many human reviewers if bots are as/more effective), and ii) chatbots for businesses (businesses could provide their business documentation to a meta LLM which could handle customer inquiries via messenger and whatsapp).

Anything else?

ldjkfkdsjnv• 2 years ago

Their whole advertising business model gets better with LLM understanding of text. They can target ads better.

Mehdi22772• 2 years ago

This is fair guess on intuition but working in recommender space on both content/ad recommendation, content understanding signals have pretty consistently across two companies and many projects tended to underwhelm and key signals are generally engagement signals (including event sequences) and many embeddings (user embedding, creator embedding, ad embedding, etc).

The main place I’ve seen content understanding help is coldstart specially for new items by new creators.

candiddevmike2• 2 years ago

Fake profiles to boost engagement/DAU? Grandma is lonely on FB now that no one is on there anymore.

aprilthird20212• 2 years ago

This isn't it. Actually FB, the app specifically, is having an unexpected uptick in users in the younger demographics, partly from their strategy of using FB marketplace and other ancillary services to bring people back to the main app.

EDIT: Also, it's very easy to creat fake profiles on FB, and other people do it all the time. Meta don't need to do it themselves

aprilthird20212• 2 years ago

Meta actually has a whole separate, existing AI research and use case for targeting ads that has seen much better results as their AI capabilities have improved. I don't think gen AI is used for this in the way most commenters think, but the improvements in AI architecture / infra, training, etc. are all helpful to the AI which helps ad targeting while simultaneously building more powerful Gen AI

mistermann• 2 years ago

Generative environments on demand for their VR goggles seems like something that could drive net new revenue some day, both hardware and subscriptions. If they can find a PR safe way to provide adult content that would be even better.

Or, virtual conversational humans for their boomer userbase could be a hit.

xyzzy4747• 2 years ago

They could make customer facing support bots for every business with a Facebook page

dudus• 2 years ago

Meta is building and scaling the infrastructure for AI and then they can resell that at a premium later. All these articles do is highlight the challenges of rolling out your own infra, if Meta solves these issues it becomes a reference and get a bigger piece of the AI pie.

They want to become the AI backend for the Fortune 500

Havoc• 2 years ago

Sounds like this opsplanner software has a fair bit of autonomy

Narhem• 2 years ago

They make people into slaves. I’d be extremely happy when Facebook gets shutdown.

tgma• 2 years ago

We have gone full circle back to building the HPC supercomputer after quarter century of clusters.

Maintaining large-scale AI capacity at Meta

Related Stories