So you want to rent an NVIDIA H100 cluster? 2024 Consumer Guide

https://www.photoroom.com/inside-photoroom/so-you-want-to-rent-an-nvidia-h100-cluster-2024-consumer-guide?slug=inside-photoroom/so-you-want-to-rent-an-nvidia-h100-cluster-2024-consumer-guide&_storyblok_published=511470179&

by ea016 • 2 years ago

297 139 2 years ago

photoroom.com

8organicbits• 2 years ago

Pretty sparse on pricing data, I guess everyone asked them to keep it private.

Tepix• 2 years ago

Genesiscloud.com mentions „starting at @2.00/hr“ for HGX100 H100.

spott3• 2 years ago

Only 200Gbps network per node.

The infiniband stuff is 400Gbps per GPU (3.2Tbps per node).

latchkey32• 2 years ago

Compute pricing isn't really private. When you're talking about high end compute, pricing is very much case by case. What is the point of posting it if changes due to everyone having different needs?

We have base pricing on our website, but I guarantee that if someone comes to me asking for a year reservation, I'm not going to give the quoted price. What I have there is just a good starting point to get the discussion going.

I also had a great dialog with GC on LI over their version of this posting, it seems they really value this customer and it is long term relationship. My assumption is the actual pricing reflects that.

One other thing on the special needs, GC mentioned they had 3 extra spare chassis in play as uptime was critical. That is not an insignificant amount of investment to have just laying around.

spott3• 2 years ago

https://gpulist.ai

No idea about how accurate that is, but if you want cluster pricing...

latchkey32• 2 years ago

I got them to add the Verified badge, but the rest is pretty much about as accurate as craigslist.

Der_Einzige2• 2 years ago

I hate how all high end markets wage wars on price discovery. Most sell their products through middle-men instead of directly for a reason.

High end furniture? Suddenly prices go away and you have to "get quotes".

High end GPUs? Suddenly you learn that spot pricing =/= website quoted prices =/= (actual prices paid with volume + related discounts)

I regularly talk to suits who are paying $$$ for knowledge about the GPU market who are somehow still in the belief that a single 1xA100 80GB costs 13$ an hour to rent through AWS. When I tried to correct them, they almost seemed not to believe me.

Things that take us tech bros minutes (checking the up-to-date price data by going to the screen in your cloud console to spin one up) or hours (emailing your cloud rep for pricing data with discounts) take suits years to poorly approximate knowledge of.

If I go into a market, and price discovery isn't easy on a product, I know I'm dancing with a good chance of being scammed, and by the most greedy, comic-book evil kind of rich people. Suits aren't immune to this, and I'm certainly not either.

lotsofpulp2• 2 years ago

Because it is always in a sellers best interest to “price discriminate”. A seller benefits most by selling at the highest price that each buyer is willing to buy at, and each buyer has a different highest price they can or are willing to pay, so price transparency works against being able to price discriminate.

https://en.wikipedia.org/wiki/Price_discrimination

Buyers obviously benefit from price transparency. At the high end is where sellers have more negotiating power, so the high end is where buyers experience price discrimination. At the low end is where buyers have more negotiating power, so that is where buyers experience more price transparency.

latchkey32• 2 years ago

Great post. The ethernet section is especially interesting to me.

I'm building a cluster of 16x Dell XE9680's (128 AMD MI300x GPUs) [0], with 8x 2p200G broadcom cards (running at 400G), all connected to a single Dell PowerSwitch Z9864F-ON, which should prevent any slowness. It will be connected over rocev2 [1].

We're going with ethernet because we believe in open standards, and few talk about the fact that the lead time on IB was last quoted to me at 50+ weeks. As kind of mentioned in the article, if you can't even deploy a cluster the speed of the network means less and less.

I can't wait to do some benchmarking on the system to see if we run into similar issues or not. Thankfully, we have a great Dell partnership, with full support, so I believe that we are well covered in terms of any potential issues.

Our datacenter is 100% green and low PUE and we are very proud of that as well. Hope to announce which one soon.

  [0] https://hotaisle.xyz/compute/

  [1] https://hotaisle.xyz/networking/

omneity• 2 years ago

Did you procure the servers directly from Dell or through a distribution partner?

latchkey32• 2 years ago

Kind of both. We started talking to Dell first and then they introduced us to Advizex. We are now effectively partnered with both companies, which is fantastic as the Advizex team have ex-Dell people working directly with us. We are lucky to have two whole teams of super talented people helping us out on this journey.

logicchains• 2 years ago

Meta had success building an ethernet cluster on Arista 7800 with Wedge400 and Minipack2 OCP rack switches.: https://www.datacenterdynamics.com/en/news/meta-reveals-deta...

latchkey32• 2 years ago

I actually have a call with Arista next week to learn more about their solutions. Especially those 7800's. They look awesome.

One "problem" we have right now is that our cluster cannot support more than 128 GPUs. If we wanted to scale with Dell, we'd have to buy 6x more Z9864F to add one more cluster, which is crazy expensive and complicated.

I want to see if Arista has something that can help us. That said, I also have to find a customer that wants more than 128 MI300x and that hasn't happened... yet.

csmpltn• 2 years ago

"Our datacenter is 100% green"

Cool, where can I read more about this? How do you power your DC?

walrus012• 2 years ago

Plenty of datacenters that are somewhere generally in the pacific NW can claim to be "Green" because their power supply is entirely hydroelectric.

https://www.nwd.usace.army.mil/CRWM/CR-Dams/

Many of those areas also happen to have the lowest $ per kWh electricity in North American, the only lower rate is available near a few hydroelectric dams in Quebec.

latchkey32• 2 years ago

We haven't announced the dc yet, but will soon. Very well known. I'm actually pretty excited about it.

epistasis5• 2 years ago

I have no idea how they are powering it, but with the speed with which solar and battery prices are falling, and the slowness of getting a new big grid interconnection, I would not be surprised to see new data centers that are primarily powered by their own solar+batteries. Perhaps with a small, fast and cheap grid connection for small bits of backup.

If not this year, definitely in the 2030s.

Edit: for a much smaller scale version of this, here's a titanium plant doing this, instead of a data center. The nice thing about renewables is that they easily scale; if you can do it for 50MW you can do it for 500MW or 5GW with a linear increase in the resources. https://www.canarymedia.com/articles/clean-industry/in-a-fir...

dpflan• 2 years ago

What are you using these for? Providing compute for customers that want to train/infer? What is the level of interest and level of success customers are seeing using these services Hot Aisle offers?

latchkey32• 2 years ago

We are a bare metal compute offering. They can be used for whatever people want to use them for (within legal limits, of course). Interest is much higher now that we've started to work with teams publishing benchmarks which show that H100's have a nice competitor [0].

I'll admit, it is still early days. We just finished up another free compute [1] two week stint with a benchmarking team. One thing we discovered is that saving checkpoints is slow AF. I'm guessing an issue with ROCm. Hopefully get that resolved soon. Now we are in the process of onboarding the next team.

  [0] https://hotaisle.xyz/benchmarks-and-analysis/

  [1] https://hotaisle.xyz/free-compute-offer/

rlupi5• 2 years ago

https://docs.nvidia.com/dgx-superpod/reference-architecture-...

NVIDIA large GPU supercomputers have separate compute-networking (between GPUs) and storage-networking (storage to GPUs, or storage to SSD, SSD to GPUs with CPU assistance). This helps avoid networking issues, even more if not using Infiniband.

From what I read here and on your website, you don't go that route. I haven't found the equivalent system level reference architecture for MI300x from AMD. I wonder if you have a link to a public document where AMD provides guidance about this choice?

derefr• 2 years ago

even more if not using Infiniband

It's interesting that the above "HPC reference architecture" shows a GPU-to-GPU Infiniband fabric, despite Nvidia also nominally pushing NVLink Switch (https://www.nvidia.com/en-us/data-center/nvlink/) for the HPC use-case.

latchkey32• 2 years ago

We have a separate OOB/east-west network which is 100G and would be used for external storage. We're spending an absurd amount of money on just cables.

It is documented on the website [0], but I do see that I did not document the actual cards for that, will add when I wake up tomorrow. The card is:

Broadcom 57504 Quad Port 10/25GbE,SFP28, OCP NIC 3.0

As far as I know, AMD doesn't really have the docs, it is Dell. Their team actively helped us design this whole cluster.

We haven't decided on which type storage we want to get yet. It'll really depend on customer demand and since we haven't deployed quite yet, we are punting that can down the road a bit. Our boxes do all have 122TB in them and we have some additional servers not listed as well with 122TB... so for now I think we can cobble something useful together.

[0] https://hotaisle.xyz/networking/

RobRivera• 2 years ago

Nice! Whats benchmark standard these days? Still Superbench or yall have something inhouse?

Re: lead time quote :O I guess I got spoiled working for one of the major cloud vendors. The thought of poor b2b vendor support never entered my risk matrix.

If you own your own cluster, the network bottleneck becomes less a dollar cost I suppose, since you arent being charged a premium to rent someone elses compute

latchkey32• 2 years ago

We don't run benchmarks ourselves. We donate the expensive Ferrari worth of compute for others to do it. This is the most unbiased way I could think of getting useful real world data to share.

The 3rd team just finished up and the 4th is getting started now. I've got 23 others in the wings.

https://hotaisle.xyz/free-compute-offer/

pheatherlite• 2 years ago

Being out of the loop for awhile. Has amd made anything similar to cuda? Are cots frameworks such as pytorch and tensorflow on par when running on amd hardware? What makes investing in amd cluster/chips worthwhile?

jononor2• 2 years ago

ROCm is the framework. Both PyTorch and Tensorflow have versions that support it. Have no experience with it, so cannot say how it works in practice.

JackYoustra• 2 years ago

Why xeon instead of epyc?

latchkey32• 2 years ago

Sadly, the only option Dell supports today. To get to market the fastest, they took their existing H100 chassis solution, swapped out the GPUs/baseboard and called it a day.

startupsfail• 2 years ago

It seems the reliability, speed and scalability drops with the Ethernet are somewhat manageable.

To quote the article - From our tests, we found that Infiniband was systematically outperforming Ethernet interconnects in terms of speed. When using 16 nodes / 128 GPUs, the difference varied from 3% to 10% in terms of distributed training throughput[1]. The gap was widening as we were adding more nodes: Infiniband was scaling almost linearly, while other interconnects scaled less efficiently.

And then they do mention that the research team needs to debug unexplained failures on Ethernet that they’ve not seen on Infiniband. This actually can be the expensive part. Particularly if the failures are silent and cause numerical errors only.

latchkey32• 2 years ago

A single switch should mitigate some of the throughput issues.

As for issues, this is why I have a full professional support contract with Dell and Advizex. If there are issues in the gear, they will step in to help out.

Especially on the switch, since it is a spof, we went with a 4 hour window.

teaearlgraycold4• 2 years ago

We just set up a small cluster of our own. We’re not using infiniband but it didn’t seem like it would be a 50 week lead time to setting it up. Where did you get that number?

latchkey32• 2 years ago

I hear a couple issues with your comment... "small cluster", "we're not using infinband"

I'm only saying what was quoted to me. The cards are easy to get, it is the switches that are more difficult.

zxexz9• 2 years ago

If you’re OK with used equipment, I find Infiniband basically on par pricewise with used ethernet gear. Mellanox stuff is super easy to test and you can run everything with a mainline kernel. MQM8700/MQM8790 switches can be had really cheap now used. Cables are dumb expensive...except a dime a dozen used. Or buy new, 100% compatible from FS or the likes. ConnectX-6 cards are quite cheap used thanks to HFT firms constantly upgrading to the latest, and many can be set to Infiniband OR Ethernet (I _think_ some of the dual port cards suppirt simultaneous). I set up a cluster that right now has like 70 of these, at least half of them used. Have not had an issue with any of them yet (I botched a couple firmware updates but every time I saw able to just reset the card and push firmware again). Every machine is connected to the IB fabric AND to at least 100G ethernet.

FWIW, used 100G Ethernet equipment is now cheap enough I’ve been upgrading my home network to be 100G. Cheaper than new consumer 10G equipment.

latchkey32• 2 years ago

If you’re OK with used equipment

I'm building a business, not a home lab.

(before you continue to downvote me, read what I wrote below)

smivan2• 2 years ago

Can you share some vendors and model names? I've been looking into upgrading my home lab from 10g to 100G, but the costs were absolutely astronomical.

I'm likely running a personal lab similar to many folks on HN - about 20-30 wired servers and a small rack of managed Unifi switches.

Appreciate you!

huqedato• 2 years ago

The only essential aspect this article doesn't answer: How much does it cost? All the rest is metadata. I would have preferred a clear table with vendors, prices and features. And less bla-bla.

ea016• 2 years ago

I couldn't share any pricing data since the discussions with providers are private. Instead, I added a graph of prices from gpulist.ai. For an Infiniband cluster, median is $2.3 per H100 hour, average is $2.47.

ilaksh• 2 years ago

$2.47 * 256 * 24 * 30 = $455k ?

silverlake• 2 years ago

Good info! I use an HPC with SLURM. 40k GPUs shared by hundreds of users. It works well enough. I don’t know how the market for cloud-based clusters works. Why didn’t OP use AWS or Google for on-demand training? Is it just down to cost?

macksd• 2 years ago

If you do, in fact, need H100s, they can be very hard to get. Even the smaller flavors of A100 you sometimes request, wait days for, and then 1 node might show up during a weekend. And for the reasons described in the article and the fact that large training jobs can be network-limited, nicer networks can be a big deal.

eigenvalue3• 2 years ago

Lots of good and detailed information here, thanks. I'm curious why Ethernet interconnect is so unreliable in practice compared to the Infiniband. I would think that at this point, after a decade or more of current Ethernet standards, all the kinks would be worked out and the worst that would happen would be occasional latency spikes and a few lost packets that could be retransmitted quickly. Shouldn't the training frameworks be more robust to that sort of thing?

rlupi5• 2 years ago

Infiniband and ethernet are very different at the lowest levels. Ethernet interconnects use RoCE (RDMA over converged ethernet), which actually encapsulated infiniband transport in ethernet, but you still pay for higher routing latency, and you need separate compute-network and storage-network to avoid queueing (lossless ethernet).

https://community.fs.com/article/infiniband-vs-ethernet-whic...

https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet

Also... don't underestimate the PCI bus bottlenecks when you put 8x 400GB networking + 8x GPUs. There are ways now to have a tree of PCI switches and avoid overloading the main one, each GPU gets its own networking card and PCI switch.

latchkey32• 2 years ago

This is a great comment.

Our cluster is 128 GPUs into a single Dell switch... should help with the queuing. We also have a separate e-w 100G network.

This is why we went with Dell XE9680 chassis... people forget that PCI switches are quite important with this level of compute. Dell has done a good job here.

eigenvalue3• 2 years ago

Interesting, thanks. From the wikipedia link, this seems like the probable culprit for why things break:

"Although in general the delivery order of UDP packets is not guaranteed, the RoCEv2 specification requires that packets with the same UDP source port and the same destination address must not be reordered."

shaklee32• 2 years ago

RoCE does not encapsulate infiniband

cavisne2• 2 years ago

ML training is a tightly interconnected HPC style workload (network bound), which is a market Infiniband has been targeting for a long time.

Nvidia saw that and bought Mellanox, and made NCCL/GPU's work really well with Infiniband.

Large public clouds already have huge investments in ethernet and don't want to be further locked into Nvidia, so Nvidia does have a roadmap for ethernet GPU clusters (roughly 1 year behind Infiniband).

But if you are building your own Nvidia cluster, it would be silly to build it on ethernet. Just buy exactly what Nvidia recommends, you are already locked in anyway.

ec109685• 2 years ago

How do the large clouds compare from an availability and cost perspective compared to finding a smaller provider and renting a dedicated cluster?

choppaface• 2 years ago

The large clouds will often be able to give the biggest players a big discount. Perhaps not on raw GPU prices (maybe extended trial), but if you already have e.g. 1PB in object storage then they might give you 20-30% discount. Moreover if your contract is jumbo, they’ll give you not only a Slack channel but send Forward Deployed Engineers to your office and/or help you build part of the model training software.

But for a deployment the size of the OP (Photoroom) I doubt any of the big clouds would offer a discount. Especially if they were not already negotiating with multiple clouds.

Probably the best argument for going with a large cloud provider on a smaller budget is that you already use some of their other services significantly and your MLE-to-devops headcount makes something like Photoroom’s test infeasible.

cavisne2• 2 years ago

Large clouds tend to be not very good for GPU clusters. The security & management of multi-tenant GPU's is very complex. They have to buy from Nvidia so there is no negotiating leverage. And they know you will be a high maintenance customer (wanting all nodes to work at all times).

So there is no big price or availability advantage for a large cloud (unless you are large enough to rent a dedicated cluster from a large cloud)

barbazoo4• 2 years ago

Electricity sources and CO2 emissions

I love that they included this in their consideration and pointed out the impact running these GPUs has on the environment.

storyinmemo• 2 years ago

Shameless employer promotion here while I work on H100 clusters today: https://www.datacenterdynamics.com/en/news/crusoe-puts-cpus-..., https://crusoe.ai/cloud/

Iceland seems to have an excess of energy to population and it's very green.

barbazoo4• 2 years ago

I applaud the vision, I wish you folks hired remote software devs

ai4ever• 2 years ago

so, ex bitcoin miners are pivoting into gpu clouds

whats the point ?

123yawaworht456• 2 years ago

it has fuck all any impact if the electricity is sourced from nuclear/hydro/solar/wind/geothermal.

barbazoo4• 2 years ago

Well, exactly, that's their point. If possible, choose a data center location with access to renewable energy.

qeternity• 2 years ago

Energy is largely fungible in aggregate, so this ends up not being true.

But I do agree with your implicit point that we should be emphasizing renewable generation instead of energy rationing.

Jun8• 2 years ago

Say you want to burn about $500 as a curiosity project for 8 nodes for a day. Any suggestions for what job to run?

teaearlgraycold4• 2 years ago

If you’re just burning money you might as well mine crypto.

turtles3• 2 years ago

As a random thought, this seems to be about the same order of magnitude compute as Karpathy's recent GPT-2 work:

https://github.com/karpathy/llm.c/discussions/677

You could take the final checkpoint from that page and run it for some additional steps and see if it improves? You could always publish the final checkpoint and training curves - someone might find it useful.

fragmede• 2 years ago

You could benchmark how fast you can count the number of words (and characters and lines) in all of project gutenberg with wc-gpu.

https://github.com/fragmede/wc-gpu

So you want to rent an NVIDIA H100 cluster? 2024 Consumer Guide

Related Stories