V100 Server On-Prem vs. AWS P3 Instance Cost Comparison

https://lambdalabs.com/blog/8-v100-server-on-prem-vs-p3-instance-tco-analysis-cost-comparison/

by rbranson • 7 years ago

141 125 7 years ago

lambdalabs.com

purplezooey• 7 years ago

damn, "includes hiring a part time system administrator".

fisherjeff• 7 years ago

Must be extremely part-time, for $10k/year total cost.

Scoundreller2• 7 years ago

They're hiring a share of colo staff.

It's a comparison to AWS, so everything that you install/do/operate on that server is extra in both cases.

Spivak• 7 years ago

I mean $10k/yr/server doesn't sound too unreasonable. Just sounds really bad when you're only looking at one server.

$100k/yr total compensation (i.e. 60k-ish salary) for someone babysit 10 servers isn't super unreasonable.

vidarh12• 7 years ago

If a single server requires anywhere close to that in sysadmin time per year, it is broken or the sysadmin in question is incompetent, or we're talking a supercomputer of much greater complexity than this thing.

To me it seems like a very conservative estimate, or allowing for said sysadmin to provide a lot of valueadd services (e.g. devops type services) that you'll typically need for a cloud setup as well.

bithavoc2• 7 years ago

Buy servers if you have stable workloads, otherwide rent virtual machines in the cloud.

wenc2• 7 years ago

The article also assumes 100% utilization on the cloud. I wonder if continual GPU-based training of DNN models is perhaps a fairly circumscribed use case?

Most DNN model training workloads are lumpy and transient.

ramraj07• 7 years ago

Practically though the moment you use an instance more than 50% of the time AWS incentivizes buying the annual plan.

NotAnEconomist• 7 years ago

If I'm reading the chart right, GPU servers on AWS are cheaper if you utilize them less than 65-75% of the time, and turn them off when not in use.

bithavoc2• 7 years ago

I think so, if you can afford turning off a server during the night then it makes sense to take advantage of some sort of hourly billing which is what most clouds offer by default.

rocky1138• 7 years ago

Using spot instances on AWS is a good way to save money on infrequent tasks, as well.

vidarh12• 7 years ago

I'd add to that: Most people have far more stable workloads than they like to think.

"Everyone" seems to think they have massively spiky loads, and almost nobody does. There may often be demands that could spike (e.g. because "someone" has not randomized the start times for batch maintenance processes through the night but start them all at the same time), but more often than not the spikiest demand comes from batch workloads where nobody cares if it takes a bit longer. In practice when constrained that often simply smooths out the overall capacity use.

(of course that does not mean that there aren't people with spiky enough workloads for it to make sense to put at least some of their workloads on VMs for cost reasons; but measure first)

Dunedan• 7 years ago

Is that really the case?

So far all companies I worked for had a wave-like pattern for customer generated workloads, simply because of the locality of the customers. It doesn't even matter if it's a globally used product, as some markets are usually stronger than others (e.g. US vs. Asia). So you have some base load, which is relatively stable, but on top of that daily or even more frequent "spikes".

For non-customer-driven workloads you even might want to have spikes. Take big data pipelines for example. While you could schedule them so that they're not spiky, you might benefit from running them as fast a possible to get your results earlier. That's something you probably wouldn't do with dedicated servers, simply because it'd be too expensive. Instead with dedicated servers you'd optimize for utilization instead of speed. When you only pay for what you use, you don't have that limitation and can fire up as many instances as it makes sense. That's actually a pretty nice benefit.

And don't forget changing workloads over the course of multiple months. The calculation looks completely different if you buy a dozen of servers now, but have your computing requirements change in a few months, so that you don't need them all anymore (reasons for such are manifold, but could be as simple as optimizations to your code).

sabalaba2• 7 years ago

I should probably change the first sentence in the article to this.

ilaksh2• 7 years ago

This is just the most extreme example. AWS is just really expensive.

If you want a VPS take a look at Digital Ocean or Linode.

wenc2• 7 years ago

I think the expense is mostly because these are GPU instances, which are not yet commoditized. Unlike VMs, multi-tenancy on GPUs is just a little bit harder.

ilaksh2• 7 years ago

Regular AWS EC2 is still a lot more expensive than the alternatives that I mentioned.

Namidairo• 7 years ago

The 15 tables in the nvidia grid documentation kind of shows how much of a mess it is.

Different resolutions and maximum tenants depending on card and license type, and you can't allocate resources homogeneously.

vidarh12• 7 years ago

Actually the savings on this server looks to me to be low compared to what I'd usually expect.

You should use AWS for convenience, not cost. They're expensive for cloud services, and even the cheapest cloud providers are expensive compared to renting dedicated for all but the most transient workloads, and of dedicated hosting providers I only know Hetzner to get close to the costs I could get for renting colo space or doing truly on-prem hosting. Even then the only reason Hetzner is competitive is because I'm in London where space/power is expensive, and they're in Germany, where it is cheap (e.g. they rent out colo space as well, and prices are at 1/3 to 1/4 of what I've paid in London).

hughesjo2• 7 years ago

It's not fair comparison unless we are comparing all in costs that include ops.

riku_iki3• 7 years ago

They include ops cost for on-prem server

dc_gregory2• 7 years ago

I'm inexperienced in the hardware front, would that machine likely not break down under a large load for 3 years straight? Nothing is set aside for hardware failure etc.

taspeotis• 7 years ago

Depends what the warranty is, I think three years is standard and you can pay for five or seven. (You pay commensurately, however.)

dc_gregory2• 7 years ago

Ah, excellent point, a warranty is something I completely overlooked. Probably some value "lost" doing it yourself due to downtime if something goes wrong, but likely negligible.

baroffoos• 7 years ago

Our on prem server has been running for about 7 years on a high load. Its a little unpredictable but I would expect a server to last longer than 3 years

iheartpotatoes2• 7 years ago

But isn't that $69,000 peanuts compared to the cost to hire sysadmins who are on call 24/7 to swap out: RAM, fans, drives, power supplies, provision new images, etc.?? So you saved on cloud costs, but now you have the admin burden: 1-3 people for $200k each. No?

badestrand• 7 years ago

From the article: "Our TCO [(Total Cost of Ownership)] includes energy, hiring a part-time system administrator, and co-location costs"

rossdavidh• 7 years ago

If you have only a single server, and you need 24/7 support, then probably it is true you don't want to hire a full-time sysadmin for one server. But, for the people who are doing this kind of thing, they probably have more than one, so the cost of sysadmin is spread across more than one server.

mbrumlow• 7 years ago

Our TCO includes energy, hiring a part-time system administrator, and co-location costs. In addition, you still get value from the system after three years, unlike the AWS instance.

They covered that I think.

slededit• 7 years ago

A lot of times when you colo you can get staff there to take care of those sorts of tasks. Their cost breakdown includes using this service.

vidarh12• 7 years ago

From having managed several racks worth of equipment in two separate data centres 1-2 hours travel from my office at the time: I cost closer to that $200k/year, but I also only usually visited the data centres 2-3 times a year, and other than that we used "remote hands" at the colo to do maintenance and be on-call 24/7. On ~60+ servers, we had maybe on average one minor incident every couple of months that required physical intervention.

Between two data centres, lets assume I did 6 visits a year, and that we had one ~30/min incident a month at $50/incident (it was less, but I don't remember the exact details, and it doesn't matter for this exercise). Let's assume I lost a whole day every visit (I didn't, though it got close at times), and "charge" $1000/day for my visits. That adds up to $6k/year for my time, and $600/year for remote hands, or ~$110/year per server. For comparison the colocation cost us ~$17k/year, or ~$283/year per server. These costs were pretty stable by number of servers, and so favored using fewer, more powerful servers than we might have otherwise.

So that added the cost of renting space at a manned colo facility instead of having the servers in the office (we did have a rack of servers that didn't need 24/7 attention at our office as well).

The rest of my time was spent on devops work that in my experience tends to be more expensive (on the basis of having contracted to do this kind of work on AWS too, and know the difference in billable hours I'd typically get per instance on AWS vs. per physical server on colocated setups) on cloud setups because complexity tends to be higher.

Scoundreller2• 7 years ago

On the other hand, they didn't calculate in any residual value after 3 years.

ams6110• 7 years ago

Because it's negligible

blihp• 7 years ago

As long as it's run within its thermal limits (i.e. not being severely overclocked and has adequate cooling/ventilation) 3 years or even 5 years isn't an unrealistic lifespan. The analysis makes this, and a lot of other, implicit assumptions which in general seem reasonable.

Symbiote• 7 years ago

I have 30 machines running at 100% load for an hour, then 25% load for an hour, the pattern repeats. Every quarter they run 100% for about a week.

Every 6 months or so a hard drive fails (out of 16 per server), no other components have failed. 10 machines are 6 years old (test system as it's out of warranty), 10 3 years old, 10 new.

There's also 20 or so other servers under different loads, I've not had anything fail other than hard drives.

When I started the job, there were spare power supplies for some 10+ year old servers in storage, so I'm either lucky or reliability is improving.

KaiserPro2• 7 years ago

No, not unless its badly designed.

I used to work for a VFX company where we would try and get as close to 100% utilization out of the farm.

We had machines that were 4 years old, still merrily plodding away. (not too many mind, they ate more power than its worth.) The things that tended to fail were Harddrives and fans.

iheartpotatoes2• 7 years ago

I thought ads on HN were discouraged?

jim_bailie2• 7 years ago

Maybe not! I wonder how much if anything was paid for this placement. And I also wonder if I should be preparing an "article" about my company. We could sure use some extra exposure.

Anyway, despite my belly-aching, it was an interesting read.

warent2• 7 years ago

YC has nothing to gain by selling ads on this forum. The business is already extremely wealthy and successful. Any ad revenue from this would be absolutely peanuts. It just doesn't add up.

ringaroll• 7 years ago

Nice.

raincom• 7 years ago

This makes sense only if your prospective clients want "lift and shift" into the cloud. But lots of people are using AWS for their services like S3, RDS, Cloud Front, Route 53, etc.

ec1096853• 7 years ago

You can still use s3, even if you aren’t totally in aws.

sudhirj2• 7 years ago

Now if they'd only throw in S3 for pseudo-infinite data storage, reliable SQS for work queue management, 10/25/100 Gigabit networking between the instances, redundant power supplies and cooling and racks in carefully selected stable locations for free, I'd buy a dozen!

sudhirj2• 7 years ago

I never get the point of these things. This is like saying a Raspberry Pi hooked up to your home router is cheaper, why bother buying Gmail for your company.

Or like saying that if you spend 20 hours every day in an Uber buying a car would be cheaper over 3 year term.

Nobody is disputing that buying similarly specced hardware on the market is likely to have a cheaper TCO, the point of cloud providers is that also have a bunch of stuff attached to the servers, which turns to be pretty important.

bubblethink2• 7 years ago

The point is that GPU instances are different. The usecases like ML training or general HPC are sufficiently isolated from the rest of your stuff that you don't really need all the bells and whistles that aws has. If you want to host a niche gpu database like production system, you may get more out of aws's ecosystem, but even then, the cost difference is huge.

dwild• 7 years ago

This is like saying a Raspberry Pi hooked up to your home router is cheaper, why bother buying Gmail for your company.

Where do I get GMAIL for my Raspberry Pi? If you order that Lambda server, you get the same good Linux interface you'll get over AWS. In fact they actually seems to offer some kind of software package that you wouldn't get over AWS.

Sure AWS has value-added, but in this case, it's mostly being able to not being billed for unused hours (what we call scalability), which for many is certainly worthwhile, but that's all. Is there anything I'm missing that AWS offer?

if you spend 20 hours every day in an Uber buying a car would be cheaper over 3 year term.

Isn't that the case? I failed to see what's wrong in that sentence. I don't own a car, I subscribe to a service similar to Car2Go, I'm at about 150$ a month right now. Sure there's value added there, but owning one also has value added, just different ones and at one point it will be more worthwhile to own one.

paulddraper• 7 years ago

An availability zone or two wouldn't hurt either.

vidarh12• 7 years ago

The price includes $15k/year colocation costs. That easily covers the redundant power, cooling, rack space, massive amount of bandwidth and redundant network network connections. Last place I handled colocated servers, we spent ~$17k combined for racks in two different data centres hosting a combined ~60 servers, so $15k/year for a single server is a very conservative estimate.

If you're buying more than one, that'll easily cover enough to cable up whatever network you want between them.

Many colo providers also have their own cloud services, and nothing stops you from using S3, SQS and the like, though of course if you use AWS services "from the outside" you do need to factor in their extortionate outbound bandwidth charges.

mippie_moe3• 7 years ago

Lambda engineer here (I'm one of the authors).

As bubblethink already mentioned, the article is arguing that batch processing jobs (e.g. those characteristic of Deep Learning) don't benefit substantially from cloud infrastructure.

High availability, edge processing, fast network upload, and the lego blocks for creating redundancy are moot.

Most training algorithms automatically recover if a node goes down. That covers all required failure handling.

KaiserPro2• 7 years ago

Using both S3 and SQS outside of AWS is perfectly possible, and the Colo provides decent bandwidth, cooling and power, that's literally the point of them

Thorrez• 7 years ago

Save $69k sounds a lot more significant than save 38%.

If you're going to be using the server less than 73% of the time, AWS sounds better.

BGZq7• 7 years ago

Keep in mind, the AWS cost is already for a reserved instance. On-demand would cost more.

mbell2• 7 years ago

I'd be curious what the TCO is when factoring in storage. i.e. What is replacing S3 for data storage in the colo setup?

secabeen2• 7 years ago

That system has slots for 16 2.5" drives in the back. I'd guess they can buy whatever commodity drive/SSD they want, and store the data there. Even throwing some cycles and memory at ZFS, the cost is small compared to the rest of the box.

You'll need additional off-site backup, but that's starting to get out of the scope of the article.

mbell2• 7 years ago

I don't think tossing 16 SSDs into a single enclosure is a fair comparison with S3. I also think that storage is absolutely in scope for an article like this.

ec1096853• 7 years ago

You can still use s3, even if you aren’t in aws.

mbesto4• 7 years ago

Our TCO includes energy, hiring a part-time system administrator, and co-location costs.

Which is a myopic view of TCO. This ignores so many things about purchasing on-prem hardware (for good or bad):

Admin:

- Cost associated with finding an admin who understands how this thing works

Speed / convenience:

- Try before you buy

- Time it takes for the box to be built, shipped and sent to the data center

- Time (and cost) it takes to install software, drivers, etc

Maintenance / Capitalization / Finances:

- 3-year? What is the useful life of this? When does it seemingly become obsolete?

- AWS will continually upgrade their hardware and you keep paying the same

- Hardware can be capitalized, which means you can push it to the balance sheet (for tax or valuation purposes)

- Spending $90k instead of $184k in year 1 with the option to turn it off if you want (no longer need). This could be very valuable for a startup who wants elastic spending patterns.

Hidden costs:

- Returns, breakage, warranty in case of a hardware failure

I understand why there is a market for this product, but it's not always an apples/orange comparison. Generally speaking, if you know what your workload is going to be, (I'd be hard pressed if a lot of orgs really know the answer to this) then on-prem hardware is not a terrible choice, but it has to be analyzed appropriately.

microcolonel• 7 years ago

- Time (and cost) it takes to install software, drivers, etc

I'd imagine this is part of having a system administrator.

vidarh12• 7 years ago

- Cost associated with finding an admin who understands how this thing works

Managing AWS resources is not "free" either, or the market for people to handle devops work on AWS related to ongoing operations would be non-existent. My experience is when I've done consulting, clients on AWS used to end up paying more per instance than on-prem clients ended up paying per physical server.

- Time (and cost) it takes to install software, drivers, etc

This is largely an initial cost of a day or two of setup when you start setting up an on-prem setup + for the first server of a totally new model. If you're buying one, then sure, you need to factor in a bit of time. If you're buying more, then if you use a competent admin the second one should be a matter of inserting boot media or (preferrably) configuring PXE booting, and picking an IP.

fucking_tragedy• 7 years ago

My experience is when I've done consulting, clients on AWS used to end up paying more per instance than on-prem clients ended up paying per physical server.

This has been my experience, as well. Clients who switch to AWS for cost-saving reasons will be disappointed.

mbesto4• 7 years ago

Managing AWS resources is not "free" either, or the market for people to handle devops work on AWS related to ongoing operations would be non-existent.

The costs of finding an AWS admin is much lower. This is, generally speaking, why many orgs still use SAP for ERP since there are a plethora of blue-chip consultancies that support those systems.

sabalaba2• 7 years ago

(Author here)

I agree with a lot of your points, especially the fact that TCO is practically impossible to calculate, is highly subjective, and should ideally include opportunity costs which, almost by definition vary from person to person.

I've definitely thought through some of your points. First I want to say that if your load is spiky you have no business buying hardware. No matter how you slice it, you're simply not going to be able to get hundreds of GPUs of throughput with an 8 GPU machine so it's not possible to get the same "product". So for those doing short bursts of hyperparameter search, stick with cloud.

But, like you said, for those with high utilization and known utilization patterns, it makes a lot of sense to go on-prem. Let's just say pretty much anybody who is buying a reserved instance from AWS should consider buying hardware instead.

- Cost associated with finding an admin who understands how this thing works

Pricing based on what co location facilities will provide you without much search and is pretty generous. $10k per year per server is silly high and should cover that search cost.

- Try before you buy

You can try essentially this exact machine on AWS. That instance and hardware are almost exactly the same. Now of course this doesn't apply to many instance types.

- Time it takes for the box to be built, shipped and sent to the data center

5 business days is our mean lead time for that unit. Most workstations are 2 days.

- Time (and cost) it takes to install software, drivers, etc

It comes with that stuff pre-installed as mentioned in the article. See Lambda Stack for driver / framework / CUDA woes: https://lambdalabs.com/lambda-stack-deep-learning-software

- 3-year? What is the useful life of this? When does it seemingly become obsolete?

I'll admit it's a long time but 3 years is about right for GPUs from my experience. I personally purchased new GPUs for my workstations in 2012 (Fermi), 2015 (Maxwell), 2017 (Pascal).

- AWS will continually upgrade their hardware and you keep paying the same

Yes, but only if you never commit to a reserved instance, in which case costs are 3x higher. If you buy a reserved instance you don't get upgraded hardware.

- Spending $90k instead of $184k in year 1 with the option to turn it off if you want (no longer need). This could be very valuable for a startup who wants elastic spending patterns.

Yea, most of these are people who are already using GPUs really often for internal training infra and are finding it expensive.

- Returns, breakage, warranty in case of a hardware failure

Our hardware comes with a 3 year parts warranty. Of course, this doesn't pay for your time lost but it's pretty rare to see parts fail and the $10k / year more than covers in-colo swap outs.

I agree with all of your points. I don't think that we really disagree with much here. TCO is hard to calculate :).

choppaface• 7 years ago

K80s are 2014 and people still use them at scale. So 3 years isn't unreasonable, right?

A major drawback of having this sort of hardware on-prem (and this unit in particular) is that it doesn't include local storage for a 100TB+ scale dataset. There's not just the admin cost of hard drives and RAID, but there's the infra cost of synchronizing data to the machine. An average, reasonable, product-oriented Engineering Manager could easily choose cloud over a $100k savings if it means less headaches. Especially since GCloud will give big discounts if they're trying to outbid Amazon. On the other hand, if Lambda offered some sort of NAS solution plus caching software (Alluxio! :), that would make the offering a lot more competitive. Oh and maybe throw in a network engineer to set up peering and such.. ;)

What might make the comparison more compelling is to take anonymized workloads from your customers, examine how often the workload results in idle (wasted) time, and to factor that into the figures. If a user's workload is elastic (many are, especially in R&D), then the user would be drawn towards ML Engine, Sagemaker, Floydhub, Snark, etc... The downside to these services is that they almost always involve expensive copying of data from NAS to worker machines. But if user utilization is high, and the dataset is static, the on-prem machine is a prime investment.

While the savings is a big and noteworthy number, iteration speed is of principle concern to readers here and likely a good segment of your customers. I love the Lambda offerings, but there are a lot of deep learning hackers who can't make the most of bare metal.

The 1080TI server is probably the killer solution at 1/10th the cost. But NVidia legal blah blah blah :(

evilmoo• 7 years ago

5 business days is our mean lead time for that unit. Most workstations are 2 days.

So this is an advert? Gotcha.

I think if you called out at the top of the article then it would have been a little more honest.

mbesto4• 7 years ago

Thanks for the response! Yes you're right, it is difficult to calculate. I hope people reading my comment took it into the context of a larger view on the process to calculate TCO, as opposed to just directly scrutinizing your products/article.

mippie_moe3• 7 years ago

Hi, Lambda engineer here (I'm one of the authors). You bring up some good points, I'd like to address some of them:

Admin:

Time (and cost) it takes to install software, drivers, etc

These tasks aren't made unnecessary by cloud. Yes, with cloud, once you've done the work, it can be forever encoded into an image or container. However, the same applies to on-prem using container solutions like Docker.

Maintenance / Capitalization / Finances:

3-year? What is the useful life of this? When does it seemingly become obsolete?

With GPUs of recent history, obsolescence is not a concern in a 3-year time frame. On AWS, people are still using K80s (released in 2014!). The GTX 1080 Ti, which was released 2.5 years ago, is selling for substantially above MSRP. This may change if competition in the GPU space increases and NVIDIA loses its monopoly.

AWS will continually upgrade their hardware and you keep paying the same

True, but this concern is mitigated by slow rate of GPU obsolescence.

Hardware can be capitalized, which means you can push it to the balance sheet (for tax or valuation purposes)

Can you say a little more about this? Not sure what you're getting at.

Spending $90k instead of $184k in year 1 with the option to turn it off if you want (no longer need). This could be very valuable for a startup who wants elastic spending patterns.

This needs to be evaluated on a case-by-case basis. A poorly-capitalized start-up with an unpredictable compute workload is not a good candidate for buying on-prem. A well-capitalized start-up that consistently uses GPUs is a better candidate.

Returns, breakage, warranty in case of a hardware failure

Any reasonable hardware provider will include an option for 3-year warranty.

colde• 7 years ago

Any reasonable hardware provider will include an option for 3-year warranty.

Did you include the cost for that in your scenario?

alvar0• 7 years ago

Hardware can be capitalized, which means you can push it to the balance sheet (for tax or valuation purposes) Can you say a little more about this? Not sure what you're getting at.

I think he means this: that hardware goes to your balance sheet, then it depreciates by a certain amount over those 3 years and that loss may be tax-deductible.

sokoloff• 7 years ago

Spending $90k instead of $184k in year 1 with the option to turn it off if you want (no longer need). This could be very valuable for a startup who wants elastic spending patterns.

I think they used the 3-year reserved instance pricing, meaning you can’t just turn it off after a year and stop the cash flow out (at least not at that price).

termie• 7 years ago

It is possible to sell (or buy) reserved instances in a secondary market: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ri-marke...

canadev• 7 years ago

Interesting note about Lambda Labs, all of the press links on https://lambdalabs.com/?ref=blog are about a ~"privacy violating Google Glass app" that recognizes faces and geotags photos of them.

I don't see why they choose to promote that link now.

zten• 7 years ago

Who's running model training 24/7 to justify reserving this instance or co-locating your own hardware? (Apologies in advance for not being very imaginative)

Their ImageNet timing fits within the bounds of a Spot Duration workload, so in the most optimistic scenario, you can subtract 70% from the price - assuming spot availability for this instance type. (Of course, there are many more model training exercises that don't even remotely fit inside 6 hours.)

bubblethink2• 7 years ago

p3dn.24xlarge's pricing makes no sense at all. It feels like aws did it to pull off some PR/marketing stunt without any real users in mind. I've tried getting spot instances for it, but aws just errors out. So they don't even have enough of them to allow spot instances. And it's a gpu machine. So the usual arguments of scaling up on demand or adapting to load don't really apply. You either have this usecase or you don't. And if you do, just buy the hardware.

hughesjo2• 7 years ago

You are not the target audience since you don't have the use case nor the budget

rb808• 7 years ago

I bought a second hand Xeon E3-1246 v3 (8 VCPU), 16GB memory for $250 on ebay. That's less than it costs to rent an a1.xlarge for 6 months. Hardware is so cheap now, esp with SSDs and memory getting cheaper. Don't automatically rent!

vidarh12• 7 years ago

Or if you're going to rent, consider dedicated hosting providers too. Providers like Hetzner often work out far cheaper (especially if you're doing anything requiring a lot of outbound bandwidth).

AWS is great for convenience when you can afford it, but it is a really expensive solution, even when factoring in the extra things you have to deal with to rent, lease or buy dedicated servers.

riku_iki3• 7 years ago

Sadly no major dedicated provider offers machines with 8 GPUs..

elchief2• 7 years ago

In my mind, the reason to use something like AWS is to a) get your servers in minutes instead of weeks and b) easily right-size your service

Once your service is somewhat stable in terms of size and you can afford longer lead times, then you should return to on-prem to save money

joefourier• 7 years ago

What's this false dichotomy between AWS and on-prem? Dedicated servers at Hetzner, OVH, Datapacket, etc. are much cheaper than AWS and can also be ready in minutes.

elchief2• 7 years ago

I should have said on-prem or dedicated (or whatever the opposite of expensive AWS is)

chx3• 7 years ago

This false dichotomy between colo and AWS is just making me exhausted 'cos I have been repeating this for so long: just rent a dedicated server. There surely are some cases where colo is the best choice but as the years, now a decade pass since I have tried to spread this, it makes less and less sense every year -- and it never did much in the first place. Maybe if you have several racks worth of equipment? I am not familiar with that size.

mippie_moe3• 7 years ago

Lambda Labs engineer here. Here's what we're trying to argue:

Many benefits of running infrastructure in the cloud are lost for offline batch processing jobs. Training machine learning models doesn't require low response times, high availability, geographic proximity to clients, etc. Yet, with cloud, you're paying for all this extra infrastructure.

The main benefits of cloud for tasks with machine learning type workloads are cost saving (if utilization is low) and no electrical set-up.

On the other hand, cloud is extremely expensive for groups that require high base levels of GPU compute. The article is arguing that such groups can save a huge amount of money by moving infrastructure on-prem.

chx3• 7 years ago

I feel I am talking to a brick wall. This often happens and this is what exhausts me. You are still arguing between cloud and colo / on-prem (even ignoring that on-prem and colo is very different but w/e) and what I am saying is that there is a third option neither the article nor your reply even acknowledges.

icelancer3• 7 years ago

I have been hard-pressed to find Threadripper 2990wx dedicated servers with 1080ti/2080ti GPUs in them for rental, much less at a reasonable cost. Colo/on-prem was our only option for our mega transcoding server.

chx3• 7 years ago

That's not a server CPU. You'd need to look for a dual EPYC 7371 server for similar performance -- 2990WX is 3-4.2GHZ 32 cores, 7371 is 3.1-3.8GHz 16 cores.

I'd email webnx sales, they have some EPYC servers although not this very new CPU but they were always very flexible. It might be more economical if you front the price of the CPUs but the 7371 is very cheap for what it is, the two should be around $3k (I saw them as a special order on gamepc for 1600, the site name is very ironic because the monsters they sell are more workstation / server than gaming pc). I am not affiliated, just a very satisfied customer, I worked with them when I was the lead architect of a US Top 100 (per Quantcast) site.

The 1080 Ti is certainly not a problem for them.

ti_ranger• 7 years ago

1)How long does it take to get a Lambda Hyperplane operational, from the point I place an order?

A p3dn.24xlarge is a few minutes.

My experience in deploying new hardware-based solutions is that it typically takes between 6 and 12 weeks.

2)How does the TCO compare when I only need to train for 2 hours a day?

3)If I was previously training on AWS p2.16xl, I could upgrade to p3.16xl with basically 0 incremental cost (for the same workload). Does LambdaLabs offer free (0 capex) upgrades? If so, how long would it take to upgrade?

ec1096853• 7 years ago

1) 5 days (according to the author) 2) Obviusly the cloud is better in that case 3) You can’t always upgrade on aws eigher if you have paid for a reserved instance: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ri-modif...

mschuster91• 7 years ago

My experience in deploying new hardware-based solutions is that it typically takes between 6 and 12 weeks.

Most of this is due to arcane ordering processes both on client and supplier side.

deepnotderp• 7 years ago

It may be useful to note that most deep learning workloads for training are pretty latency insensitive and are pretty flat throughout the day.

Scaevolus• 7 years ago

If you're not deploying in a datacenter, you can save even more money by building a workstation with a few 2080 Ti cards, which cost $1200 and give 90% of the speed of the $3000 Titan V: https://lambdalabs.com/blog/best-gpu-tensorflow-2080-ti-vs-v...

altmind• 7 years ago

Exactly my thoughts - instead of one datacenter class gpu, you can use multuiple desktop components for better performance and more performance per buck. I've seen some sweet supermicro 2U chassis that can fit and run cool four 2080Ti.

I remember, though, that NVIDIA disallows in EULA the use of nvidia driver blob in consumer gpu in datacenter environment. Time will show how legally enforcible is this restriction.

freeone30002• 7 years ago

We have one of these. Aside from the mechanical differences (it's actually a 2.5U server due to how the power cables connect to consumer cards), you're losing out on NVLink. You're going to be restricted to single-card training, which means either batching or other restrictions to ensure it fits in the memory of a single card. This setup is able to train four one-card models in parallel; it is not able to train a four-card model.

m0zg• 7 years ago

If it's really on-prem (i.e. not in the "datacenter" as per NVIDIA EULA), you could spend a lot less than $100K+ for a lot more throughput by purchasing consumer-grade cards and HEDT gaming hardware.

Sure you'll have 4 GPUs per box and not 8, and sure, each GPU will have 11GB and not 32, but the whole machine (_with_ the GPUs) will cost just a tad more than a single V100. So if you don't really need 32GB of VRAM per GPU (and you most likely don't), it'd be insane to pay literally 5x as much as you have to.

manigandham• 7 years ago

Cloud computing was never about price, it was about the ability to provision and operate infrastructure instantly through an API.

If you can take advantage of that flexibility to build reactive capacity then you can save money, but that wasn't the initial driving point.

latch• 7 years ago

you MIGHT save money.

The price difference can be substantial enough that you might simply be better off having it "over provisioned" off hours.

Also, not having "reactive capacity" might simply mean that your 95th percentile goes from 100ms response time to 300ms. Which again, might be a more cost effective approach.

I think the initial development and ongoing cost of maintaining that reactive capacity is also substantial enough to be considered.

Most of the apps that truly need the elasticity should probably going hybrid anyways. Baseline on dedicated hardware, spikes on spot instances.

icelancer3• 7 years ago

If you can take advantage of that flexibility to build reactive capacity then you can save money

Almost everyone thinks they can; very few small businesses have the sysadmin/devops to pull it off.

boulos• 7 years ago

Disclosure: I work on Google Cloud.

First, thanks for writing this up. Too many people just take a “buy the box, divide by number of hours in 3 years approach”. Your comparison to a 3-year RI at AWS versus the hardware is thus more fair than most. You’re still missing a lot of the opportunity cost (both capital and human), scaling (each of these is probably 3 kW, and most electrical systems couldn’t handle say 20 of those), and so on.

That said, I don’t agree that 3 years is a reasonable depreciation period for GPUs for deep learning (the focus of this analysis). If you had purchased a box full of P100s before the V100 came out, you’d have regretted it. Not just in straight price/performance, but also operator time: a 2x speedup on training also yields faster time-to-market and/or more productive deep learning engineers (expensive!).

People still use K80s and P100s for their relative price/performance on FP64 and FP32 generic math (V100s come at a high premium for ML and NVIDIA knows it), but for most deep learning you’d be making a big mistake. Even FP32 things with either more memory per part or higher memory bandwidth mean that you’d rather not have a 36-month replacement plan.

If you really do want to do that, I’d recommend you buy them the day they come out (AWS launched V100s in October 2017, so we’re already 16 months in) to minimize the refresh regret.

tl;dr: unless the V100 is the perfect sweet spot in ML land for the next three years or so, a 3-year RI or a physical box will decline in utility.

freediver• 7 years ago

The actual cost of running a cloud instance is inflated. The cheapest way to run them is using spot/interruptible instances which for most deep learning jobs will suffice. If anything there will be some upfront cost to set it up in a way that it automatically manages interruptions, storage etc. Also by not limiting yourself to AWS you can have many other options.

With this setup you can get 2x4x V100 on Azure for a total of $42k/year (assuming running 24/7).

Even if one spent $40k to write code for spot instance management this is by far the cheapest solution for GPU compute both short term and long term.

source for calculation: https://cloudoptimizer.io

V100 Server On-Prem vs. AWS P3 Instance Cost Comparison

Related Stories