jonathanlei

It's as difficult as a serverless provider to grow as it was for CPUs before GPUs came along.

Many companies overinvest in fully-owned hardware, rather than renting from clouds. Owning hardware means you underwrite unrented inventory costs and prevents you from scaling. H100 pricing is now lower than any self-hosted option, even without factoring the TCO & headcount.

(Disclaimer: I work at a GPU cloud Voltage Park -- with 24k H100s as low as $2.25/hr [0] -- but Fly.io is not the only one I've noticed purchase hardware when renting might have saved some $$$)

[0] https://dashboard.voltagepark.com/

Absolutely - GPUs are definitely not a very liquid asset. As someone who works at a GPU neocloud provider (Voltage Park), server assets at scale definitely face a huge slippage, you can buy for $1 and get quotes for $1.50 but only be able to sell for $0.60

Hello! If you're interested in monetizing those GPUs, I'd be happy to rent them (all 400!) and offer those to customers of the cloud I work at :)

jonathan [at] tensordock.com

Congrats on the launch! This is huge, and it's really cool to see a cloud provider moving this direction - auction pricing for customers so that you always know you're getting the best deal on the market, while providing 100% utilization for you :)

I'm curious what some of the numbers mean: e.g. what does 688/1464 GPUs available indicate in the left gray box? What about there being 1040 GPUs in light gray, and 8 in dark gray?

Hmm I did include a training workload as the second chart. My test workload was relatively small so I guess if the workload I ran spends a bit less GPU time comparatively to the CPU, given equal CPU for all workloads, would be an equalizing factor.

But even looking at the Lambda Labs benchmarks, I am surprised that the H100 PCIE barely outperforms the A100 SXM, for example. And it is meant to be a replacement for the A100 PCIE. 20% generational improvement yes, but I would have expected more?

Oops, just noticed the link isn’t clickable, here you go! https://tensordock.com/benchmarks

Utilization is not going to be 100% --- ever. Discounts are often given, and servers break down, etc.

Thanks! :)

We do have a managed container hosting service, but it's built on our own backend that auto-scales nodes for you when average GPU utilization surpasses a certain point, but it's not K8s -- which would have been a pain to configure given the distributed nature of all of our servers.

https://dashboard.tensordock.com/deploy_container

Hi! Jonathan from TensorDock here :)

We have our own supply base (sourced through https://tensordock.com/host), operate some of our own servers, and are not related to any other marketplace :)

We think we have better security & reliability than Vast.ai due to virtualization rather than Dockerization, as well as more strict access controls [1]. Additionally, you can run Windows VMs on us if you want :)

[1]https://tensordock.com/security

Jonathan from TensorDock (https://tensordock.com/) here - we listed two of our A100 and H100 clusters on the site.

The IB equipped on our clusters (can't speak to others) is 8x 400 Gbps. Most customers training foundational models are able to fully utilize that fabric in parallel.

Non-paywalled snapshot: https://archive.ph/r6mkJ

Why?

A less anti-bill view: https://www.cnbc.com/2022/08/30/california-passes-bill-aimed...

I think if you offer to sell to them, they will probably give you a lowball offer.

Instead, maybe consider trying to grow it on your own, see where you can take it, and if it shows some reasonable traction, a big player would probably definitely offer to swallow you up at a good offer so that they get the better technology that's already been demonstrated to work in a production environment.

Just my $0.02 :)

Disco Diffusion (and generated art in general) is really cool: https://replicate.com/nightmareai/disco-diffusion/examples

Happy to give you whatever credits you need (would $25 be enough?) to try running some cool stuff on our infrastructure. And feel free to email me the results of whatever you do at jonathan@tensordock.com :)

Yeah, feel free to shoot me an email for like $25 of credit. For startups, we usually do $100 (just send a request from your startup email address).

Even with $5 credits, we had people abusing the system and creating multiple accounts to cryptocurrency mine. So we chose $1 as an amount that would be so low that nobody would want to take advantage of, haha :)

Whoops, apologies for missing this! For our core cloud product, we only partner with established providers. Large-scale compute wholesalers with $5m+ of compute each in secure data centers. These companies' entire businesses are built on selling secure compute to customers like us and other medium/large businesses. Basically, this isn't some random dedicated server host off of LowEndTalk :)

We have data protection agreements with all of them, and we can also do bare metal machines on request so that you have full control over your physical machine.

We will consider! We are working with two potential suppliers in the area... expect more news by the end of the year if those deals go through! :)

Absolutely! You can use our Windows 10 instances and cloud game (like how airGPU sells cloud gaming containers on us), render blender scenes, or do CAD on remote desktops. These are dedicated virtual machines for you to use :)

@Ryan - I think OP was asking about the beehives that we sell on the website, haha :)

It was started by Mark... he raises bees in his backyard, so we have some kick-ass honey raised right from our New England backyards :)

1 pound - https://buy.stripe.com/aEU28j7CR6tV81i7su 3 pounds - https://buy.stripe.com/8wM6oz3mB7xZbduaEH

Will definitely update! Sorry about the aggressive-sounding questions. Fraud was a very big issue in our early days. People would ask for refunds after crypto mining and then dispute the charges, etc... but we added other protections, so I think we can get rid of those :)

Thanks for the comment! Our core GPU cloud product is all data center-based. People can modify VMs after created, giving greater flexibility. Storage is 3x-replicated, 10 gbps networking, etc. We think of this more as an AWS replacement than a Vast.ai competitor.

We are actually working on something very similar to vast.ai (https://www.tensordock.com/product-marketplace) set to launch into a soft beta mode within the next two weeks and probably a real "Show HN" by the end of August. We'll have a few dozen GPUs scheduled to come online during the launch week at prices similar to Vast.ai. This would be with with full virtualization, which we think is better than Docker containers because we customers can run Windows VMs and do cloud gaming/rendering, thereby generating hosts more income. We might also add VM disk encryption later on, which would be more secure. Still, they are very large, so it'd be large road uphill, but we're working on something similar.

Also, if I remember correctly, with Vast (as a former user myself), an issue can arise when you have a VM in the stored state but someone is claiming the GPUs running an on-demand workload, which prevents you from being able to pull your data out. Because VMs are all booting from network storage and can be rescheduled to other compute nodes, you won't face that issue on our core cloud product here :)

Not yet, but it's something we can definitely accommodate. If you just send us a link to a .qcow2 or .img file that's a cloud image configured with DHCP, we'll be able to set you up in no time :)

- Jonathan @ TensorDock

Just a side note here, it seems that these are spot instances, whereas all our VMs are non-interruptible. So there is a bit of difference (e.g. you probably wouldn't do a weeklong Blender render on a GCP VM if it could be interrupted and lose your work, whereas you can definitely run it on TensorDock because our VMs are reserved).

Of course, you can set up data checkpointing to save your data, but overall, it is a bit of an extra hassle to run on spot/interruptible instances, and if you do get interrupted, you are wasting valuable time waiting for stock to free up again.

Oh yes! Our standard Ubuntu 18/20 OS images have nvidia-driver-515 + Docker + NVIDIA-Docker2 preinstalled. The all three ML images also come with CUDA if you need that. We'll probably add cloudinit support sometime so that you'll be able to script background workloads that autoshutdown the VM once completed, but at the moment, you'd have to manually SSH into your VM and run it...

Thanks, Jonathan @ TensorDock

Thanks for the comment! Yes, Lambda Labs recently lowered pricing, so they're now roughly equivalent to us, and they beat us on some configurations.

I haven't used them, so please fact check me, but it seems like the machines come with directly attached storage. So, if you're using an 8x V100 and want to switch to a 1x RTX 6000, you'd have to spin up a new server and manually migrate your data over.

We built our platform with networked storage. You can spin up a CPU-only instance for $0.027/hour (<$20/month), upload your data, convert it into a GPU instance to train your models, and then convert it back. We frequently see users converting servers from 8x A100s (to train workloads) back to 1x RTX 4000s (to run inference). This kind of flexibility saves people time, which equates to money given how expensive ML developers are now.

(Our networked storage model also enables people to shut off their VMs and save money)

I'm sure Lambda Labs is working on something similar, but it seems they are doing dedicated servers based on how they advertise.

I think we also have a higher variety of GPUs (10 SKUs with us vs 4 SKUs). This lets people switch from between, say, an NVIDIA A6000 to A5000 to A4000 to truly "right-size" their compute so they don't pay for anything they don't need.

Cost-wise, we also have better long-term pricing like GeForce 1070s for $100/month in Boston or $150/month in Singapore Equinix SG1 - which is really good pricing for an APAC city in my opinion (https://console.tensordock.com/order_subscription), and we're working on a marketplace to let compute suppliers list their compute on our platform to get closer to cheapest for those who really care about cost (https://www.tensordock.com/product-marketplace).

Cool, let me know if you need anything at jonathan@tensordock.com!

Unlike other managed Docker hosting services, these are fully virtualized VMs, so you can use whatever tools you want, and most software applications work out-of-the-box. You can cloud game (or run virtual workstations) through Windows, too.

Our ML images come with Docker and NVIDIA-Docker2 preinstalled, so that should work with whatever workloads you'll be running :)

Wow, cool! Yes - interruptible can be very cheap... I'll add it to our backlog so we do that instead of idly mining

I was wondering, do you happen to have an API for listing servers? We're launching a marketplace later in August (https://www.tensordock.com/product-marketplace), and we expect pricing to be really really cheap. Like #1 in industry cheap while retaining.

Interruptible, if we add that, would probably be even less than those prices listed.

It'd be really cool if we could auto-update availabilities of GPU servers through an API so that we can list our servers on your tool as well :)

Thanks for the comment! Quick answer: it's complicated.

Long answer: We own a substantial amount of compute ourselves, as far as Singapore where we have fully-owned hardware at Equinix SG1. We started with our own crypto mining operation just outside of Boston, but as wholesale consumers approached us in 2020 due to pandemic surges, we added business internet and power backups. Suddenly we were operating a rudimentary "office data center." Two large reseller sites sell on our fully-owned hardware. Boston's electric costs are very high ($0.23/kWh), so we're gradually moving more hardware to tier 3/4 data centers that are cheaper on a per-unit basis.

But, we partner with 3rd parties too (4 large scale 1000+ GPU operators each to be exact) to resell their compute. This is also how we'll enter Europe... we're working closely with an existing supplier that colocates servers at Hydro66 in Sweeden and another at Scaleway Paris. We provide the software, they provide the hardware, and we pay them a special rate based on the volume we're doing. Partnering with others is the only way we can handle large scale without insanely high capex costs (that being said, we do get preferential pricing as an NVIDIA Inception Program member, which we take advantage of for our own fully-owned hardware).

We have a doc in it here: https://docs.tensordock.com/infrastructure/reservable-instan...

We're also working on a marketplace (client site: https://www.tensordock.com/product-marketplace, host site: https://www.tensordock.com/host). We expect a beta version to be up and running in the next ~2 weeks. With this, we'll have a script that hosts will use to install a small version of OpenStack. Then, they set prices, and customers can deploy directly on that hardware. By aggregating all these hosts together on the same marketplace, we hope we can slash the price of compute.

So far, owning our own hardware has allowed us to negotiate better rates and enter markets where previous services don't exist (namely, Singapore, where we sell subscription servers with 1070 GeForce cards for $150/month — unheard of pricing for an APAC city). Eventually, we hope there'll be suppliers in every city selling on our marketplace or core cloud product so that we can really become the #1 place for ML startups to provision compute. In a way, we want to be the Amazon of cloud computing. Amazon, in a way, created a global marketplace. Yes, they sell their own products, but they also sell others' products. By doing so, you know that you're getting a good deal on whatever you buy. We want to end up being the same thing for compute, but that's still a few years off :)

TL;DR - we own a lot of hardware, and we resell a lot of hardware. But in the future, we want to focus on the reselling aspect to truly be able to nail the user experience and handle demand surges while maintaining low costs.

Prospective student from the area here —- here are my two cents, but keep in mind these are the opinion of someone who doesn’t know what he is talking about.

From an educational quality standpoint, they are comparable. I personally like the rural feeling of UMass Amherst a lot more than the city surroundings of Boston University.

Educational quality also stems from the quality of your classmates. In that regard, UMass Amherst is a bit worse than Boston University, which has better rankings and attracts students whose families are more affluent. Finding a job with a BU degree is going to be easier than with an Amherst degree from what I understand.

Personally, I’d choose UMass Amherst because it’s cheaper and some of my smart friends go there (due to affirmative action going to another level this past year blocking out Ivies for a bunch of Asians).