GPUs to demo inference at scale: calling for ideas/use-cases

https://salad.com/resources/gpu-benchmark-stable-diffusion

by bobjmiles • 3 years ago

5 1 3 years ago

bobjmiles• 3 years ago

HN Hey!

At salad.com we're launching a GPU Cloud that runs atop consumer hardware. Great for inference/tuning, not great for training (because of networking, vram restrictions, etc).

We've built a fully managed container service so you can easily deploy replicas at massive scale and we just ran a Stable Diffusion demo: churned out 9 million images in 24hrs across 750 GPUs (100+ images/second and 5000 images/$). Architecture and specifics here salad.com/resources/gpu-benchmark-stable-diffusion

We're thinking our next run will be to customise the SD models (various checkpoints & LoRAs) to demo something closer to real-world.

We'll also run whisper-large at massive scale to demo a million hours of transcription, and get a benchmark price on this.

Perhaps webscraping/common-crawl at scale with GPU processing at the edge: building datasets for LLM training?

Question for the HN community, what would you run?

We'd love your suggestions, and perhaps hand over the keys to see what you can do with 1000's of GPUs!

Bob - Salad Founder

GPUs to demo inference at scale: calling for ideas/use-cases

Related Stories