Nvidia H100 and A100 GPUs – comparing available capacity at GPU cloud providers 3 years ago
Consumer GPUs are great for inference at scale.
We're a distributed GPU cloud with 10k+ consumer GPUs on our network. We can crank out 4500+ stable diffusion images per dollar (highest in the market), all on consumer 3060s.
The high-end GPUs definitely have a role but for a bulk of production/inference serving can be done on consumer GPUs at 80-90% of the cost.