All GB/s without FLOPS – Nvidia CMP 170HX Review

https://niconiconi.neocities.org/tech-notes/nvidia-cmp-170hx-review/

by dannyw • 3 years ago

140 74 3 years ago

niconiconi.neocities.org

gymbeaux8• 3 years ago

I love finding hacky ways to save money on hardware, but unfortunately NVidia is just too good at the game and the 170HX was destined for the landfill at birth. God forbid a few of us enthusiasts get A100 performance for under a grand.

The next best thing is a 3090 or the like with a broken PCIe power connector or some other minor defect. My 3090 is simply missing the bit that holds the clip of the power connector in, however it’s a snug fit anyway and with the cables crammed into my case as they are, I don’t think it’s going anywhere. I paid $200 less than market for that 3090 as a result. Less than a gram of plastic. $200 off.

Meanwhile, as the article points out, AMD is nowhere near as hostile towards its customer base, and modified Radeon cards can apparently be had for $100 or so (from China). The caveat of course is no CUDA support, so it’s kind of moot.

renewiltord3• 3 years ago

AMD is hostile in a much more meaningful way - APIs are junk, drivers are buggy, hardware doesn't work. Useless for AI. You will get nothing done. Save your money. Better to get working hardware for more than non-working hardware for less. (Tokens/s)/dollar is 0 on AMD.

You will waste time.

consp• 3 years ago

Maybe also blame Nvidia for making a closed source API in your rant.

mananaysiempre2• 3 years ago

That’s not hostility, that’s just suckage. Suckage can be solved with time, if years of it on occasion. Hostility can only be solved by the customer base jumping ship.

Zardoz84• 3 years ago

drivers are buggy

so you don't use linux

JonChesterfield• 3 years ago

You on windows by any chance?

Havoc• 3 years ago

Tokens/s)/dollar is 0 on AMD.

You do realise llama.cpp works on some and cards right?

nullifidian3• 3 years ago

The situation has been changing rapidly since summer. According to my research, if you combine their desktop and cuda bugginess (for 7900 XTX) they are approaching the nvidia level of bugginess on linux, which is surprisingly quite high (much much higher than what it was say 5 years ago when nvidia just worked). One just needs a glimpse at their forum https://forums.developer.nvidia.com/c/gpu-graphics/145 to gauge the situation.

I just bought a 4090 and the desktop experience I get is much worse than what I had with the gpu embeded in the Ryzen 7950x: Wayland doesn't work, in Xorg there is tearing in mpv, alt-tab sometimes breaks in gnome. When I launch memory intensive cuda kernels the whole desktop becomes unresponsive. The drivers spews Xid errors in dmesg and breaks for certain applications, such as embergen.

OtomotO3• 3 years ago

My last nvidia card was bought in 2007.

Since then I am on AMD. I game, I build games, work on GPU related stuff.

I refuse to buy Nvidia until they open source their drivers.

I don't care about windows, I am on linux solely and there, from my experience, AMD is doing an excellent job

omneity3• 3 years ago

There is some CUDA support on AMD. I'm using it on a daily basis, it's much more production ready than you would expect. Do you use pytorch or something else?

gymbeaux8• 3 years ago

How do you have “some” CUDA support? I’m aware of the AMD HIP API that ports CUDA code to run on AMD GPUs, but that’s not CUDA at that point. Im also aware of the geohot (George Hotz) project for bringing native CUDA to AMD GPUs, but I think he abandoned it because AMD wasn’t throwing him any bones.

Cupy on Python is mostly what I use.

7speter5• 3 years ago

Could a card like this work for something like object recognition? Those tiny google tpus that are supposed to be 25 dollars are 100 dollars now, and I’m wondering if the prices get low enough, would these be a viable alternative (along with an undervolt)?

gymbeaux8• 3 years ago

Raspberry Pis and other “small things” like Project Coral are ridiculously overpriced. It’s ironic because whenever I see someone post on Reddit about their tiny rack of Raspberry Pis in their closet, people always ask in the comments what they’re using the cluster for, and the OP always says something along the lines of “just fucking around.”

If I’m on a budget, my first choice is probably going to be an older, used Nvidia GPU. Maybe a 1050 or one of the older Quadros. The Quadros will get you the most VRAM per dollar, but the GPU will be rather weak, in the realm of a 1050 perhaps, while the 1050 will have better/longer driver support and be physically smaller, quieter, and use less energy.

If those aren’t an option, I would take a closer look at ROCm and HIP. It seems like AMD is prioritizing support for newer GPUs like the Radeon VII, so a $50 RX480 is probably not a good investment, despite the low price.

benreesman• 3 years ago

That’s a lot to parse, so I’m kinda hand waving, but the memory bandwidth emphasis seems like a great fit for most LLMs at least if not also some ViTs and other attention-style architectures on both training and inference? Certainly sounds like the price is right.

Am I missing something key either conceptually or by failing to read all the stats closely?

buildbot3• 3 years ago

It’s tensor cores and floating point math have been either artificially or actually disabled - it would be very very slow. And 8gb of vram is really low as well.

ethanwillis• 3 years ago

"Unfortunately, all Nvidia GPUs since recent years have VBIOS digital signature checks, making VBIOS modification impossible"

This is not necessarily true. As seen with android devices you can force digital signature checking mechanisms by varying voltage levels in order to get the device to completely skip the checks as if they were never there.

https://research.nccgroup.com/2020/10/15/theres-a-hole-in-yo...

I'm sure a similar strategy could be developed here.

dmitrygr• 3 years ago

Glitching almost always requires removing capacitors. Good enough for dumping things out of a device once or twice. But GPUs that consume hundreds of watts will not be stable without those bypass capacitors.

So, sure, you can skip the verification checks, but your GPU won’t be stable enough to be useful for anything

Namidairo• 3 years ago

Well, I believe there are some modified versions of nvflash floating around that'll let you flash anything with a valid signature.

Of course, the only thing that'll POST are going to be just other vendor images from the same card model usually. (For different power limits, usually)

justinclift2• 3 years ago

Isn't the flash chip on nvidia boards a generic thing that someone could buy themself, flash using existing eeprom writing gear, then solder onto the board?

Also, as the chip and board here seems like an A100 reference design, using an A100 VBIOS image shouldn't fail any signature checks.

formerly_proven• 3 years ago

Yes, but the public key and product ID for the verification is in the GPU, not the external flash.

phire• 3 years ago

Probably, but the A100 BIOS probably can detect it is running on something other than a A100 and bail.

The lack of memory would be the most obvious difference. The A100 has 80GB, this has 8GB.

And I really suspect Nvidia probably some way of explicitly locking a chip to a given product ID, like efuses that the BIOS firmware can check on boot.

sweetjuly• 3 years ago

I'm sure a similar strategy could be developed here.

Slow down there. Glitching is almost never a practical long term strategy. It can take hours (or even days, depending on the target) to successfully bypass a check just once without other follow on effects. Glitching is useful if you need to bypass some mitigation once, such as to extract cryptographic keys, but it's not something you want to do every time you turn on your PC. Glitching gets substantially less reliable with every passing generation due to scaling (increased density/lower Vth increases the odds that you'll corrupt something else, particularly with EM fault injection) and design complexity (glitching out-of-order cores is a HUGE pain).

jamesblonde2• 3 years ago

The best value GPU right now for SoTA LLMs are probably the Nvidia RTX 6000. You can connect them together with NVLink, they have 48GB, and you can fit 4 of them in a PCI 4.0 high end consumer motherboard. Enough to fine tune even LLama-2, albeit with a batch size of 1 :)

CamperBob2• 3 years ago

Unfortunately, you have to specify which generation of "RTX 6000" you mean, since nVidia likes to reuse product names for no good reason besides confusing customers. The one you're talking about is around 8 grand USD. The one that comes up first when you Google "RTX 6000" is $2400 and change, and not particularly useful.

jamesblonde2• 3 years ago

Thanks for the clarification. I bought them for $5k (without VAT) in Sweden last month.

brucethemoose22• 3 years ago

Enough to fine tune even LLama-2, albeit with a batch size of 1 :)

At FP16? I think you need much less (~2 48GB cards) to finetune 70B with increasing levels of optimization.

I think you can even do it on a single card with QLORA.

buildbot3• 3 years ago

I have not been able to get it working with 2 4090s yet, but I think that's because the library I choose to use (axolotl) does not support model parallel at all.

buildbot3• 3 years ago

Only The RTX A6000 (ampere) (Not RTX 6000! those are 24GB Turings!) has NVlink, and as far as Nevada states, they only work in 2x NVLink. Memory pooling is not really a real thing as much as they say it is.

They run about 4K on eBay, and have 303TFlops of tensor core perf in sparse Bfloat 16. So 150 Dense. They do have 48GB of memory which is great, but at 768GB/s. Source: https://www.nvidia.com/content/dam/en-zz/Solutions/design-vi...

The 4090s run 1599 for a founders edition or a Gigabyte Windforce V2 (my choice). They have at least 165 Tflops of Bfloat16, 330 TFlops of FP16 with FP16 accumulate, and 660 TFlops if you use sparsity. They also support FP8 at 660 TFlops Dense, 1321 TFlops Sparse(!!).

Unless you need the 2x 25% slower memory, the 4090s are much better choices. You get the same scaling over 2 cards anyway.

There is also the RTX A6000 Ada, which is 8K, and based on the same Ada chip as the 4090, except with 48GB of memory. Lower power and clock speeds result in slightly lower peak TFlops numbers. You really really pay for memory.

mdekkers• 3 years ago

As if NVIDIA is going to let the non-wealthy plebs anywhere near DYI AI

blackoil• 3 years ago

170HX was created to solve two problems NVidia had,

    1. A small group of transient users were buying cards in bulk preventing the long-term users from getting it.
    2. They will dump these cards back in market in few years, creating a glut creating fluctuation in market.

170HX and its limitations make perfect sense when you look from that perspective. Had LLM boom and 170HX wouldn't have happened, NVidia would have been struggling with a saturated market and tanked stock price. So yeah, it sucks that you can't use the perfect piece of hardware but than that was its purpose.

gymbeaux8• 3 years ago

Did this and similar GPUs actually solve the supply-demand problem of their consumer GTX/RTX cards? I remember the consumer GPUs being price-gouged until crypto crashed, not before.

7speter5• 3 years ago

This thing sold for 4200 dollars so no. Consumer nvidia cards were cheaper and offered good enough hashrates. Also you couldnt find any gpus on shelves for a solid year.

latchkey5• 3 years ago

It is actually nice to see people trying to make old crypto mining hardware work. For the most part, it is all e-waste, but I'm sure there is a few gems out there.

My operation was majority on super old model AMD gpu's (RX470 - 580), so there really isn't much use for them now anyway.

sodality2• 3 years ago

Surely they’re good for budget ML hobbyists? I’m stuck inferencing on CPU and it’s pretty painful, I figure a GPU upgrade would be good for some workloads, though VRAM would be a struggle (maybe stick to whisper models…).

latchkey5• 3 years ago

One would think, but only 8gb vram, no display port, and they are air cooled (big heat sink) too. Plus, there are about 120,000 of these cards sitting in a warehouse in the middle of nowhere; selling one or two at a time really isn't economical.

That said, if you have a few big container trucks and want to pick them all up, I can put you in touch with the right people... heh.

fisherking• 3 years ago

Could you send one over? my rx 580 is beginning to explode and as a student I dont have much money to buy another one

latchkey5• 3 years ago

They don't explode.

loufe2• 3 years ago

I wasn't able to block the obnoxious anime girl on the right side with ublock. I've never had that before. Does anyone know how to find a way around that?

mholm• 3 years ago

The problem is that it isn't an element, it's a background-image to the body. You'd need to overwrite the stylesheet for body, or block the image's url (/img/niconiconi.png)

loufe2• 3 years ago

Thank you!

All GB/s without FLOPS – Nvidia CMP 170HX Review

Related Stories