"Tinyboxes finally have a buy it now button"

https://twitter.com/realgeorgehotz/status/1828197925874463166

by hedgehog0 • 2 years ago

197 104 2 years ago

twitter.com

sdfdsfsdfsdfs• 2 years ago

https://archive.md/RTvI8

WithinReason2• 2 years ago

https://tinygrad.org/#tinybox

Looks like good value, but I wonder if it would get CPU/RAM bottlenecked, especially if you want to train something with a lot of preprocessing in the pipeline. Something comparable I've found with 7x4090 which comes to about $50k, but with much better CPU/RAM (3x CPU, 4x RAM, 5x SSD):

https://www.overclockers.co.uk/8pack-supernova-mk3-amd-ryzen...

gpjt• 2 years ago

Closer to $42k, i think, at least if you're comparing it to the Tinybox price -- the price in pounds on the site includes VAT, which you wouldn't pay as a business or if you were getting it for export outside the UK, whereas you'd need to add on VAT if you were getting a Tinybox in the UK.

SushiHippie• 2 years ago

6x PCIe 4.0 x16 (64 GB/s)

Wikipedia [0] states that PCIe 4.0 x16 has a throughput of ~32GB/s, what does the (64 GB/s) indicate on the website, is this just a typo and you have 6x ~32GB/s or does it mean in total you can "only" expect a throughput of 64GB/s all lanes slots combined?

If so, wouldn't you also be bottlenecked by the PCIe bandwidth (when moving data between CPU and GPU)?

[0] https://en.wikipedia.org/wiki/PCI_Express#Comparison_table

arder2• 2 years ago

Well they're specifying the AMD EPYC and one of the things that the server line of AMD CPUs do that the consumer grade ones don't, is they have lots of connectivity. So for example an AMD EPYC 8324P is a 32 core CPU with 96 lanes of PCI Gen 5. Given that the 4090 GPU is PCI Gen 4, I think that's where you get the discrepancy. The 6 GPUS are connected in parallel to the CPU with 6 x16 connections (96 total lanes), the CPU could do this at Gen 5 (64GBs for each GPU) but the 4090 GPU is Gen4 only, so you'll only actually get 32GBps per connection.

jasomill• 2 years ago

Most EPYCs have 128 PCIe lanes, so I'd expect a full x16 link for all six GPUs.

Pedantically, the combined bidirectional bandwidth of PCIe x16 is ~64 GB/s, as it's a full-duplex ~32 GB/s link, but that's an awfully misleading spec if this is the intent (akin to claiming Gigabit Ethernet is 2 Gb/sec).

andersa10• 2 years ago

It's 32GB/s in both directions. So when exchanging data two GPUs each can do this at 64GB/s. Is that a useful way to measure it? Who knows.

coder543• 2 years ago

It’s weird how non-specific the CPU is there. Why wouldn’t they list a CPU part number? We don’t even know what generation of Epyc it is. (I get that it’s not the focus… but it is still important.)

threeseed• 2 years ago

Very unusual specs on paper.

- Air cooling 6x4090 and a 32 core CPU for sustained peak workloads.

- 3200W total power when a single 4090 can draw close to 600W.

Maybe they are targeting startups who aren't interested in overclocking.

jagrsw4• 2 years ago

This thing could run off a single US outlet (1600W) with some throttling, if I'm not mistaken?

Shame it wasn't designed for EU sockets. 230V*16A = 3700W, or double that on separate breakers!

bguebert• 2 years ago

On their site they say it can run at 220V 15A if you got that.

https://docs.tinygrad.org/tinybox/

apexalpha• 2 years ago

3200W seems perfect for a EU socket.

Seperate breakers aren't really a thing here, at least in my country, usually if you need more power you draw 400V

arder2• 2 years ago

I think the plan with this all along was George went off and built exactly what he thinks he needs for his specific work and then just makes it available. So is the wildly underpowered CPU bad? I don't know, I don't know his use case.

It also seems just weird from a business point of view. He's not going to sell many, he's not going to offer support, he's not at a scale where vendors are going to offer much particular support, and despite being absolutely tiny in scale he's still offering two totally different SKUs.

WithinReason2• 2 years ago

A 4090 draws 450W unless you unlock the power limit

Timshel• 2 years ago

Transient might be an issue. GN discuss transients of ~30/40% over the nominal 450w (https://www.youtube.com/watch?v=j9vC9NBL8zo&t=616s).

And with a distributed training you can end-up with "synchronized" transients over all cards :(.

zten2• 2 years ago

The workstation equivalent, the RTX 6000 Ada, defaults to 300W. You can get most of the performance of a 4090 by capping the power.

BaculumMeumEst• 2 years ago

So is the plan for these to quietly update the hardware as better consumer hardware becomes available? This is a really interesting idea but as a small fry I would definitely be building myself if I went this route.

a1o• 2 years ago

They have a real website, why not link to that instead of Twitter?

davidcollantes• 2 years ago

Agree. See https://tinygrad.org/.

vessenes• 2 years ago

I thought to myself this morning: "boy, that $15k pricetage is tempting." Then I thought to myself "how many times have I downloaded a github repo only to hand-replace cuda with mps, and then tried to figure out if there's a version of xformers that will work this week with my m3?" and then I thought "boy, that $25k is tempting." (15k: Radeon / 25k: Nvidia).

For those wondering, 3200W power, in residential / low-end commercial in the US, they say you'll need two separate circuits, they have a built-in power limiting utility in the OS which will let you safely run on one circuit at reduced speed.

The only part of this that gives me pause is interconnect -- over PCIe, 64GB/s stated. This is much, much lower than infiniband -- can any ML engineers comment on using this box for a full finetune of, say, LLama 3.1 / 70b?

sidewndr46• 2 years ago

a 220 volt 20 amp circuit should be good for over 3500 watts constant load in North America. Why is it requiring two circuits?

hashhar• 2 years ago

IIRC USA is 110V, not 220V?

luma• 2 years ago

Because most households in the US might have maybe 3 breakers setup this way, all of which are likely running critical infrastructure already.

Most folks aren’t going to unplug their water heater to turn on their AI.

michaelt4• 2 years ago

Most likely what they actually mean is:

This server has two IEC C20 connectors, rated for ~16 amps, each feeding a PSU rated for 1600W (i.e. 16A @ 100v)

If you're plugging in to 110v you shouldn't plug them both into the same outlet, as a 20A circuit can't supply 32A.

As each PSU is rated for 1600W you'll have to plug both in to get 3200W even if you're running on 220v - although they'd only draw ~7.2A each in that case.

US Residential 220v dryer outlets are usually wired one-circuit-to-one-outlet, and multi-way adaptors are discouraged. So although plugging two 7.2A loads into a single 20A feed would work from a current perspective (and indeed it's common in Europe), I don't know how easy it is to do legally.

If you're in a data centre with a 3-phase 220v power you probably know what you're doing. Your UPS guy will probably thank you if you split your load over two phases instead of putting the whole load onto one phase.

amluto4• 2 years ago

There are few or no 220 volt circuits in North America. Your choices in that range are 208V or 240V.

But yes, a power supply can draw around 240V times 20A = 4800VA, which is nearly 4800W if the power factor is close to 1. An office in an office building is more likely to have 208V.

andersa10• 2 years ago

You can't fine tune a 70B model with this. It barely even fits the weights before it runs out of vram. Need a bigger machine.

StrangeDoctor4• 2 years ago

I think the idea is you network them together if you need more and most models can be split nicely.

anon389r58r583• 2 years ago

The networking of the tinybox is woefully inadequate. I.e. it only has an OCP 3.0 interface which is unoccupied. If you can fit everything onto one tinybox, then you'll be good, if you cannot, then you'd be better off by having a more professional workstation solution like e.g. NVIDIA RTX cards which have more memory.

georgehotz3• 2 years ago

That OCP 3.0 card has the same link bandwidth as the GPUs, so you can scale out without much loss of all-reduce bandwidth. In practice, for all models except the largest, the ~16GB/s all-reduce is totally fine. You just need to make sure you can all-reduce all weights in your training step time.

Say you are training a 3B parameter model in BF16. That's 6GB of weights, as long as your step time is >=500ms you won't see a slowdown.

plasticchris• 2 years ago

I looked at the specs at the start of the year and just built something with the high end of consumer parts at around 4k usd. I was able to replicate the mlc 2x7900xtx results running some LLMs. Good enough to run most of the big models in gpu memory with a little quantization.

roschdal• 2 years ago

What is the most profitable thing I can do with 10 of these fine Tinybox boxes?

x_may• 2 years ago

Sell them and invest the money

onionisafruit• 2 years ago

At least that’s what geo decided is the most profitable thing to do

1970-01-01• 2 years ago

Looks like the Mac cheese grater now has serious competition.

bob10294• 2 years ago

I had a preorder in for this but I canceled a few weeks back.

My experience trying to run machines this powerful in residential settings has been extremely poor.

All of the Seasonic power supplies that go beyond 1kW or so will trip my shitty (i.e. probably defective) Siemens AFCI breakers. Not even the same circuit all the time.

Even after violating local electrical code, I have found that living with a 1500w+ monster inside my house during the summer at 100% utilization is a complete joke. Unless you live in the perfect datacenter climate (i.e. the people who designed the tiny box), this thing needs to be inside. All of that wattage is pure heat being dumped into your home. The HVAC solutions in most residences were not designed for this kind of heat load. It would be like running your oven with the door hanging open all day. For those of us in places like Texas, this machine simply would not be feasible to run for half the year.

teruakohatu2• 2 years ago

All of the Seasonic power supplies that go beyond 1kW or so will trip my shitty (i.e. probably defective) Siemens AFCI breakers. Not even the same circuit all the time.

I don’t know much about US electrical standards but aren’t your residential circuits rated for 1800w or 2400w? Here in New Zealand they are 2400w and people regularly plug in 2400w fan heaters without issue.

The HVAC solutions in most residences were not designed for this kind of heat load. It would be like running your oven with the door hanging open all day. For those of us in places like Texas, this machine simply would not be feasible to run for half the year.

Yes it wouldn’t be pleasant running this 24/7 in summer in any living space. But you could install a heatpump with 7kw of cooling capacity which should handle it (adding to the electricity bill).

cptskippy2• 2 years ago

In the US the National Electric Code caps draw at 80% of rated load. So a 15A circuit is permitted a 1440W load even though it should handle 1800W.

bob10294• 2 years ago

I don’t know much about US electrical standards but aren’t your residential circuits rated for 1800w or 2400w?

The residential AFCI issue I describe isn't about the wattage directly. It's about transient currents generated by large switch-mode power supplies being detected as arc faults. Similar concern as with induction motors.

ofrzeta• 2 years ago

Crazy. Our hair-dryer has 2kW.

caeril• 2 years ago

If you're spending $15k on a box, you can also spend $1200 for a small insulated shed kit and $800 for a small mini-split heat pump. I live in a much warmer summer climate than Texas and this solution works fine for me for my small network cabinet.

groby_b• 2 years ago

If the main argument for the box is compute/$, "and then you need to spend another 20% on top to even make it work" is not the most winningest position. (20% because you also need to pay an electrician for the two-circuit wiring. Well, three, you want to run the heat pump too)

At that point it isn't super price-efficient, it's an absolute space hog, and you need to maintain a whole bunch of infra. Still might work for you, but it's losing a lot of general appeal

whamlastxmas• 2 years ago

Could you not duct the heat through a hose and out a window? Like with portable AC units

jejeyyy77• 2 years ago

yep, or drop it in your basement. also, will help heat your home during winter.

Arelius2• 2 years ago

All of the Seasonic power supplies that go beyond 1kW or so will trip my shitty (i.e. probably defective) Siemens AFCI breakers.

In my experience, the Siemens AFCI just do that. I recommend switching them out for Eaton AFCI. That fixed all my nuisance tripping, especially from induction in other lines

bob10294• 2 years ago

I didn't realize Eaton had AFCI breakers listed for use with Siemens panels (or that they work better). I swapped it out for a non-AFCI Siemens, but if I can make it code compliant I'd much rather that.

EncomLab• 2 years ago

"tinybox red is 6x 7900XTX, tinybox green is 6x 4090"

So the red is ~$5k in gpu's - where is the other $10k going?

andersa10• 2 years ago

Motherboard and CPU, memory, NVMe drives, PSUs, SlimSAS cables and breakouts, a custom machined case, assembly, support.

You're free to try building one yourself for cheaper. If you consider your time for researching/assembling/testing it to be worthless, and are happy with a contraption in a miner frame, then you can probably do it.

zten2• 2 years ago

PC builds seem to short circuit everyone's pricing logic and drive any labor cost down to $0, just because they're willing to do it for free. Anything above that $0 is considered overpriced.

scoot• 2 years ago

The $15k tinybox red is the best perf/$ ML box in the world. It's fully networkable, so that's the metric that matters.

No it isn't. Capex is only part of the equation. Opex (power and cooling amongst other things) is important. And networking at scale isn't cheap either.

zelcon• 2 years ago

Unless you're making money off it, $15k + however much you have to spend on installing a new breaker panel is too much to spend on hardware that will be outdated in 2 years. If you're making money off it, but you're still cheap, then buy a Supermicro + H100s and colo it in a datacenter. If you're not cheap, you'll just use Azure. So I'm not sure who this product is supposed to be for.

justinclift• 2 years ago

Anyone know how well this would compare to the Nvidia based workstations at the GPTShop.ai place?

ie: things like this https://gptshop.ai/config/indexus.html

ddmf• 2 years ago

Got confused with George Motz and thought this was some super burger making thing.

"Tinyboxes finally have a buy it now button"

Related Stories