The tiny corp raised $5.1M

https://geohot.github.io//blog/jekyll/update/2023/05/24/the-tiny-corp-raised-5M.html

by super256 • 3 years ago

629 317 3 years ago

geohot.github.io

freediver• 3 years ago

True entrepreneur. Having a vision and ignoring naysayers. Go George!

nwienert2• 3 years ago

Gotta have money to start to keep taking risks

SXX4• 3 years ago

While it's not available to many of us (and especially those outside of US) it doesn't take that much money to just tinker with whatever you want. Just have to cut time wasted on news and politics, socialising and literally anything outsidie of being hacker or entrepreneur.

Most people just would never take that risk and will stick to their well-paying job.

coffeebeqn• 3 years ago

The way they make money is for AMD to buy them out if they’re successful

vruiz2• 3 years ago

Which would make total sense for AMD if they pull it off.

aeyes• 3 years ago

Why would they buy something that is open source? They could acqui-hire but Hotz doesn't strike me as a person that would stay at a big corp like AMD for a significant amount of time.

patentatt• 3 years ago

If they can achieve something competitive with CUDA for $5m, why hasn't AMD done it yet?

nacs• 3 years ago

AMD/ATI has always been terrible at drivers/software for their GPUs going back decades.

It’s really as simple as that and it still hasn’t changed so nvidia is dominating them in AI as a result.

1letterunixname3• 3 years ago

Their stuff was garbage back when there was VLB.

crpowers• 3 years ago

AMD has been tackling exactly the wrong problems. They poured their money into a porting solution for developers to take CUDA code and run it on their GPUs. I guess they didn't find it worth it to really compete. I doubt it's about being able to tackle this problem with $5m, but rather convincing the company they can win.

1letterunixname3• 3 years ago

I still don't understand what problem they're trying to solve in EPYC in the hypervisor space with encryption.

They should've been adding tensor cores and neural acceleration to their CPUs. The need for headed graphics cards is moot and wasteful. NVIDIA solved this with the A100.

NVIDIA may spin into a mainstream enterprise CPU and systems vendor as a sales channel for converged CPU-GPU solutions beyond what they're already doing.

sjtgraham• 3 years ago

You can say this about every startup that has ever existed.

mhh__3• 3 years ago

Vision.

That being said I am still sceptical.

Shekelphile2• 3 years ago

I doubt they have even spent $5M on developing ROCm even though it's been a thing for nearly 10 years. AMD is just notoriously stingy about investing in things outside their core business.

foobiekr7• 3 years ago

Every single startup since maybe 1980 that is not borderline criminal has faced at least one incumbent that is better funded and has, frankly, more talent with better experience.

The reason they sometimes win is huge structural or incentive issues in the large companies. And they don't win often.

AMD has failed to fix the problem for years. Is it because the business structure doesn't incentivize it? Is it because the company's entrenched culture is opposed to compensation that might attract the right talent (often out of "fairness")? Is it because there's some internal owner for the function that keeps fucking up but for political reasons the CEO won't replace them and no one else can work on the thing?

Any of these are possible. I have seen - personally witnessed - all of these and more at large companies. We don't know the reason, but we can sort of guess as to the shape of it.

reaperman3• 3 years ago

Love this overall. Wonderful. But I wouldn't say Cerebras failed just yet -- they're committing to OpenXLA which may provide a better dev-experience in the long run than Nvidia lock-in.

neom2• 3 years ago

Seems like a much better mission for George Hotz to go on than single handedly trying to fix Twitter.

vippy5• 3 years ago

George Hotz is a talented engineer but he absolutely does not have the social science background to "fix" Twitter.

threeseed3• 3 years ago

Being an engineer at Twitter is much more akin to an enterprise like a bank or telco.

You need to be a team player and work across different groups e.g. Product, Testing, SRE etc in order to successfully get features into Production.

So being a talented engineer is useful but having high emotional intelligence and being able to negotiate and collaborate is far more important.

confoundcofound• 3 years ago

You don’t need a social science background to fix Twitter.

asdfman1233• 3 years ago

It wasn’t even fixing Twitter. It was marginally improving search.

threeseed3• 3 years ago

From the actual engineers that were posting on Twitter marginal is putting it lightly.

The quality, arrogance and ignorance of enterprise SDLC was akin to that of a 1st year grad.

stefan_• 3 years ago

I thought he was doing AI self-driving. I see we are now moving the goalposts to "generalized intelligence" so I suppose there was no point for him to put up with just cars anymore.

dclowd9901• 3 years ago

I’m not convinced ai driving can even work without GAI. Seems like a logical next step to the gigantic yak shave that is autonomous driving.

ergocoder• 3 years ago

Reading with the term "moving goalposts" leaves a really bad impression.

It is like it is wrong to stop working on one idea and start working on the other.

reaperman3• 3 years ago

I think that was a great mission for the individual, George Hotz. It's beneficial to be humbled by complex systems and imposing bureaucracy -- it makes future endeavors better informed.

plorkyeran• 3 years ago

There isn't really any evidence he was humbled by the experience.

tzhenghao4• 3 years ago

I think the only way to start an AI chip company is to start with the software. The computing in ML is not general purpose computing. 95% of models in use today (including LLMs and image generation) have all their compute and memory accesses statically computable.

Agree with this insight. One thing Nvidia got right was a focus on software. They introduced CUDA [1] back in 2007 when the full set of use cases for it didn't seem very obvious. Then their GPUs had Tensor cores, and more complementary software like TensorRT to take full advantage of them post deep learning boom.

Right as Nvidia reported insane earnings beat too [2]. Would love more players in this space for sure.

[1] https://en.wikipedia.org/wiki/CUDA [2] - https://www.cnbc.com/2023/05/24/nvidia-nvda-earnings-report-...

foobiekr7• 3 years ago

I know the founders of an AI chip company that taped out and got working chips on their first go. They got their chip done, it’s pretty solid. Chip has great perf and is super power efficient, a solid delivery. I knew they'd nail it and they did.

The SW story is a train wreck, though. The problem basically was that they couldn’t hire any good SW people. As I said I know the founders. They are both genuinely decent guys, they put their own money in so they have some (well, minimal) skin in the game, and they know a ton of expert-level embedded and systems coders with between 20 and 40 years of hard core experience. As far as I can tell, they weren't really able to get anyone that we know in common to join. I certainly did not, and no one I know did either. Last I heard they'd had to hire third choice guys in Europe to do the work and it wasn't going well.

There's a pretty good reason for it, and it comes down to a sociological problem. HW people don’t value SW people. It's just basically true and has been true everywhere I've looked. Maybe if you're doing a system (like a router or maybe a drone) then the HW people will begrudgingly admit that the SW is a major part of the delivery, but that isn't true for chip companies (including chips-on-reference-boards).

You can rest assured that at a chip company, all of the high comp people in the company are going to be on the ASIC team and the SW team will never be on the same tier. The argument is always the same, no matter how many times it bites the companies on the ass and sends them careening into the dumpster: “yes, but the chip without SW is the chip! we can buy SW, if we have to. SW without the chip has zero value.”

Almost every chip company ends up like that, and the kind of low level, experienced SW people that work in the space know to avoid them and work at systems companies instead.

As far as I've been able to determine, with _maybe_ the exception of Cerebras - maybe - this is the situation that has played out at all of the 201x AI chip companies. They get founded by ASIC guys, most of whom have more than a small chip on their shoulder about the relative value of ASICs-vs-SW. These guys are all ex-SGI, ex-Sun, ex-Google, ex-Nvidia, ex-Intel HW guys who saw SW people making a lot more, not just in broader industry terms over the last few years, but at hardware-focused companies. In general, ASIC guys make less than SW guys unless they are the very narrow set of top level architects. IMHO from a value creation standpoint, that is _super unfair_ and I am not here to justify it, but it is how it is. The result poisons ASIC companies. SW people who know what needs to be don't won't go to them most of the time, for good reason, and so they fail.

So I will say, given that, starting with SW first is brilliant.

sanxiyn6• 3 years ago

I also know the founder of an AI chip company (ex-AMD), they taped out in 2022, got working chips on the first try. Miraculously, they hired a good software team, some even with previous compiler experience. In their brochure they write things like:

In 2022, FuriosaAI remained the only startup to submit results in MLPerf Inference... This time, through purely enhancements in the compiler, our team was able to double the performance on the exact same silicon.

https://www.furiosa.ai/

Maybe it helped they are based in South Korea. Other places to work in South Korea doing system programming is not very attractive.

typon• 3 years ago

As a SW person who's worked at AI chip company, this is 100% true.

wahnfrieden4• 3 years ago

were the SW folks offered compensation commensurate with their low perceived status? like HW gets C/cofounder, SW gets early employee package. Because that's a different story about money and ownership. of course good people reject wage labor.

Mistletoe• 3 years ago

Can anyone comment on the TinyBox they are taking preorders for?

The tinybox

738 FP16 TFLOPS

144 GB GPU RAM

5.76 TB/s RAM bandwidth

30 GB/s model load bandwidth (big llama loads in around 4 seconds)

AMD EPYC CPU

1600W (one 120V outlet)

Runs 65B FP16 LLaMA out of the box (using tinygrad, subject to software development risks)

$15,000

1letterunixname3• 3 years ago

There's a reason no one uses ATI GPUs in datacenters. Their dev support is shit.

Don't waste your money.

Buy 6 RTX 4090's and a decent ECC-memory server, and call it a day.

wmf6• 3 years ago

Didn't read the article.

guraf• 3 years ago

I thought you weren't allowed to use Nvidia's consumer GPUs in the datacenter?

wmf6• 3 years ago

We can't really comment much on it because a bunch of specs are lacking. Is it using 6x 7900 XTXs? Which Epyc CPU (Epycs vary in price from $1K to $11K)?

staunch2• 3 years ago

I like George's style and wish him well. But I'm not optimistic about their chances of selling $15k servers that are $10k in parts (or whatever the exact numbers are).

It's just too easy for anyone to throw together a Supermicro machine with 6x GPUs in it, which is what it sounds like they'll be doing.

My guess is they'll end up creating some premium extensions to the software and selling that to make money. Or maybe they can sell an enterprise cluster manager type thing that comes with support. He's good at software so it makes sense for him to sell software.

And maybe the box will sell well initially just as a "dev kit" type thing.

wmf6• 3 years ago

To me this looks like a way to mask donations.

Animats• 3 years ago

And maybe the box will sell well initially just as a "dev kit" type thing.

Price: $15,000.

If they had a "lite" model that sold for $1500, and were actually shipping....

Shekelphile2• 3 years ago

It's just too easy for anyone to throw together a Supermicro machine with 6x GPUs in it, which is what it sounds like they'll be doing.

HPC compute is well advanced past just slapping GPUs into generic supermicro servers anyway. Without semi-custom hardware and equivalents to nvlink/nvswitch AMD won't ever be competitive in the HPC space.

minhazm• 3 years ago

selling $15k servers that are $10k in parts

Have you seen what a DGXA100 costs? It starts at $199k for 8 40GB A100's, which have a list price of $10k each. So the GPU costs are $80k. What do you get for the extra $120k? 1TB ram, 2 2TB NVMe OS drives, 4 4TB NVME general storage, and 8x200Gbit infiniband. I would guess no more than 20k all of the remaining hardware. So that's a ~$100k computer selling for $200k. And that's with NVDA likely making massive margins already on the A100 and the Infiniband hardware.

The reality is that companies want to buy complete solutions, not to build and manage their own hardware. A $15k a computer that's $10k in parts is not a large markup at all for something like this.

quickthrower24• 3 years ago

What is the cost of an equivalent setup using A100s?

I have no idea what I am doing but here goes!

By [1] we have 156 FP16 TFLOPS, taking their non "*" (* = with sparsity) value. So you need 5 So $40,000? pls the other stuff, and someone to make a profit putting it together say $50,000?

So this setup is 3 times cheaper for the same.

If I am allowed to use the sparsity value it is 1.5 times cheaper.

[1] https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent...

redox992• 3 years ago

A100s have a much worse performance per dollar than 3090s/4090s (which are the direct competitors to the 7900x)

extasia• 3 years ago

Fwiw, I can't think of a single popular neural net architecture that takes advantage of sparsity.

nightowl_games• 3 years ago

I wonder what Tenstorrent is doing. Didn't realize they are Canadian.

https://tenstorrent.com/research/tenstorrent-raises-over-200...

Havoc• 3 years ago

Glad he is going ahead with this. Will make for many entertaining live streams no doubt

j0hnyl• 3 years ago

what does he mean by "tape out" chips?

bigdict2• 3 years ago

https://en.m.wikipedia.org/wiki/Tape-out

detaro2• 3 years ago

Industry term for a chip design being sent to manufacturing/being produced.

bloggie• 3 years ago

When transforming the logic gate design of a chip into the lithographic plates used for chip production, the plates were originally made by applying tape to create the photo masks. The name stuck and it now means to move a semiconductor project from the design phase to the manufacturing phase.

bigdict2• 3 years ago

The end goal is getting acquired by AMD of course.

thatguyknows• 3 years ago

I was always surprised at how AMD hasn't already thrown a bunch of money at this problem. Maybe they have and are just incompetent in this area.

My prediction is AMD is already working on this internally, except more oriented around PyTorch not Hotz's Tinygrad, which I doubt will get much traction.

tzhenghao4• 3 years ago

I think AMD is going down a different path, ie. ROCm then partnering with ML frameworks further up the stack for first class support.

https://pytorch.org/blog/pytorch-for-amd-rocm-platform-now-a...

lbhdc• 3 years ago

He mentioned ROCm, and apparently had lack luster experience with it.

The software is called ROCm, it’s open source, and supposedly it works with PyTorch. Though I’ve tried 3 times in the last couple years to build it, and every time it didn’t build out of the box, I struggled to fix it, got it built, and it either segfaulted or returned the wrong answer. In comparison, I have probably built CUDA PyTorch 10 times and never had a single issue.

meragrin_• 3 years ago

Now only if they would support their hardware and Windows.

PeterisP2• 3 years ago

AMD is not going down the path of ROCm; perhaps they claim to do so, but as evidenced by the lack of both effort and results, they clearly are not.

The parent post is surprised that they still aren't making the appropriate investments to make it work. They kind of started to do that a few years ago, but then it fell on the wayside without reaching even table stakes, which in my opinion would require providing a ROCm distribution that works out of the box for most of their recent consumer cards (i.e. those cards which the enthusiasts/students/advocates/researchers might use while choosing which software stack to learn, and afterward base corporate compute cluster purchasing decisions on whether they support the software they wrote for e.g. CUDA+Pytorch), and they seem to be failing at that.

SXX4• 3 years ago

AMD is limited by numerous patent and other legal issues. For this reason small company that releases everything as open source have some chances to beat AMD on their own hardware.

mortenjorck• 3 years ago

Obscurity is only a viable form of security until the patent holder becomes aware of you.

camdenlock• 3 years ago

I’d love to see this succeed, because I own a 7900 XTX already, but there’s so much already built on top of PyTorch. Why would anyone port it all to tinygrad or whatever? Bummer.

pizza• 3 years ago

In theory pytorch compiler can boil down to 50 or so fundamental functions and tinygrad IR to 12. So possibly you could just re-map a fairly limited set of base instructions. Devil’s in the details though..

currymj• 3 years ago

people already ported a lot of stuff from pytorch to jax.

if you're a research scientist or grad student, to a certain extent a lot of projects are "greenfield" so it's easy to jump on a new framework if it is nice to use and offers some advantage.

quickthrower24• 3 years ago

I started tinygrad in Oct 2020. It started as a toy project to teach me about neural networks

Shows you what is possible in 2.5 years. Keeps me motivated to learn.

vippy5• 3 years ago

The math isn't super difficult. Some books will try to throw a mess of differential equations at you, but some simple calculus is all you need for backpropagation.

quickthrower24• 3 years ago

I have been through the math thanks to the youtube videos by A. Karpathy. Deriving some of the differentials, e.g. for batchnorm seems fairly hard (hard as in slogging through something with many steps where you can't make a mistake at any step). But the principles are quite simple - I think by design. If they were hard to compute or reason about then the neural net wouldn't work very well!

sva_• 3 years ago

Doing the compute efficiently, especially from Python, is the tricky part.

whalesalad• 3 years ago

Didn't realize the 7900 XTX was so good at this kind of work. Glad to have went with it over the 4080.

jitl• 3 years ago

The whole point is that it’s not good at this work — and it’s a $Xx billion opportunity to make it work.

csense• 3 years ago

I respect Geohot's reputation and this company looks amazing. I might be in the market to work there... except "No Remote."

For such a smart guy, locking yourself out of a ton of talent by requiring software developers to be on-site in 2023 seems...out of character, to put it politely.

(Rephrased, my original post was a bit too ad hominem and accumulating downvotes rapidly. I wanted to delete this entire comment but apparently HN no longer allows comments to be deleted.)

throwawayadvsec• 3 years ago

what about IP theft though?

detaro2• 3 years ago

what about it?

duxup• 3 years ago

How big is Tiny Corp?

I doubt they need mass volumes of employees at this stage and they maybe want to work closely with the people they choose?

ftxbro• 3 years ago

For such a smart guy, locking yourself out of a ton of talent by requiring software developers to be on-site in 2023 seems...out of character, to put it politely.

I mean a lot of smart people seem to do their hacking by themselves. I'm thinking like Fabrice Bellard. This is at least a step beyond that.

paxys• 3 years ago

Software is part of it, sure, but I doubt anyone can realistically work on this project/company without being around a bunch of specialized hardware and iterating on prototypes. Hard to contribute to any of that from home.

foobiekr7• 3 years ago

A lot of people have now had the direct experience that new things which are highly technical or highly collaborative are not really compatible with the remote work thing. I know that's hard for a lot of people to hear, but the world is not web apps (which do remote well) and a lot of projects benefit hugely from being able to grab the two or three people and get into a room with a whiteboard.

dasil003• 3 years ago

Let's be honest here: remote and in-person is a major tradeoff with significant pros and cons on both sides. Remote work increases your talent pool by 5 orders of magnitude and removes commute overhead, but in-person work increases your communication bandwidth and team cohesion in similarly dramatic ways. Hybrid solutions put a hard cap on the upside of either path and therefore tend to give you the worst of both worlds. It makes perfect sense to be opinionated about this, especially at the startup stage. I respect the decisiveness (even though I would be very reluctant to go back to full-time commute).

quickthrower24• 3 years ago

Also in person is a good filter if you are getting too many applications, which I am sure this will assuming the pay is good (but those $100 bounties implies maybe not - anyone who could do those in say 15 minutes would be worth more than $400/h)

ActorNightly3• 3 years ago

Let's say that you are someone with a dedicated drive to contribute to AI, with the ability to grasp the complex program space, and spend many hours figuring out the solution to the problem. I.E the perfect tinygrad candidate.

There is a high chance that you are probably neurodivergent to some extent.

So instead of WFH, where you can remove distractions and work on your own time, you are now forced to abide by someone else's schedule, take time commuting, e.t.c

In office work is for people/positions that require hands on work with hardware, or you are hiring for replaceable positions where people don't have dedication to the cause and are going to do as little work as possible for the same pay. Tinygrad is neither.

cyrux004• 3 years ago

From Twitter

Remote work is available to everyone on GitHub. If you submit a bunch of good PRs and show me you are easy to work with, I'm down to pay per project.

Source https://twitter.com/realGeorgeHotz/status/166153013618397184...

I'm not anti remote, I'm anti full time remote. It's hard to build a culture

PeterisP2• 3 years ago

The big benefit of remote is being able to live 500 miles away and/or in a different country; and that requires being full-time remote.

If he's anti full-time remote, then his pool of candidates is still limited to those who live in San Diego or very close to it.

ActorNightly3• 3 years ago

If culture is the thing that is holding your company to it's purpose, you aren't going to succeed.

In the same way Comma went from "Our goal is to solve AI, Comma body is the next big thing" to George peacing out because now they are just doing the busy work to make more money.

Kiro• 3 years ago

Remote work requires very different management and tooling. I've seen remote companies fail the last couple of years where this was not taken into account. It's hard to run a remote company.

mhh__3• 3 years ago

Good idea. I don't think George Hotz has the skill set to actually deliver on a lot of this stuff (specifically I suspect trying to replace the compiler for the GPU is something that he will probably make a song and dance about with some simple prototype but then quietly scrap it because even for AI workloads its still a very very tricky problem) but he has the strength of vision to get and direct other people to do it for him.

wahnfrieden4• 3 years ago

do you think those talented workers will accept giving him ownership over that work and its value?

mhh__3• 3 years ago

Traditionally the way startups entice people is by giving them equity

dharma1• 3 years ago

So much untapped potential in AMD, and funny they keep failing at the software and geohot has to save them

turnsout• 3 years ago

I clicked through to the previous blog post, to read more about the unit of a "person" of compute [0]. Definitely worth a read, if only for this quote:

One Humanity is 20,000 Tampas.

I'll never think of humanity the same way!

[0] https://geohot.github.io/blog/jekyll/update/2023/04/26/a-per...

kevmo314• 3 years ago

Nvidia or AMD, the real winner here is truly TSMC.

godelski9• 3 years ago

There’s a [Radeon RX 7900 XTX 24GB] already on the market. For $999, you get a 123 TFLOP card with 24 GB of 960 GB/s RAM. This is the best FLOPS per dollar today, and yet…nobody in ML uses it.
I promise it’s better than the chip you taped out! It has 58B transistors on TSMC N5, and it’s like the 20th generation chip made by the company, 3rd in this series. Why are you so arrogant that you think you can make a better chip? And then, if no one uses this one, why would they use yours?
So why does no one use it? The software is terrible!
Forget all that software. The RDNA3 Instruction Set is well documented. The hardware is great. We are going to write our own software.

So why not just fix AMD accelerators in pytorch? Both ROCm and pytorch are open sourced. Isn't the point of the OSS community to use the community to solve problems? Shouldn't this be the killer advantage over CUDA? Making a new library doesn't democratize access to the 123 (fp16-)TFLOP accelerator. You fix pytorch and suddenly all the existing code has access to these accelerators. Millions of people now have This then puts significant pressure on Nvidia, as they can't corner the DL market. But it is a catch-22 because the DL market already is mostly Nvidia so it takes priority. Isn't this EXACTLY where OSS is supposed to help? I get Hotz wants to make money, and there's nothing wrong with that (it also complements his other company), but the arguments here seem more for fixing ROCm and specifically the pytorch implementation.

The mission is great, but AMD is in a much better position to compete with AMD. They caught up in the gamer's market (mostly) but have a long way to go for scientific work (which is what Nvidia is shifting focus to). This is realistically the only way to drive GPU prices down. Intel tried their hand (including in supercomputers) but failed too. I have to think there's a reason that's not obvious to most of us as to why this is happening.

Note 1:

I will add that supercomputers like Frontier (current #1) do use AMDs and a lot of the hope has been that this will fund the optimization from two places: 1) DOE optimizing their own code because that's the machine that they have access to and 2) AMD using the contract money to hire more devs. But this doesn't seem to be happening fast enough (I know some grad students working on ROCm).

Note 2:

There's a clear difference in how AMD and Nvidia measure TFLOPS. techpowerup shows AMD at 2-3x Nvidia, but performance is similar. Either AMD is crazy underutilized or something is wrong. Does anyone know the answer?

wmf6• 3 years ago

It's often less work to start from scratch than to fix an extremely complex broken stack. Of course people also say this when they just want to start from scratch.

RDNA 3 has dual-issue that basically isn't used by the compiler so half the FPUs are idle.

cma• 3 years ago

Nvidia has stuff like hardware sparsity support. Modern methods (RigL) can let you train sparse for a 2X speedup.

Memory bandwidth (sparsity helps) and networking connectivity (Nvidia bought Mellanox and other networking companies) are important too. They are also using a lot of die space on raytracing stuff that they don't waste on the datacenter versions presumably.

hedgehog3• 3 years ago

I know a fair amount about this problem, my last startup built a working prototype of a performance-portable deep learning framework that got good performance out of AMD cards. The compiler stack is way harder than most people appreciate because scheduling operations for GPUs is very specific to the workload, hardware, and app constraints. The two strongest companies I'm aware of that are working in this area now are Modular.AI and OctoML. On the new chip side Cerebras and Tenstorrent both look quite interesting. It's pretty hard to really beat NVIDIA for developer support though, they've invested a lot of work into the CUDA ecosystem over the years and it shows.

tzhenghao4• 3 years ago

This. Modular and OctoML are building on top of MLIR and TVM respectively.

It's pretty hard to really beat NVIDIA for developer support though, they've invested a lot of work into the CUDA ecosystem over the years and it shows.

Yup, strong CUDA community and dev support. That said, more ergonomic domain specific languages like Mojo might finally give CUDA some competition though - it's still a very high bar for sure.

godelski9• 3 years ago

This seems pretty reasonable and matches my suspicions. It is not hard for me to believe that CUDA has a lot of momentum behind it, not just in users, but in optimization and development. And thanks, I'll look more at Octo. As for Modular, aren't they only CPU right now? I'm not impressed by their results, as their edge isn't strong over PyTorch, especially scaling. A big reason this is surprising to me is simply how much faster numpy functions are than torch. Like just speed test np.sqrt(np.random.random(256, 1024)) vs torch.sqrt(torch.random(256, 1024)). Hell, np.sqrt(x) is also a lot slower than math.sqrt(x). It just seems like there's a lot of availability for optimization, but I'm sure there are costs.

When we're presented with problems where the two potential answers are "it's a lot harder than it looks" and "the people working on it are idiots" I tend to lean towards the former. But hey, when it is the latter there's usually a good market opportunity. Just I've found that domain expertise is seeing the nuance that you miss when looking at 10k ft.

nl2• 3 years ago

So why not just fix AMD accelerators in pytorch? Both ROCm and pytorch are open sourced. Isn't the point of the OSS community to use the community to solve problems?

Because there's no real evidence that AMD cares about this problem, and without them caring your efforts may well be replaced by whatever AMD does next in the space. Their Brooks language[1] is abandoned, OpenCL doesn't compare well, ROCm is like the Sharepoint of GPU APIs (it ticks boxes but doesn't actually work very well).

So why not just fix AMD accelerators in pytorch

Why not just buy NVidia? They care deeply about the space, will actually help you if you have trouble, etc etc.

Even using Google TPUs is better: Google will help you too.

While everyone using NVidia isn't great for the market as a whole as an individual company or person it makes a lot of sense.

Read "The Red Team (AMD)" section in the linked article:

The software is called ROCm, it’s open source, and supposedly it works with PyTorch. Though I’ve tried 3 times in the last couple years to build it, and every time it didn’t build out of the box, I struggled to fix it, got it built, and it either segfaulted or returned the wrong answer. In comparison, I have probably built CUDA PyTorch 10 times and never had a single issue.

This is geohot. He knows how to build software, and how to fix problems.

Note that "Our short term goal is to get AMD on MLPerf using the tinygrad framework."

There's a clear difference in how AMD and Nvidia measure TFLOPS. techpowerup shows AMD at 2-3x Nvidia, but performance is similar. Either AMD is crazy underutilized or something is wrong. Does anyone know the answer?

From the linked article:

That’s the kernel space, the user space isn’t better. The compiler is so bad that clpeak only gets half the max possible FLOPS. And clpeak is a completely contrived workload attempting to maximize FLOPS, never mind how many FLOPS you get on a real program

[1] https://en.wikipedia.org/wiki/BrookGPU

godelski9• 3 years ago

This is geohot. He knows how to build software, and how to fix problems.

This is a nonsequitor. This feels like when my uncle learns that I know how to program he asks me to build a website. These are two different things. I do ML and scientific computing, I'm not your guy. Hotz is a wiz kid but why should we expect his talents to be universal? Generalists don't exist.

And we're talking the guy who tweeted about believing that the integers and reals have the same cardinality right? Between that and his tweets on quantum we definitely have strong evidence that his jailbreaking skills don't translate to math or physics.

He's clearly good at what he does. There's no doubt about that. But why should I believe that his skills translate to other domains?

STOP MAKING GODS OUT OF MEN. Seriously, can we stop this? What does Stanning accomplish? It's creepy. It's creepy if it is BTS, Bieber, Elon, Robert Downey Jr, or Hotz.

Read "The Red Team (AMD)" section in the linked article:

Clearly I did, I quoted from it. You quoted from the next section (So why does no one use it?).

mastax• 3 years ago

So why not just fix AMD accelerators in pytorch?

It doesn't fit the business model. I mean sure, they'll sell AMD computers now like a bootleg Puget Systems. But why buy from the bootleg when I can just buy from the real thing (or AWS) and run tinygrad on there if I want?

So the play is, get people using your framework (tinygrad), then pivot to making AI chips for it:

In the limit, it’s a chip company, but there’s a lot of intermediates along the way.

Seems far fetched but good luck to them.

nibles_and_bits• 3 years ago

Intel Arc A770 16gb $349 had 40 TFLOP of FP16 which is close to 7900 xtx on the FLOPS/$ scale.

Intel's software is much better (MKL, vtune, etc for GPU) and getting better.

sheepscreek• 3 years ago

This is great news. I’ve oft wondered the same about AMD’s GPUs. NVIDIA’s got a clear monopoly.

He made a very good point about how this isn’t general purpose computing. The tensors and the layers are static. There’s an opportunity for a new type of optimization at the hardware level.

I don’t know much about Google’s TPUs, except that they use a fraction of the power used by a GPU.

For this experiment though, my sincere hope is that all the bugs are software only. Supporting argument - if they were hardware bugs, the buggy instructions would not have worked during gameplay.

nalzok• 3 years ago

The main advantage is in the tinygrad IR. It has 12 operations, all of which are ADD/MUL only. `x[3]` is supported, `x[y]` is not.

Can someone educate me why that is the case? Does `x[y]` require a Turing-complete kernel to compute?

brrrrrm• 3 years ago

layer of indirection introduces scatter/gather and other dynamic loads, which is tricky to optimize

impulser_• 3 years ago

Why wouldn't AMD throw a few million at this? Worst case they lose a small amount of money, but best case they finally get good software for their hardware.

The past decade or so, they haven't been able to create any good software for their hardware. They made small improvements but the competition, Nvidia, has also made improvements to their already good software.

It too the point where their software is the reason why most people/companies don't use their products. Their drivers for their customer products are just as bad.

They are very competitive in hardware, but Nvidia dominates them at software which make companies buy Nvidia. No one wants to deal with the pain of AMD software.

AMD is a better company to work with than Nvidia, but it not worth it when it comes to dealing with their software lol.

tormeh• 3 years ago

AFAIK the "AMDs drivers are bad" meme is outdated. Sure, their AI/ML software is garbage, but the graphics drivers are fine.

aiappreciator• 3 years ago

The cutting edge for graphics is all raytracing, and Nvidia still dominates. DLSS3, pathtracing, etc, these are for 'graphics', but heavily dependent on AI post processing, so Nvidia still rules.

So in the gaming market, Nvidia still commands a huge premium. No AMD card can play Cyberpunk on overdrive 4k.

tgsovlerkhgsel2• 3 years ago

I just had my AMD-based machine crash - repeatedly - every time I tried to use Google Maps inside Firefox for longer than a few minutes.

I haven't confirmed, but I strongly assume that either their graphics drivers or something Ubuntu does with Wayland are not fine.

YetAnotherNick• 3 years ago

Maybe AMD is the the one who is investing $5.1M in his company.

foobiekr7• 3 years ago

AMD can't be bothered. They've had literally years to fix the SW issue that allows Nvidia to stomp them in the space quarter after quarter with no fix in sight.

revelio3• 3 years ago

Being able to make stable software isn't just a matter of wanting it though. It's possible to be both bothered about something and still fail.

re-thc2• 3 years ago

Why wouldn't AMD throw a few million at this?

They tried at some point. Like another commenter pointed out elsewhere HW people just don't care about SW. They think HW is superior and SW is the joke. I don't think much of the culture has changed.

A lot of their understanding = workaround the HW, make it work, etc.

daveed• 3 years ago

I don't want to cast any judgement, I just want to ask what the initial product is. The claim is they sell computers, and there's a link to the tinybox. There's a $100 preorder, for a 15k computer (I guess I'd have to pay 14.9k eventually?).

And then we get a computer that... how do I interact with it? Will it have its own OS? Some flavor of linux? Is the intent to work on it directly, or use it as an inference server, and talk over a network?

digitallyfree• 3 years ago

I think the tinybox is meant to be a training/inference server meant for tinygrad and filled with those AMD cards. Very likely it will run Linux.

asdfman1233• 3 years ago

Is this the guy who couldn’t add a feature to Twitter?

thesausageking2• 3 years ago

For background, "geohot", is George Hotz, who's a known hacker / tech personality[0]

This project fits the pattern of his previous projects: he gets excited about the currently hot thing in tech, makes his own knockoff version, generates a ton of buzz in the tech press for it, and then it fizzles out because he doesn't have the resources or attention span to actually make something at that scale.

In 2016, Tesla and self-driving cars led to his comma one project ("I could build a better vision system than Tesla autopilot in 3 months"). In 2020, Ethereum got hot and so he created "cheapETH". In 2022 it was Elon's Twitter, which led him to "fixing Twitter search". And in 2023 it's NVIDIA.

I'd love to see an alternative to CUDA / NVIDIA so I hope this one breaks the pattern, but I'd be very, very careful before giving him a deposit.

[0] https://en.wikipedia.org/wiki/George_Hotz

normaldist• 3 years ago

Comma didn't fizzle out.

https://twitter.com/comma_ai/status/1578517666632900608

thesausageking2• 3 years ago

They're slowing burning through their VC money trying to make a business out of the hobbyist market while Cruise and Waymo have fully autonomous cars deployed in SF and are scaling up.

stoniejohnson• 3 years ago

You're creating a false narrative man. Him creating tinygrad makes sense given he created a self-driving car company which leverages deep learning (and now uses tinygrad in prod, go figure).

It's not like he decided to hop from self-driving cars to 'AI' because fads changed.

sangnoir• 3 years ago

The same self-driving company he left? His "Amazing Journey" post[1] for Comma was illuminating; once a company is "[...] no longer a race car, [but] a boat" GeoHot is likely to bounce. I like AMD as much as the next guy, but I don't want to rely the software being provided by a personality who only really gets excited by working on new projects before the "wild success" stage (if we can call it that for Comma)

1. https://geohot.github.io//blog/jekyll/update/2022/10/29/the-...

foooobaba• 3 years ago

He has been working on tinygrad since atleast 2021. https://geohot.github.io/blog/jekyll/update/2021/06/13/a-bre...

fastball• 3 years ago

AI was already red hot in 2021, so not sure that detracts from OP's point.

I know it seems like ages ago at the current pace, but ya gotta remember that GPT-3 was released in mid-2020.

xpose2000• 3 years ago

To me this is fair criticism. I was thinking the same thing. However, the world does need a viable competitor to Nvidia in AI.

AI is not going anywhere. This is not a fad like some of the others mentioned but more likely than not where the next decade of innovation is built on.

TheAceOfHearts• 3 years ago

My opinion of geohot definitely dropped after he started tweeting how easy it would be to fix Twitter, and then he started soliciting free work. He obviously underestimated the difficulty of shipping a feature across web and mobile. Hacking a prototype is trivial. Making it work well for all platforms, fully accessible, and across all supported languages is a bigger hurdle. It just gave me the impression that he thought frontend development was trivial and he'd just be able to hack it out in a day.

anonymoushn• 3 years ago

Twitter doesn't work well and isn't accessible so we needn't burden every product change with these additional concerns.

godelski9• 3 years ago

For me it was when he said that the cardinality of integers is the same as real numbers. Then I saw his twitter and all the politics and crazy stuff about QM.

baby2• 3 years ago

I don’t remember that he said it was easy, I read some of his tweets and listened (twice) to the twitter space he did with elon and the big take away for me was that he was convinced twitter wouldn’t be able to move much without a heavy refactor. I can believe that. Elon who disagreed with him at the time seemed to have changed his mind now.

pavlov2• 3 years ago

Why is Twitter such a Waterloo for all these obviously accomplished people?

It seems like they've been assuming Twitter is the way it is because it was staffed by technically incompetent leftists, and if only they could apply their own get-things-done attitude and "neutral" politics, then the problem would be trivially fixable.

Where does this fallacy come from? Is it because of the illusory simplicity of the tweet format? Something like: "We just need to come up with the right algorithm and do an embarrassingly parallel run over these tiny 280-character chunks of text. How hard can that be. In my own Very Serious Day Job, I deal with oompabytes of very complex data. This tweet processing stuff should be child's play in comparison."

MangoCoffee• 3 years ago

agree. he was talking about how he can do this and that to fix Twitter then just fizzled, blame the Twitter's code and said Twitter need to rewrite from the ground up.

spaceman_2020• 3 years ago

People completely forget how complex things can get when you have to serve millions of users across platforms, devices, countries, and accessibility settings, and all of it needs to work because you've got hundreds of millions of advertising dollars paying for views and engagement.

blitzar• 3 years ago

how easy it would be to fix ... obviously underestimated the difficulty of ...

If I had a dollar for every job that I didnt get where I estimated the correct degree of difficulty, and they laughed, went with the person who said it would be easy and they could bang it out in a day of sleeping - I would be rich.

The loud optimist wins the 5.1 mil every time.

ilyt3• 3 years ago

To be entirely fair many things would be easy to fix if you can throw away everything and make clean implementation. It's the existing codebase and data that makes that difficult.

Of course, he should know better than to throw claims like that.

ncr1002• 3 years ago

His attitude towards society appears not to be one of being serious, and legally correct, at all times.

It's likely there was some humor and bravado, as is the culture of his east coast origins.

The truth of his engagement with Twitter was, Just based on my watching him and his live streams during that time, that he was looking for a thing to do while he ceded control over comma AI, to a new executive leadership group.

randyrand• 3 years ago

But....Comma still exists, and they launched the 3rd version of their hardware.

Ninjinka• 3 years ago

comma hasn't "fizzled out", it's in active development and drives me to work.

vippy5• 3 years ago

This is the George Hotz MO: I trust him to create a lot less than I trust him to optimize, and I trust him with optimization only in a very narrow technical sense. "Optimizing" Twitter by reducing headcount to 50, regardless of the social or revenue consequences, is actually stupid. Optimizing a whole mess of software that exists between your tensor and the hardware is a decent idea.

georgehotz2• 3 years ago

For reference, comma released a new version of openpilot today, and we now sell comma threes.

https://blog.comma.ai/092release/

djd20• 3 years ago

Congrats! Just read up on thr taco bell drive. Great achievment! https://blog.comma.ai/taco-bell/

sidcool2• 3 years ago

Are you really George Hotz?

penguin_booze2• 3 years ago

Tres commas: https://youtu.be/2XxwhMcUjYU?

ncr1002• 3 years ago

Congratulations, sincerely.

I'm glad that the company exists. I'm glad that smaller car manufacturers were exploring integrating the self-driving software, too.

And, from a celebrity media perspective, I think your engagement at Twitter was underreported especially in gossip mongering pseudo serious technology fan sites like this one.

It was an interesting shift for you, as you mature out of your first big startup into kind of a journeyman phase, in my view. (Tasting different experiences.) Something that is relatable to large numbers of us.

yellow_lead• 3 years ago

Some of the highlights from the Twitter saga:

1. Soliciting others to do his work - and offering internships to others that can help, in a kind of internship MLM (He was an intern at the time).

2. Complaining about how Twitter doesn't run on his laptop

baby2• 3 years ago

Did comma actually tank? I was under the impression that they actually had a good product and were really honest about it too.

YOJ3• 3 years ago

that is indeed still the case, the comment is just false. comma just released support for ford cars aswel. and open pilot got rated the best self driving tool above Tesla by consumer reports

ohashi• 3 years ago

Did he fix Twitter yet?

prepend• 3 years ago

That’s funny, I remember blackra1n and it was pretty clever.

andrewmcwatters• 3 years ago

This is a pretty disingenuous take on a guy working with others to create solutions in spaces where there is literally no other comparable competitor.

What kind of comment is this on a site that used to be called Startup News? Even if that doesn’t resonate with you, isn’t what he’s talking about pure hacker ethos anyway?

agnosticmantis• 3 years ago

The human brain has about 20 PFLOPS of compute.

Where is this number coming from? The number of spikes per second?

Edit: doing a quick search, it doesn’t seem like there’s a consensus on the order of magnitude of this. Here’s a summary of various estimates: https://aiimpacts.org/brain-performance-in-flops/

reaperman3• 3 years ago

No idea. I don't think there's any scientific consensus on even an upper limit of a human brain's FLOP equivalence.

re-thc2• 3 years ago

Where is this number coming from?

Used 20 PFLOPS of compute to simulate it.

chubs• 3 years ago

He claims there's a $999 AMD card that gives 123 TFLOPS, and his tinybox will cost $15k for 738 TFLOPS. In other words, the tinybox will have 6 of these GPUs, eg $6000 cost price. It seems a steep markup from 6k to 15k, and if the software is open-source, i'm not sure why you wouldn't build your own? Or is it worth 9k for a custom motherboard that can fit so many GPUs. Or can you buy 6-GPU motherboards off the shelf? Just curious what people think. Not being disparaging, kudos to geohot :)

mattnewton• 3 years ago

As someone who has built their own deep learning rigs in the $10k BoM range, frankly it's a pain in the ass and I would gladly pay that in the future. I probably will pay lambdalabs a much larger markup.

sbrother• 3 years ago

As someone who bought a deep learning rig for about $12k from lambdalabs years ago, I can't recommend them strongly enough. The support (and not having to deal with building it out myself) was well worth the markup. They're also just really great to deal with.

zdw• 3 years ago

Logic boards and CPUs with the necessary PCIe lanes capable of feeding the GPUs, storage fast enough to do the 30GB/s that is in the spec sheet, power supplies and a case to hold such a power hungry system, etc. is not cheap.

$15k seems like a kind of low - if you've tried to spec out a server with similar quantities of accelerators recently, you'd have trouble hitting that figure, even if using consumer grade GPUs.

AlotOfReading• 3 years ago

You can buy 6+ gpu motherboards, but the consumer ones are solely for mining because no normal person has that many cards and no consumer CPU exposes enough lanes to properly support that many GPUs. A lot of enterprise vendors exist to sell you that kind of system in the enterprise space, but you should expect to still be paying 5 figures minimum. $15k seems like the low end to me.

SkyMarshal• 3 years ago

I think the only way to start an AI chip company is to start with the software. The computing in ML is not general purpose computing. 95% of models in use today (including LLMs and image generation) have all their compute and memory accesses statically computable.
Unfortunately, this advantage is thrown away the minute you have something like CUDA in your stack. Once you are calling in to Turing complete kernels, you can no longer reason about their behavior. You fall back to caching, warp scheduling, and branch prediction.
tinygrad is a simple framework with a PyTorch like frontend that will take you all the way to the hardware, without allowing terrible Turing completeness to creep in.

I like his thinking here, constraining the software to something less than Turing complete so as to minimize complexity and maximize performance. I hope this approach succeeds as he anticipates.

nonethewiser• 3 years ago

Can anyone elaborate on how or why Turing completeness requires these sub optimal patterns?

I recall reading about avoiding Turing completeness for similar reasons to avoid the halting problem.

Other times these programmers apply the rule of least power—they deliberately use a computer language that is not quite fully Turing-complete. Frequently, these are languages that guarantee all subroutines finish, such as Coq.

https://en.m.wikipedia.org/wiki/Halting_problem

mlazos3• 3 years ago

It isn’t that there are suboptimal patterns, it’s just the more expressive your language can be at runtime, the less you can reason about statically. An example is data-dependent control flow. If you can’t reason about what branch your code is going to take statically (without your runtime data) it is harder to generate fast code for it.

mlazos3• 3 years ago

This 95% of models are statically computable thing really shows how much he is trivializing this problem. I’d be interested to see his SW stack compile MaskRCNN. His ISA is massively under-defined and people will not change their model code to run on this accelerator unless his performance beats cuda significantly and even then they still won’t - usability matters more than performance every time. In the end you need a compiler, and it needs to be compatible with an existing framework which is not trivial at all, since they are written in python.

sanxiyn6• 3 years ago

I agree writing something that needs to be compatible with, say, PyTorch is a significant undertaking, but why is that necessary? I also agree some models like MaskRCNN is not static, and people will not change their model code, but I don't think it matters.

Let's say you want to run LLaMA. LLaMA is a tiny amount of code, say, 300 lines. LLaMA is static. It doesn't matter people will implement LLaMA with PyTorch and not tinygrad, geohot can port LLaMA to tinygrad himself. In fact, he already did, it's in tinygrad repository.

What I am saying is while running all models ever invented is harder than running LLaMA and Stable Diffusion (Stable Diffusion port is also in tinygrad repository), that's not necessarily trivializing the problem. It is noticing that you don't need to solve the full problem, there is enough demand for solving the trivial subset.

While developers will choose usability, users will choose cheap price. If they can run what they want on cheaper hardware, they will. I already have seen this happening: people don't buy NVIDIA to run Leela Chess Zero, they just run it on their hardware. It doesn't matter everyone working on LC0 model is using NVIDIA, that's irrelevant to users. LC0 model is fixed and tiny, people already ported the model to OpenCL, OpenCL port is performant, it runs well on AMD. The same will happen to text and image generation models.

ipsum23• 3 years ago

I have some experience in this area, having both worked on machine learning frameworks, trained large models on datacenters, and have my own personal machine for tinkering around with.

This makes very little sense. Even if he was able to achieve his goals, consumer GPU hardware is bounded by network and memory, so it's a bad target to optimize. Fast device-to-device communication is only available on datacenter GPUs, and is essential for models training like LLaMA, Stable Diffusion, etc. Amdahl's law strikes again.

sanxiyn6• 3 years ago

Eh... no? Stable Diffusion works fine on single device. Ditto for smaller LLaMA.

ipsum23• 3 years ago

That's for inference, I'm referring to training.

Lapsa2• 3 years ago

I find it peculiar. just recently folks were tryharding to make everything and your kitchen sink Turing complete and now it creeps in menacingly on its own

Lapsa2• 3 years ago

also, I like the spirit of: "I don’t want to live in a world of closed AI running in a cloud you’ve never seen". shake those shills a bit

smasher1642• 3 years ago

Turing-completeness != un-optimizable! Literally the areas of type systems and compilers exist to serve this endeavor. It's gotta be a meme at this point every time someone brings up the halting problem or rice's theorem.

bmacho• 3 years ago

CS education considered harmful!

redox992• 3 years ago

I don't think anybody claims it's unoptimizable. It's just a harder task, compared to a more constrained system.

smasher1642• 3 years ago

Right, but the type system is the constraint. Nobody's gonna take the untyped lambda calculus and run it on an accelerator. Even something like turing-completeness can be a type annotation, for example, the totality effect provided by languages like Koka.

fancyfredbot• 3 years ago

AMD only enabled their ROCm stack on consumer cards last month. This finally corrects a huge mistake - Nvidia made cuda available on all their cards for free from the start and made it easy/cheap for people to get started. Of course once they'd started they stuck with it... I hope it's not too late to turn this around.

russ• 3 years ago

Hope George pulls this off. Just sent this to my dad, who turned down starting ATI with the Lau family on account of me being born. =)

willzhang100• 3 years ago

Where does the figure that the human brain is 20 pflops come from?

neilv• 3 years ago

I don't know whether it's a factor in the alleged software quality issues he mentions, but it's not unusual for a company that thinks of itself as a hardware company to neither understand nor respect software enough.

Even if adopting "hardware/software co-design", leadership might be hardware people, and they might not understand that there's tons more to systems software engineering than the class they had in school or the Web&app development that their 5 year-old can do. That misunderstanding can exhibit in product concepts, resource allocation, scheduling, decisions on technical arguments, etc.

(Granted, the stereotypical software techbro image in popular culture probably doesn't help the respect situation.)

sidcool2• 3 years ago

George was in the list of my favourite programmers till a few months ago. Now there are Jeff Dean, John Carmack and Karpathy

Tepix3• 3 years ago

When you click on the strip link to preorder the tinybox, it is advertised as a box running LLaMA 65B FP16 for $15000.

To be fair, the previous page has a bit more details on the hardware.

I can run LLaMA 65B GPTQ4b on my $2300 PC (built from used parts, 128GB RAM, Dual RTX 3090 @ PCIe 4.0x8 + NVLink), and according to the GPTQ paper(§) the quality of the model will not suffer much at all by the quantization.

Just saying, open source is squeezing an amazing amount of LLM goodness out of commodity hardware.

(§) https://arxiv.org/abs/2210.17323

coolspot• 3 years ago

What case/MB/GPUs do you use for your dual 3090 build? Liquid cooled cards?

Tepix3• 3 years ago

I'm using air cooling. Gigabyte X570 Pro, RTX 3090 FE, be quiet! Pure Base 500DX mesh case with four 140mm fans currently. It's not quiet under heavy load!

The GPUs have gotten improved thermal pads. The 8 core Ryzen 3700X is a 65W model. It appears to be fast enough not to be a bottleneck for this purpose. 1200W PSU.

I may swap the fan in front of the GPUs for a high-rpm model.

Also for longer runs i throttle the GPU power draw. It doesn't cost much performance.

magixx• 3 years ago

Are you able to memory pool two 3090s for 48gb and if so what's your setup?

I looked into this previously[1] but wasn't super confident it's possible or what hardware is required (2x x8 pcie and official SLI support?). AFAICT still would look like two GPUs to the system.

[1] https://discuss.pytorch.org/t/is-there-will-have-total-48g-m...

Tepix3• 3 years ago

You can memory pool with the right software however pytorch supports spreading large models over multiple GPUs OOTB. Just pass the --gpu-memory parameter with two values (one per GPU) to oobabooga's text-generation-webui for example.

paulus-saulus• 3 years ago

I reached the limit of free articles. impossible to browse and read the page

bgitarts• 3 years ago

Why isn't AMD and Intel working on an alternative?

wahnfrieden4• 3 years ago

geohot gave me some good advice/feedback from using my Japanese reading app, Manabi Reader: https://reader.manabi.io

I'm always grateful for his user feedback, it led to next-level improvements. Thanks geohot

The tiny corp raised $5.1M

Related Stories