HN user
Working on software projects
https://huggingface.co/models is usually good place to look, you can sort by trending and filter by the task you care about (ex: Image-Text-to-Text). The first page will usually have the leading edge/newer models.
Yeah, some people say they got replacements through the warranty. The problem is, this thing is really big and heavy, so boxing it up is a real pain, especially if you've had it a while and already threw out the original box.
Unfortunately those "solutions" don't work, the person who had a potential solution was able to at least go through the inputs, this is not the case here, you can't even go through the inputs.
I've tried all the potential solutions this morning. It seems permanent unless Samsung somehow finds some magic to fix it, especially since the soundbar won't connect to WiFi/internet and doesn't do anything with the USB plugged in.
Their consumer cards are the entry point for many researchers and students though, so it pays off eventually when they become engineers working with the expensive enterprise cards.
It's not just the home insurance either. Last week, I bought a car and 3 of the big insurance companies refused to insure it without a 15 days waiting/underwriting period (Geico which I've had for many years, State Farm and Progressive). It was pretty surprising, it seems like they are trying to stop offering insurance without explicitly getting out of the state or something. Thankfully, AAA gave immediate coverage.
You'll need something like EPYC/Xeon CPUs and motherboards which not only have many more PCIe lanes, but also allow bifurcation. Once you have that, you can get bifurcated risers and have many GPUs. And these risers use normal cables not the typical gamer pcie risers which are pretty hard to arrange. You won't get this for just $200 though.
For the chassis, you could try a 4U rosewill like this: https://www.youtube.com/watch?v=ypn0jRHTsrQ, not sure if 6 3090s would fit though. You're probably better off getting a mining chassis, it's easier to setup and cool down, also cheaper, unless you plan on putting them in a server rack.
Yeah, it's really just trying to get as much compute as possible as cheaply as possible interconnected in a reasonably fast way with low latency. Slow networking would be a bottleneck and expensive high end networking would defeat the purpose of staying cheap.
I meant each card is limited to 240w, instead of the usual 450w. Also, it's more like 4 circuits after all, because the main cpu/mb/2gpus are on a 15a too.
Never got around to writing some public docs. It's essentially bunch of GPUs on custom aluminum extrusion frames sitting in a server rack, connected to romed8-2t motherboard through pcie splitters.
Power limited to 240w, negligible performance loss while halving energy usage, uses 3 20a circuits.
Performance can range anywhere from 2x4090=1xa100 to 4x4090=1xa100 depending on models, etc.
It's great value for the money, and very easy to resell as well.
Really depends on the model and the software tricks you're using. With DDP and gradient accumulation, you can reduce the bandwidth bottleneck by quite a bit. We've trained with 4090s running at x4 lanes with very small impact. And running at x4 means you can stuff up to 26-28GPUs on a single cpu node (say epyc) and get PCIe latency and get rid of networking hassle.
Local workstation is much cheaper in the long run.
Even ignoring that, most of the development is running experiments. You're gonna be hesitant to run lots of experiments if they each cost money whereas when you pay upfront for the hardware, you're gonna have the incentive to fully utilize it with lots of experiments.
I'd go with rtx 4090 and deal with memory limitation through software tricks. It's an underrated card that's as performant as cards that are magnitude pricier. It's great way to get started with that budget.
I've been seeing a bunch of them in Bay Area, I thought it was already launched and started deliveries. In person, it looks like something out of a movie set.
The release event is in like ~30 minutes on their discord, probably the announcement went out a bit early.
1. You can pair normal atx PSUs for the motherboard/CPU and server PSUs for the GPUs using breakout boards.
2. You can power limit GPUs down to 250W and barely lose any performance depending on your use case, highly recommend it. So any PSU that can provide those is good.
3. HP 1200w power supplies are both plenty and cheap on ebay, even though they are rated at 1200w, because they are so cheap, you're better of just running them at ~500w and buy multiple of it instead of overheating a single one. A nice benefit of running them at lower wattages is the very loud tiny fan doesn't have to spin as hard and create a ton of noise.
4. Not needed, but having a single cable might be convenient, they are pretty expensive though.
5. You don't need to do anything special here, except if you add too many GPUs, the motherboard might have issues booting because the 75w per gpu draw is too much, but usually those motherboards will have an extra GPU power cable (like the ROMED8-2T) and some risers let you hook up the power cable directly to them so PCIe is only used for data transfer.
6. It's not the outlet, it's the circuit that matters. And keep in mind that whatever power wattage you set on the GPU, you need to account for ac/dc loss, so you need to add an additional ~10-15% to the usage.
If you power limit it to 250W, each additional GPU is essentially an extra ~280W or so. If you plan on having like 8 GPUs or more and you plan to run them 24/7, you're better off just calling a local colocation center and run it there, since they have much cheaper electricity cost, it comes out cheaper for you and you have all the benefits of being in a datacenter.
My guess is this isn't their competitive edge, network effects, products, data and distribution is.
In a way, it takes away their competitors edge while racing to the bottom to compete with open source. At the same time, they establish themselves as experts and keep attracting great talent that wants to publish their work openly. And it benefits all of us, so good marketing amongst developers too.
I think less effort has gone into image editing compared to image generation so far. That said, we're building some photo realistic image editing tools at https://www.faceshape.com, focused on face editing for now. Current models don't perform as well, but next generation currently under training will.
I'm always curious to know what kind of AI image editing people are interested in, can you share what kind of edits you'd like to do? There's the usual edits like background removal or object removal, but those are more general tools that are getting incorporated into lots of apps natively (say Google Photos).
I wrote it from the perspective of a small startup (<10 people, bootstrapped or small funding). I think it's far cheaper and easier to build a nice competitive mobile app/saas than to build a really useful model.
But yes I agree, it will be very competitive with much smaller margins.
I've tried it, sure it's good, but not even close to the real thing. But yes it's getting cheaper through better hardware, better data and better architectures. Also it builds on Facebook's models that were trained for months on thousands of A100 GPUs.
It's easier for large rich companies with infrastructure and datasets. It's very hard for small startups to build useful real world models from scratch, so you see most people building on top of SD and APIs, but that limits what you can build, for example it's very hard to build realistic photo editing on top of stable diffusion.
Yeah I agree it's too slow for autocompletion at the moment, but this would be for full feature implementations, not just autocomplete. For example, if I have a repo I want to add a table and rest api implementation in, it can do this: https://imgur.com/a/mIJvaJr (ignore the formatting errors in the UI, somehow parts of it show up in as code and others not, but api wouldn't have this issue, especially since you can use the system message to enforce output format).
I'm happy to wait even 30-60 seconds for this which I can easily evaluate, criticize (and the model will correct it) and then proceed to just patch and move on. I think the results from this will be much better with the 32k model, but remains to be seen.
This is awesome, can't wait to get api access to the 32k token model. Rather than this approach of just converting the whole repo to a text file, what I'm thinking is, you can let the model decide the most relevant files.
The initial prompt would be, "person wants to do x, here are the file list of this repo: ...., give me a list of files that you'd want to edit, create or delete" -> take the list, try to fit the contents of them into 32k tokens and re-prompt with "user is trying to achieve x, here's the most relevant files with their contents:..., give me a git commit in the style of git patch/diff output". From playing around with it today, I think this approach would work rather well and can be like a huge step up from AI line autocompletion.
It has no post-2021 knowledge, but while playing with it, I found that you can just paste the documentation (no need to even format it) and it'll just "learn" it. For example, safetensors wasn't available back then apparently, I just copied the docs into it and was able to get it write pretty good pytorch code that incorporates safetensors.
Have you looked into cloudflare tunnels/ngrok? You can expose any service to internet without dealing with ip addresses, firewalls or opening ports, etc.
And tailscale can also give you a private access to any resources even if it's behind firewalls and all that (say you want to access the gpu cluster from your laptop at home).
And they can all run in user space without elevated privileges.
L5 used to be the terminal level, at least for most engineers. L8+ is very rare in my experience.
Also: Almost everyone using GPUs for work will join NVIDIA's Inception program and get rebates... So why look at retail prices?
Out of curiosity, does that also apply for consumer grade GPUs?
The article mentions the 8 bit quantization, I believe this is about training in fp8 as native format. The latest GPUs provide huge flops for those, Tim Dettmers updated his gpu article and he talks about this, the claim is 0.66 PFLOPS for an RTX 4090.
I think it's a combination of many things. Very powerful compute, very large datasets, transformers and diffusion models and a ton of research by lots of researchers easily available on arxiv, etc.
From my experience playing around with dreambooth in the last few weeks generating images of a specific person or pet (not just a generic concept), it surprisingly works really well. But you have to make sure to feed it enough pictures, make sure to label the images properly, use smaller learning rate, use prior preservation loss and make sure to not overfit, etc.
For the animation stuff where you need frame to frame consistency, the new diffusion based video models show that it's possible [1][2]. These are not open source yet as far I know, but it's highly likely that we'll get them within a few months.
Yes, right now you have 3 options:
- dreambooth, ~15-20 minutes finetuning but generally generates high quality and diverse outputs if trained properly,
- textual inversion, you essentially find a new "word" in the embedding space that describes the object/person, this can generate good results, but generally less effective than dreambooth,
- LORA finetuning[1], similar to dreambooth, but you're essentially finetuning the weight deltas to achieve the look, faster than dreambooth, much smaller output.
Developing in container environments with limited access might help, but I think there's a performance hit for heavy processing/ML training unless you use privileged mode which kinda defeats the purpose.