ttsiodras

Just chiming in - the claims above are real, I have very similar numbers in a cluster of 2x GX10 I have access to.

Instructions to reproduce, and benchmarks here: https://forums.developer.nvidia.com/t/deepseek-v4-flash-offi...

I hear you on the insane amount of time vllm takes to launch (atlas is a move in the right direction in that regard).

But mostly I wanted to raise awareness to readers of your article that no, if you want to do inference, paying 15K for a single 96GB card almost certainly makes no sense. Buy 4 GX10s with the same money, and enjoy dramatically better models and user scalability.

Regardless - thanks for putting the effort to share your findings! I keep postponing doing the same... there's tons of things everyone is re-discovering on their own.

Interesting article.

IMHO, the author could have done two things better:

- vllm instead of llama.cpp. With NVIDIA HW, there is huge difference in multi-user loads and caching with vllm; when he was complaining about what happens when more than one user uses the model, and about losing caching, I was "well, duh".

- The budget he used for a single card could have instead be put to far, far better use with SPARKs. I have access to a cluster of 2 x GX10 - total cost less than half what he paid, even today - and I am running vllm and Deepseek v4 Flash. The difference compared to any Qwen is tremendous - I've NEVER seen it loop, and in all my experiments so far, it's the most Sonnet-y model I've ever tried (antirez seems to agree, hence his ds4 fork).

If you're wondering about how I set it up in the 2 GX10s: https://forums.developer.nvidia.com/t/deepseek-v4-flash-offi...

Performance: 2K t/s prefill ( very useful for feeding tons of source code into its massive context window ) and around 50-60 tg/s in my coding sessions in the pi.dev harness. With the money the author paid, he could have bought 4 GX10s, and double both numbers ( vllm basically scales almost linearly with tensor parallelism ).

Thanks, Mark - for both setups!

It's very nice seeing it put to use in actual Spectrum machines - love it :-)

Oh wow! I never expected to see this in a real Speccy until I fix the one I have back home (I'm an expat).

Thank you so much! :-)

Thank you!

I have been using xterm's default dabbrev-expand to do the same via "Alt-/" ( https://github.com/ttsiodras/dotfiles/blob/master/.Xresource... ) - which works regardless of what shell you're in.

But I was curious for your approach... so I asked Claude to convert it to bash: https://claude.ai/public/artifacts/01a49347-1617-4afe-8476-0...

Works like a charm - pinned it to Ctrl-k, which was free in my setup. I guess I don't have to depend on XTerm for this any more :-)

Thanks!

As soon as I met Forth, I felt I had to hack my own (https://github.com/ttsiodras/MiniForth/) - and did so over a single week, two years ago. I targeted the Blue Pill and the original Arduino, but developed via cross-compilers so that I could test my code in the host.

I became so obsessed with the project that I was looking forward to tinkering with it after coming back from work every day; so it was hacked in 5 evenings and a weekend. It was that much fun, to build a Forth.

I highly recommend the process; I think the only other time I felt so enlightened was when I first met Lisp macros (https://www.thanassis.space/score4.html#lisp).

This is why I come to HN - brilliant ideas that I should have thought of before, but didn't!

I just changed my server, and uninstalled the - now truly useless - fail2ban. I use SSH keys of course, but without fail2ban my server's logs were constantly flooded with hacking attempts.

No longer - wireguard for the win. Thank you, chlorion!

signal-to-noise ratio: off the charts.

Many thanks, OP.

dkozel and your kind, whoever you are, wherever you are, and whatever you are doing right now, you are legend.

Both of you are :-)

Heart=warmed. Happy holidays to both of you!

Thanks for sharing! The slides are here: http://vger.kernel.org/~acme/prez/linux-plumbers-2022/

I was not aware of gsocket - thank you, very useful!

I wonder if using RLE encoding would not have been but vastly simpler and more compact...

No, mate - it wouldn't. Don't be confused by the "printf"-dump; the Python script processed it into pairs of (frequency,delay). That is, when you see...

    989 Hz @ 15209
    989 Hz @ 15213
    989 Hz @ 15218
    784 Hz @ 15222
    784 Hz @ 15226

    (989, 15222-15209), 
    (784,  ... -15222),

I wouldn't have gone to Huffman compression if I had a simpler choice.

:-)

Well, if the bird in question used an LZ77 followed by Huffman, he could compress all that wood/chuck stuff down to almost nothing. So he could chuck a lot :-D

Spot on! I planned initially to control the volume with a potentiometer - but abandoned the plan when I got to the "sounds good enough" :-)

Mostly, my mistake was I had the phone too far away - notice in the video that when I get it close, it sounds just fine.

Live and learn :-)

I played the game with them, when they were very young. They loved every minute of it - with their uncle performing impromptu dramatization-translations :-)

I am seriously considering playing it again with them this summer :-D

A USB tester - it just reports the voltage (5V) and the current. As you can see in the video, the current hovers around 10mA, spiking to 20mA when we use the speaker.

If you are OK with external USB drives, I did just that with my Atomic PI [1] (Atom-based, 35$ SBC, 3.5W idle, running Debian on its internal EMMC, with two external USB drivers in ZFS-mirror configuration).

[1] https://www.thanassis.space/atomicpi.html

Thank you! I had noticed the same problem, but was not aware of this excellent solution. I just applied it, and it works perfectly... and I removed the dozens of Haskell libraries that were there just for Pandoc.

For anyone else interested, the same solution applies to shellcheck - just install shellcheck-bin from AUR.

Was not aware of PixelExperience - thanks.

Amazing.

Direct link to shadertoy: https://www.shadertoy.com/view/ssBGRG

Youtube video with intermediate steps: https://www.youtube.com/watch?v=RJf2nVOkOb4&t=1s

Interesting interview - thanks John! Didn't know there was a UFS2 "phase" before ZFS... I wonder how much time those fscks took! :-)

Thank you for your kind words. The funny thing is that I am not a sysadmin - or maybe I am one in spirit :-)

After so many months, I had no idea what tools I was running in that command line... And therefore, the effect of any signal seemed far more dangerous to me, than extracting the actual command from my shell's memory.

The answer to that is simpler: Try doing the Ctrl-z/fg sequence in your bash with this one:

    while true ; do sleep 1 ; done

Simply put: C-z followed by fg is not bulletproof. Not to mention that I had no idea what I was running in there, and how any signal would impact it... So I wanted to find a safer way to dump what was already there, in my shell's memory.

Anyway, I hope you guys enjoyed reading this regardless :-)

The script was running in production, doing actual work. Stopping it - when you don't remember a damn thing about it and how it worked - was not an option I wanted to consider at the time.

Hi Rory. I am well aware of the flags you described - and indeed, they'd clean things up here. I am in fact tempted to go back and update the post - and either replace the shebang, or add the "early abort" flags.

But TBH, keeping that script (that I wrote in 60 seconds) nice and clean was the least of my worries... Making the damn toolchain work had far, far higher priority :-)

And after that, booting the LEON cores of course.

OP here - just a note, that "tverbeure" is the Tom I mention in my blog post.

None of this would have been possible without you, Tom! Thanks for everything.