gfysfm

This is really cool! It's interesting that you can "cheat" by knowing the historical situation in advance. But I guess there's no way around that - if the situations involved were hypothetical, it would be unsatisfying when you guessed "wrong". It'd be neat to see really obscure examples drawn from history.

Hi, I wrote the post! Also not a ML researcher, just an interested engineer, so I'm sure I got some things wrong.

MoE should be the better tradeoff for the local/single-user scenario since the downside of batching being harder / less efficient doesn't matter.

What I meant was that the single-user scenario is going to get dramatically worse throughput-per-GPU, because they're not able to reap the benefits of multi-user batching (unless they're somehow doing massively parallel inference requests, I suppose).

Is it really that the matrixes being multiplied are larger? My mental model is that the purpose of batching isn't to get larger input matrices. It's to move the bottleneck from memory bandwidth to compute.

As I understand it, you want larger input matrices in order to move the bottleneck from memory to compute: if you do no batching at all, your multiplications will be smaller (the weights will be the same, of course, but the next-token data you're multiplying with the weights will be 1xdim instead of batch-size x dim), so your GPUs will be under-utilized and your inference will spend more time doing memory operations and less time multiplying.

The post has no numbers on the time to first token for any of the three providers.

I probably should have hunted down specific numbers, but I think people who've played with DeepSeek and other models will notice that DeepSeek is noticeably more sluggish.

Hi, I wrote the post. Thank you!

That’s how it does work, but unfortunately denoising the last paragraph requires computing attention scores for every token in that paragraph, which requires checking those tokens against every token in the sequence. So it’s still much less cacheable than the equivalent autoregressive model.

It’s partially because my articles often sit in draft for a while. Then while editing I have another thought that spins off into a separate article, which then gets posted in quick succession.

I wrote the article. Thanks for the feedback!

Man, that sounds awful. Was there no source control/code review process at all? Just everyone making changes to the same set of files?

Yeah, I agree this post would have been better with a concrete example. It's hard to talk about a specific project though, since it comes down to describing in detail facts about a company's internal workings (often embarrassing facts). I couldn't figure out how to anonymize it sufficiently.

Nope! I threw this up in the time I had just to see what it'd look like. I think it'd be a lot of fun to try different formats or different prompts (e.g. eliciting CoT before making a decision).

This repo is sort of a companion piece to https://github.com/sgoedecke/fish-tank, which is a more physics-based 2d simulation. Interestingly, 4o-mini absolutely cleans up in the 2d space but doesn't win all the time in poker.

Great way to get your account banned!

The specific example here - pushing an organization to use your bespoke Python framework, that you believe to be the best solution for all Python programming tasks - does not inspire confidence.

Thanks! Mind if I quote you in the README?

Just woke up and it seems to be working now. No harm done, I guess?

Cool, thanks. I'll try this out if I ever want to switch to WebRTC.

Haha, no worries. I did.

As for udp connections, WebRTC looks like the only mature solution (but it's not designed for games and is way more complicated than game devs want it to be). Alternatively, I know about https://github.com/networkprotocol/netcode.io but it's very much in the early stages - afaik you even need a Chrome extension to use it on the client. It's not easy to do udp on the web.

That would work for me.

Wrt mining/ads, I'm considering to default to mining if I detect ad-blocker, or you're on Chrome + Desktop PC (not laptops).

Do what you've gotta do to get paid for your work, but if I found out this was happening it would strongly discourage me from remaining on the website.

Write game bots to test your load non-synthetically.

Can you expand a bit more on that? Cool tips, for sure. Personally I'd appreciate a flip-switch between ads and mining, as long as it defaulted to ads.

Good idea!

Why implausibly?

Whoops, nice catch.

Thanks for the catch!

Thanks everyone for giving this a go! I'll admit I indulged my "I want it to look like a terminal" urges at the expense of proper usability or mobile users. It's far from a finished product, as many of you noticed, but it's great that people were able to have some fun with it anyway.

It's worth noting that the only real criticism you level against Shakespeare here - that "shall I compare thee to a summer's day..." is incoherent - rests on an embarrassing misreading of the sonnet. Far from being a strike against Shakespeare, his point is that summer days are not temperate but hot! Likewise, Autumn is too short, May is too windy, and so on. The subject of the poem, Shakespeare claims, does not suffer any of these flaws. I think this serves as a useful yardstick for the quality of your other claims about Shakespeare.