HN user

Creamsicle47

6 karma
Posts0
Comments14
View on HN
No posts found.
Kimi K3, and what we can still learn from the pelican benchmark 5 days ago

You have to look at the size of each expert

Yes, this part is accurate. Expert density determines how much raw compute each hidden state gets.

The number of the experts tells you about the diversity of its skills.

Most people misunderstand this part. Counter-intuitively experts don't develop diverse skills, they instead balance compute during the forward pass, allowing models to increase their parameter count without the MLP layers exploding in memory + compute requirements.

0ThreadHN
Mesh LLM: distributed AI computing on iroh 11 days ago

Thanks for answering, that makes sense. Also - your setup seems like it could greatly benefit from speculative decoding. Have you guys given any thought to how that might work in this system?

P.s. for #2, you can probably do something like RAFT-styled interleaved computation. But this could get tricky unless you commit to a sharding scheme that makes it easier.

0ThreadHN
Mesh LLM: distributed AI computing on iroh 11 days ago

Hey, this is a super cool project. It's great to see a lot of the IPFS stuff resurfacing again.

A few questions:

1.) How does this handle privacy? If you're distributing compute this way then all actors in the compute graph will also know the sequence being computed.

2.) Any safeguards against malicious actors poisoning model activations?

0ThreadHN