HN user

intorio

35 karma
Posts0
Comments12
View on HN
No posts found.

Fair use only applies in certain contexts, of which writing commercial software is not one of. 'Snippets' are not fair use and it is shocking how many people here think they are.

At least for a commodity chip like Broadcom Tomahawk (100G), the latency is 500ns with L3 enabled, and 300ns if only L2 is enabled. Compared to the Mellanox SB7700 at 90ns ethernet has some catching up to do if latency is the end goal.

Ethernet tooling for HPC has a ways to go, but I suspect in the future it will be more competitive. Especially if specialty fabric vendors cut down on R&D.

CLOS fabric designs seem to be winning the war these days which I think favors Ethernet in the long run. Better flow distribution on aggregate links and now widespread support for MC-LAG means you can build a really wide CLOS network with L2-only.

Maybe I'm just bad at negotiating, but I haven't been able to get vendors to quote me less than double the cost of the Cumulus hardware quotes I've seen (primarily Quanta). Also, looking at support costs, the quotes I have for Cumulus are about 20% cheaper than the cheapest quotes I've seen from Juniper/Dell, just for software support.

That said, NBD or sooner hardware replacement plans are a bit weird with Cumulus. I think they expect you to keep onsite spares, which still ends up being cheaper.

I think if Cumulus becomes more popular, we might see some community code that implements validation and confirmed commits and more features. If we decide to go with Cumulus, and we are thinking about it, we might write something like that.

Are you running the newest firmware on the PERC? The firmware used to limit you to only Dell certified drives but they removed that after about a year, probably due to backlash. I know we told our rep we wouldn't buy the new cards until they got rid of that.