HN user

grw_

376 karma

email: georgerw@gmail.com telegram: @cuffslide

Posts6
Comments67
View on HN

Ah right, yes- I think we're talking about the same thing- this driver just chooses to pretend to be a RoCE v2 device (instead of e.g MLX Nic in IB mode), but nothing would change if it did I think. Or at least thats what the libibverbs abstraction promises.

There's no IB OR Ethernet underneath- I could have implemented this properly as it's own distinct transport kind, but seemed easier just to pretend to be something that is already known.

The 'the chip that understands both TB and IB and translate RDMA requests between the two' in this instance is your CPU, so orders-of-magnitude worse latency than an ASIC, but still better than anything on top of IP/Ethernet. I think there's also potential to do device-initiiated RDMA, where e.g GPU itself can write to some mailbox and have message appear across the abstracted transport in another GPUs mailbox. Even if the CPU is involved in shuffling pointers across mailboxes it doesn't necessarily mean it'll be a bottleneck

Yeah, thunderbolt-net is IP on top and it does work as you say, with a few caveats:

- On a single cable with two rails available, the thunderbolt-net grabs one and uses that. Without patching the kernel, there's no way to make it present a second interface using the remaining pair.

- If you had a second cable between the machines (for 4 total rails), thunderbolt-net will still only grab one rail, because the abstraction across which it's making the links sees an identical peer at the end of both links and so falls into the same trap as above. There is no LRO/GRO anyway (or it's buggy- I forget) on the linux version.

- Why you only get 10G rather than 20G on single pair- actually, this might be something specific to the Strix Halo SoC that I was testing on- on a different (still AMD) chipset and an Apple TB5 Mac I did see closer to 22G in one direction, but still 8 in the other. The Strix Halo NHI seems to be 'stripped down' (as expected, for mobile) in ways I don't really understand.

- Intuition on why- I can't point you to the line number, but I think it has to do with a fixed 4kb page size when communicating with the NHI that ends up becoming a bottleneck, perhaps 16kb pages on aarch64 apple help here?

I actually didn't know there was more to InfiniBand than verbs (at least at this abstraction level, above PHY), so probably the answer is 'not much more'. The device imitates a RoCE V2 device and the higher level abstractions I used on top were GPU-ish libraries like NCCL and JACCL.

Good q about 'bridging into actual InfiniBand', I don't know the answer there either. My naive understanding would be that: since this is host-initiated RDMA (it's still the host cpu invoking into dma buffers, though they may be device-memory mapped), actually it should work fine, at least between two machines? I'm curious enough to try- I have a couple of machines with thunderbolt AND RoCE-capable NICs- the experiment is to see if we can use this across diverse transports simultaneously? I think this is what it does already (since the MacOS FA57 vs linux native are already 'different transports'), but say if you have a better scenario to demonstrate what 'bridging into actual infiniband' would look like!

A USB4 40Gbps cable consists of two 20G tx/rx pairs. The in-kernel networking implementation is single-stream and just uses one pair, and won't e.g. stripe across both pairs or across multiple cables, which was the main bandwidth unlock in TFA. Doing so would be a much more complicated undertaking, since now you've re-introduced out-of-order delivery which complicates re-assembly of large packets, retries, handling loss etc. The verbs interface is a lot simpler than that of a full IP stack, so although was possible to get this working across rails, may not be so simple for something pretending to be ethernet.

author here! it's not on top of USB4NET, no (RXE can already do that, it's compared in the benchmarks). it's built with the same tb primitives as the networking stack obviously, just assembled differently to emulate a verbs device instead of a nic. happy to answer any other q!

I don't think Threadripper platform is to blame that you bought a board with potentially the worst possible pcie lane routing. Latest generation has 88 usable lanes at minimum, most boards have 4x 16x, and Pro supports 7x Gen 5.0 x16 links, an absolutely insane amount of IO. "At the mercy of motherboard design"- do the absolute minimum amount of research and pick any other board?

Interesting, thanks. Whilst ".2G Twice a week, for 6 months" doesn't strike me as a lot- could be someone who just goes clubbing twice a week- apparently enough to cause brain lesions. Anyone know if the study is reputable, or could answer the question whether the "lesions" could be something like regions of the brain responsible for/exacerbating mental issues which have been altered by the treatment- ie, the method of action for treatment.

My comment wasn't implying that importance of wind in the UK energy mix (or lack of investment in nuclear) is specifically the reason for high costs but I agree that government policy is indeed to blame.

As to whether the cause is Tory ban of onshore wind- I'm skeptical. I don't think wind is any cheaper than gas once you include the necessary energy storage capacity- especially when there is huge amounts of gas sitting under the north sea which the new Labour government is planning to ban pumping: https://www.forbes.com/sites/gauravsharma/2024/07/12/sudden-...

That's great- but UK has almost highest energy cost in Europe- almost €0.48/kWh. By comparison France (which has consistently invested in nuclear for the past half-century) has almost half that. Until UK has enough energy that it can be priced as in other European nations it will suffer with low competitiveness of energy-intensive industries.

IPMI 2 years ago

yes, it does-

lspci | grep ASP c2:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 52)

Thermal sensors are a case in point. Today, one that costs about $1,000 and weighs as little as five sachets of sugar

Seriously, we're using _sachets of sugar_ to convey weight?

Later in the article:

Two decades ago, he adds, a less sensitive thermal sensor weighing a kilogram cost ten times as much

And:

A poncho called Noa lite, developed by Fibrotex, an Israeli firm that supplies America’s army, weighs less than 700g

Why do journalists do this?

Weird, I had one of these too and it crashed constantly- I would reach into my pocket and if the phone felt hot I'd know without looking that it had crashed and had already drained the battery

I don't think it's fair to say it's exclusively used for money laundering, it surely can be used to obscure the origin of funds but I think 'money laundering' label requires that the motive be financial profit. If you just want to preserve privacy while interacting with ethereum network without financial gain, tornado is a great tool to accomplish that.

My guess is that it's a similar social phenomenon to flat-earthers, qanon, etc, in that a group of like-minded people form and customs emerge in which making ever-more outlandish claims are taken as stronger commitment signal to the group. I was reading last week about an 'audiophile ssd' product and I browsed an audiofile forum thread[0], expecting to have a chuckle at users ridiculing it but there's actually 6 pages of people praising it and finding new adjectives to describe the sound it produces. The most popular posts [1] also detail other absurd things they have done to produce a better sound- "I can opt to convert a FLAC to WAV format and play that from RAM, this somehow lowers the perceived noise flow and makes music sound more organic and detailed"

[0] https://audiophilestyle.com/forums/topic/62753-nvme-ssd-desi... [1] https://audiophilestyle.com/forums/topic/62753-nvme-ssd-desi...

applications can be stateful but not require persistence, mostly real-time applications like game servers, VOIP, etc.

For houses it's completely plausible that the motivation is for the asset to appreciate in value and even if the marriage ends in divorce, both parties end up being able to extract some value from it. For a retail diamond the purchaser is likely to see zero value recovered from it whether divorce happens or not.

For cars, I suppose my hypothesis would say cars likely to depreciate very quickly (such as high-end SUV) are more suitable as engagement gifts than practical (prius or such), which fits roughly with my observations in real world

The economics behind diamonds are better explained by an sociologist, not a geologist- the high cost and useless-ness of the gift are a feature not a bug! The burning of significant amount of wealth is a costly signal of commitment to the receiver. I heard from a friend who worked at a diamond company (and as such could purchase stones with significant discount to market price) that his fiancée had specifically rejected the idea of receiving a stone from his company on the grounds that it being 'discounted' devalued the gesture.

It only _directly_ affects the poor- regardless of whether I as can afford these things, it benefits me to live in a society where everyone (including "the poor") is educated to a high standard, is not living in fear of financial distress due to minor illness, and is able to pursue their vocations out of choice rather than necessity