Ultra fast Thunderbolt NAS with Apple M1 and Linux

https://chrisbergeron.com/2021/07/25/Ultra-fast-Thunderbolt-NAS-with-Apple-M1-and-Linux/

by cyberge99 • 5 years ago

220 99 5 years ago

cyberge996• 5 years ago

The maturity of Thunderbolt hardware and software support should have TB on a short list of high speed local connectivity, high-bandwidth multi-disk access options.

CharlesW4• 5 years ago

Especially now that it's USB 4.

cyberge996• 5 years ago

They're very similar, but there are differences. Most notably, Thunderbolt 4 guarantees a minimum speed[0] for compliance, whereas USB4 has a lower minimum.

Also, I believe USB4 was designed for smaller files / blocks and Thunderbolt is designed for more raw output.

[0] https://liliputing.com/2020/11/differences-between-thunderbo...

reacharavindh4• 5 years ago

Not to take away anything from the fun of figuring out Thunderbolt on Linux, and such. But, I wonder if a NAS with a 40 Gbps or even 100Gbps NIC hooked up to a Thunderbolt to Ethernet adapter would achieve a better result without extra effort.

minhazm• 5 years ago

The fastest thunderbolt to ethernet adapter I could find is 25 Gbps and it's $1500. 10 Gbps ones are in the $200+ range as well.

osmarks• 5 years ago

Buy an eGPU enclosure and put a NIC in that?

lostlogin• 5 years ago

For 10Gbe it’s slightly cheaper if you go to SFP+, but then you need sfp cables. I went this route as they are also cooler. Then I got sick of Thunderbolt finicky config and went for cheap Mellanox PCI cards which are truely excellent and basically zero config.

The dongle is QNAP QNA-T310G1S Thunderbolt 3 to 10GbE Adaptor, US$169 on Amazon.

reacharavindh4• 5 years ago

True. I assumed it was a simple adaptor for TB3 to 40Gbps Ethernet or atleast QSFP+ Since TB also advertises 40 Gbps, but looks like expensive niche. At $1500, you might as well spend time to figure out how to get Linux to work better with Thunderbolt card.

For other devices in the network that do not impose Thunderbolt though, Melanox 40G NICs are surprisingly cheap.

jasonjackson• 5 years ago

probably, but thunderbolt would cost 5x less for equivalent 40Gbps NIC

schmookeeg• 5 years ago

This has not been my experience -- I use both thunderbolt 3 and Mellanox ConnectX-3 40gbps cards, and have been looking for an excuse to add a Mini M1 to the homelab.

The cables and cards for PC Thunderbolt retrofit are far (like 3-5X!) more expensive than the MLNX cards and optical cables.

$0.02 :)

kllrnohj• 5 years ago

You can find tons of used 10-40Gbps NICs on ebay for $50 or less. For a simple point-to-point connection with both sides having PCI-E slots that's going to be easier & cheaper to set up than thunderbolt will be. It'll also be more reliable & more practical since getting 40Gbps over Thunderbolt requires extremely short cables, whereas you can get DAC cables for 40Gbps QSFP or Infiniband up to 7M or so. And there's no confusion around what the cable can actually do, unlike the nightmare that is finding the right USB-C cable with the right signal integrity requirements at the right distance for what you actually need/want.

xiconfjs• 5 years ago

this blog uses webp as image format without a fallback option. Had to switch to Firefox because Safari wouldn't display the pictures.

mpweiher• 5 years ago

Odd. Displays fine on my Safari 14.1.2. Checked and the images do appear to be web for me as well.

Current Big Sur release, not a Beta.

definitelyhuman• 5 years ago

Safari 14.1.2 on latest Catalina doesn't render

stalfosknight• 5 years ago

Works fine in the latest Safari.

cyberge996• 5 years ago

Thanks for pointing this out. I’ll add a fallback using the “picture” element (TIL).

I also use some javascript to delay rendering until the browser window stays on an image. That could very well be introducing some wonkiness.

interactivecode• 5 years ago

you should use the built in lazy loading https://caniuse.com/loading-lazy-attr or intersection observer for this https://developer.mozilla.org/en-US/docs/Web/API/Intersectio...

jeffbee2• 5 years ago

Where by "ultra fast" it is meant that it is much slower than a directly-attached SSD that occupies the same port.

reacharavindh4• 5 years ago

Local storage and Networked storage always have their places. Local solid state storage almost always gets you better performance because you negate the overhead of network. Unless, you’re talking fancy arrays connected to saturate 100G NICs.

Local storage falls apart the moment you need that storage accessible from another machine…

jeffbee2• 5 years ago

This doesn't say anything about how such storage would be accessed by the mac and some other clients at the same time. It literally just runs iperf over tb. As a matter of architecture it leaves the question of whether it would be preferable to directly attach the storage to the Mac and export it to the rest of the network from there.

systemvoltage• 5 years ago

Dumb question: Can you use Mac's shared folder for another machine to access files?

SilverRed4• 5 years ago

Networked storage also falls apart the moment you need anything better than basic file storage. So many filesystem features do not work networked, most network file mounts do not support any kind of modern security and the ones that do are often super slow.

Everything has its place, but I wasted so much time trying to get a nas to work as storage for my home server.

dylan6042• 5 years ago

That SSD has 0 redundancy so you're trading the purpose of the NAS by going for speed. If that's what you want, then you're wasting time looking at NAS equipment anyways.

So what's your point?

CoolGuySteve• 5 years ago

Most SSDs have a lot of redundancy in the form of multiple flash chips, it's just not exposed to you, the user, by the controller chipset.

There are also PCI-E x16 with multiple NVMe slots. There's nothing stopping some company from making something similar for Thunderbolt.

And I guess the cost is too high and the connectors would be too bulky, but it would be nice to have something like a SODIMM for flash that separates the memory from the controller.

renewiltord• 5 years ago

Well, yes, that is implied by the fact that it says “NAS” in the title.

In general, you can usually do faster local than over the network. People use a NAS for other reasons.

whalesalad3• 5 years ago

Just yesterday I was thinking how badly I want an obscenely fast local datastore for Docker. I am managing a handful of clients right now who are all using Docker containers for their microservice environments and my local machine runs out of disk daily.

What I really want is lightning quick network storage, but I don't think it would be feasible to roll out 10g networking in my current rental home.

devonkim2• 5 years ago

10gbe can be done over RJ45 with a direct cable connection and is something I've done before in a pinch between my personal NAS and a client machine (just be careful about your storage protocol because NFS has its pros and cons just like iSCSI and FCoE, and with macOS as a client you're likely in for a Bad Time like I did).

vitejose• 5 years ago

Out of curiosity, what issues did you run into with NFS on macOS? I'm looking into building a NAS for a network of macOS, Linux, and BSD machines and figured NFS would be my first choice.

cbushko2• 5 years ago

Would running something like the docker proxy below on an old laptop help?

  TLDR: A caching proxy for Docker; allows centralized management of (multiple) registries and their authentication; caches images from any registry. Caches the potentially huge blob/layer requests (for bandwidth/time savings), and optionally caches manifest requests ("pulls") to avoid rate-limiting.

https://github.com/rpardini/docker-registry-proxy

whalesalad3• 5 years ago

At some point you need to build images locally or store images locally to run a local docker-compose cluster.

I need a fat block store for that.

kilroy123• 5 years ago

Just curious what size is your hard drive?

I had this pain with my last macbook. I ALWAYS ran out of disk space. Never again! I went with a 1 TB drive for my M1.

dijit• 5 years ago

I offload docker to another machine and I'm still using about 400GiB of storage on my relatively new MacBook; and I basically never use this machine outside of work (and even then I didn't use it for like 4 months because I preferred linux). it has no local media or anything.

I'm shocked by how much disk I've used actually.

https://imgur.com/a/iHV4IKn

biztos• 5 years ago

I got the 2TB drive so I can run out of even more disk space! ;-)

yawniek• 5 years ago

why don't you simply use one or many nvme thunderbolt/usb disks and move the caches there?

ehutch79• 5 years ago

Not the op, but if the nas was pre-existing, and already serving other network clients, no reason to add a second one.

cyberge996• 5 years ago

Author here: I actually used a thunderbolt nvme and adapter for a bit. Then, for kicks I got another one and combined them in a raid-0 on macos. Realizing I didn’t want to run nvme’s, in raid-0 over two cables in any sort of production (homelab - I use production loosely here), is what sent me down the Thunderbolt rabbit hole that precipitated this blog post.

FullyFunctional4• 5 years ago

Not OP, but my files are shared with many hosts. Having local files are a PITA to manage coherent state.

pr0nin• 5 years ago

What about separate Linux box laying around just to run docker? 8GB M1 Mac just don't have enough RAM for docker VM.

Looks like local docker server should just work exactly same way as local docker while you at same location.

whalesalad3• 5 years ago

Then you get into the problem of low latency directory sync between your box and that host. I’m still trying to crack that nut.

pronoiac• 5 years ago

In this post I discuss how you can upgrade a NAS Server by adding Thunderbolt 3 for lightning fast connectivity at 20 or 40Gbps.

This is an interesting angle of attack, though I'll note that 10Gbps Ethernet was added as an option on the M1 Mac Minis.

MayeulC• 5 years ago

here's an idea: use link aggregation yo make it 10 G faster, then.

I wonder what's the simplest option. could it be made to work with a virtual interface, like wireguard?

smaccona• 5 years ago

10Gbps was already an option on the Intel Mac Minis. My guess is that Apple envisioned a use case for it to be a file server or similar.

boynamedsue• 5 years ago

I just wish Apple would offer a native iSCSI initiator.

CharlesW4• 5 years ago

It sure would be convenient if they Sherlocked the developers of commercial solutions. The globalSAN initiator's been working well for me, FWIW.

mbreese• 5 years ago

globalSAN initiator's been working well for me

It doesn't look like it's going to be supported for M1 Macs or is really all that compatible with Big Sur. From their site, it seems that you have to jump through a few hoops to add a kernel extension to get it to work[1].

Sounds like a good opportunity for Apple to roll something into the OS.

[1] https://support.studionetworksolutions.com/hc/en-us/articles...

insaneirish• 5 years ago

There's something weird here:

  root@nas-04:~ # ifconfig thunderbolt0 10.1.1.2

  root@nas-04:~ # ifconfig thunderbolt0
  thunderbolt0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
          inet 192.168.10.2  netmask 255.0.0.0  broadcast 10.255.255.255

The IP on the interface is different than the one that it was theoretically configured with. My suspicion is that perhaps the IP used was changed at some point and this is just an editing slip.

Now that the interface has an IP, we have to add a static host route entry for it to communicate:

If both sides are configured with an appropriate netmask on the same subnet, there's no need for a specific host route.

cyberge996• 5 years ago

Author here: You’re correct, it’s an editing error. There’s another where I do ifconfig on the mini, but I pasted from the nas. I’ll correct the post.

FullyFunctional4• 5 years ago

Funny I went down this rathole very shortly after discovering the "Thunderbolt networking" change in the Linux kernel. My hope was the use the Linux box (with 10 GbE) as a bridge/switch to give my Mac cheap 10 GbE access. While point-to-point did hit more or less the expected speed, bridging was horribly slow. Ended up just getting an Sonnet Solo10G SFP+ adapter (which is great).

21com• 5 years ago

bridging was horribly slow.

Can you quantify that? (I'm wondering if it might be good-enough for my needs.)

FullyFunctional4• 5 years ago

I'm afraid it was 2+ years ago so I don't remember, but it must have been slower than the comparable speed from the gigabit wired connection. FWIW, I was so happy with the Sonnet (not as hot as some alternatives I've used) that I bought two more. However, unless you are using fiber or for other reasons preferring SFP+, going with the grain and getting 10Gbase-T is probably easier.

Getting back to the OP, this could be interesting for me if and only if I can get fiber based Thunderbolt-4, but it seems to not yet be available.

Teknoman1172• 5 years ago

This is cool, but my main takeaway is that you can get around needing the thunderbolt enablement header by shorting two pins together and use it on any motherboard?

What the hell Intel...

unicornfinder• 5 years ago

If memory serves the main problem is some Thunderbolt cards don't ship with the necessary firmware on them, so would need to be used at least once in a supported motherboard before just used in any system.

zhengyi13• 5 years ago

That seems even more WTF, and puts me in mind of the old Winmodem setup. Not precisely analogous, but boy it seems weird.

rsync• 5 years ago

"What the hell Intel... "

I don't understand any of it ...

If I wanted to add a LPT parallel port to my motherboard, I would find an appropriate add-in card with the physical port and plug it into the PCI bus and locate an appropriate driver.

Thunderbolt is just another kind of port, right ? If you buy the add-in card to get the physical port and you have a driver to drive the card ... what in the world is "enablement" and why would it be required ?

Teknoman1172• 5 years ago

The enablement header was a pretty what the hell thing to being with, I totally agree. I always saw it as them trying to avoid the development of thunderbolt add in cards for some reason.

But to find out is does so little that shorting two pins together gets around it is just too funny.

conradev• 5 years ago

I was actually looking for something like this. I have one computer that I want to back up to both local hard drives and to the cloud.

A NAS is great, because you can back up to it and have the backup stored on local hard drives, but it also has a CPU and it can be responsible for uploading the data to the cloud. This is better to do from a dedicated device, especially if you are uploading large amounts of data with limited bandwidth, which I am.

The constraint is that locally backing up to a NAS is slow, even though it is sitting next to my computer. Instead of buying into 10GbE networking, it’d be nice to just use a Thunderbolt cable.

wil4213• 5 years ago

Running FreeNAS on a 1GbE network and I wouldn’t call it slow at all. The initial back is much much faster than my 1TB personal media library I sometimes backup to OneDrive. I think I have 3TB of video from Usenet that is downloaded onto a windows box, uses a 10 year old intel as an unpack scratch pad, and then is stored on the NAS box. Sure the initial library transfer is slow but if you can use a wire it’s not that slow. Certainly not Comcast 25Mb upload slow.

Agree about 10Gb devices being expensive especially if you’re invested in the UniFi ecosystem. I’m tempted to do a direct 10Gb SFP+ connection using used intel or Mellanox 10Gb NICs but I haven’t pulled the trigger. 10Gb Thunderbolt NICs are too expensive and give off questionable amounts of heat.

SFP+ is the way to go right now and gives you options. Fiber, DAC (copper), or Ethernet.

Fire-Dragon-DoL• 5 years ago

I have a freenas too. Uploading to it gives me 100 mb write speed, which is definitely enough for most of my needs.

lizknope2• 5 years ago

I got 2 used SFP+ cards and a DAC cable for $60 total on eBay. 10G between my file server and primary desktop is great. 1G for everything is fine.

brian_herman• 5 years ago

Does anyone know if I installed ubuntu on a m1 apple mac would I get pci passthrough to gpu. That would be no right because that isnt possible due to the hardware. I was going to do the ubuntu conversion myself but I am running a developer beta and the commands don't work and if I wait longer it might just be in the next LTS or release of ubuntu by the time Monterey is stable.

lxe• 5 years ago

As NVMe disks and interfaces have gotten faster I’ve found myself saturating a 10Gb ethernet network.

I love this race between network speeds, drive speeds, and RAM speeds. The classic "net vs disk vs ram vs cache vs register" interview question is much trickier these days.

SilverRed4• 5 years ago

I recently moved to somewhere with a real fiber internet connection and was impressed that I could get up to a gigabit speeds but when I tested it I found my wifi network could only handle about 100Mbit and sometimes it was dropping to 50. First time I have ever had wifi actually be the limiting factor. Thankfully my apartment has wired Ethernet to most rooms so my steam downloads are flying along now.

genewitch4• 5 years ago

I think most people will find that they'll get <100mbit over wifi. I had to design, buy, and implement 802.11ac stuff to get close to gigabit. And then wifi6/802.11ax came out and nothing wanted to play with Linux and wifi6. So I got a fiber media converter and put a switch on my desk backhauled over fiber to the "house switch" which is in turn backhauled to the server barn switch via fiber. Doing this prevents lightning from blowing up my wifi access points and NAS boxes, which has happened before, necessitating the need for fast WiFi in the first place.

solarkraft• 5 years ago

This is cool, I’ve been thinking about something like it.

However I’m still slightly disappointed because I expected an article describing an M1 Mac Mini as a NAS, running Linux (this dream probably isn’t technically there just yet, as Thunderbolt [and network?] drivers are so far still missing from the Asahi project).

ballenf• 5 years ago

I wonder if there will be "reality" tv shows about data hoarders one day. Probably have to wait for data to be easily visualized with VR or AR.

In fact, I really love the idea that I could have to walk to a particular room in order to view a particular set of old photos. Wearing some AR glasses, of course. And that I could grant view access to visitors. Would suck if the photos accidentally glitched through the floor and I had to dig down through a foot of concrete to get them back though.

johnwalkr• 5 years ago

This year I set up 10Gbps networking in my home lab since I am lucky enough to have 10Gbps internet and it didn’t make sense to pay for that but not have it for all my machines.

Or at least I tried to set it up. 10G pcie cards are now cheap enough, but routers are not. Thunderbolt adapters are also expensive, bulky and use a ton of power[1]. I’m keeping my eye open for a surplus router and cheaper adapters but in the meantime I am only using a 2 port card in my main machine to serve at 10Gbps to one other machine. I wonder if I could also serve my m1 MacBook using thunderbolt 3.

[1] In my work life I also tried to use Ethernet in some embedded projects. It turns out Ethernet is just power hungry in general. Even 100Mbps Ethernet uses about 0.5W per port. There are standards for lower power but I’ve yet to see anyone use them.

bpye• 5 years ago

Have you considered using an x86 machine with dual 10G NICs as your router, and then all you’d need is a 10G switch which may be cheaper than a router. I’ve considered the approach but haven’t undertaken it yet.

Ultra fast Thunderbolt NAS with Apple M1 and Linux

Related Stories