Mini-ITX Seaberry Adds 11 PCIe Slots to a Raspberry Pi

https://www.jeffgeerling.com/blog/2021/mini-itx-seaberry-adds-11-pcie-slots-raspberry-pi

by geerlingguy • 5 years ago

111 70 5 years ago

h2odragon3• 5 years ago

We're like one or two hardware generations away from having a VME backplane like environment where we'll be able to do some truly nifty and nasty things, at home and hobby levels.

The politics of getting a standard out that isn't owned by (and therefore twisted to favor) a couple of big companies is challenging, but it's happened before, right? (crickets)

tjoff• 5 years ago

VME?

h2odragon3• 5 years ago

https://en.wikipedia.org/wiki/VMEbus

myself248• 5 years ago

https://en.wikipedia.org/wiki/VMEbus

Imagine something like ISA but 32 bits wide, with a GPIB-like arbitration scheme. On a backplane that you plug CPU cards and peripheral cards into.

dragontamer3• 5 years ago

I'm pretty sure PCIe is that today.

PCIe CPUs may have failed commercially, but they prove that its possible (Xeon Phi).

PCIe GPUs are common. 4x or 8x GPUs are your typical mining rigs.

Perhaps most common is PCIe SSDs (aka: NVMe or M.2 slots).

------

So our modern PCIe "backplane" gives us storage, extra compute (GPUs most commonly), longer-distance network access (fiber optics, ethernet, etc. etc.). What else do you want?

Sound cards (especially MIDI instruments support) maybe? Dedicated video processors? A big problem with dedicated devices is that modern CPUs are so powerful... there's not much point to buying an ElGato H264 encoder anymore when you can just buy a 16-core CPU instead and do it all in software.

----

Your standard Threadripper motherboard has 4x16 PCIe slots, and I'm pretty sure 8x PCIe to each of the slots, with full support of bifurcation (splitting into smaller PCIe slots). Surely that's enough for any of your hobby projects?

h2odragon3• 5 years ago

What got me off about VME when i looked at it was the immense amount of small market, single purpose hardware that had been thrown on it and was "technically" compatible. It enabled some stuff like timed traffic lights in a way similar to how Arduinos have opened up hobby level "printer-oid" mechanisms.

For reasons i cant articulate well I'm itching for a bus where CPU and GPU and RAM and IO all live together in hippie harmony. The hardware's all there and dancing to some little basic machine controller CPU anyway so lets open all that up and make everything a first class citizen, hopefully on some interconnect that will last a few years and gather some interesting surplus hardware to filter out to hackers.

As to modern hardware, if i had all my dreams coming true, I'd be grabbing one of the "mining" motherboards with all that I/O broken out into channels and putting SDR's on them to make pretty waterfalls of the biggest bites of the radio spectrum i could visualize.

geerlingguy4• 5 years ago

The tl;dr is it's a standard mini ITX-sized motherboard for the Raspberry Pi Compute Module 4. It has a Broadcom PCIe switch that it uses to power 4 mini PCIe slots, 4 M.2 E-key slots, 1 M.2 M-key slot, a x16 slot in the standard ITX location, and an edge x1 slot. It also has dual/triple-redundant 12v power supplies (barrel, aux +/- ATX, and PoE++ header, though it doesn't use a 24-pin ATX connector, so you need to power it with an adapter or via wall wart (I used a 96W barrel plug adapter).

Through the course of testing this thing, I also identified a bug in the PWM fan controller circuit—and apparently it's also a small bug present on the CM4 IO Board's implementation, but typically not a fatal one.

otachack• 5 years ago

Always enjoy the blog/video posts you do, Jeff. Thanks for creating this awesome, in-depth content!

_joel• 5 years ago

Seconded, I'm envious of your new NAS though :)

mashc5• 5 years ago

Do you have any more info on the PWM bug? We’re having some issues with PWM on the RPi4 and I’m wondering if it’s related…

ChristianGeek• 5 years ago

Once you get to this size and price, there’s no point in using a Pi…there are much more powerful options available.

wongarsu• 5 years ago

I guess it makes sense as a development board: if you are building a product around a Pi Compute Module you can use this board for quick prototyping before building your own product-specific board. But any hobbyist looking at this should seriously consider the various Pi alternatives around the $200 mark first. Or just buy a used PC.

geerlingguy4• 5 years ago

This is spot on. I mention in the post and the video it's an expensive board, and very specialist.

The target audience is basically 'person who wants ARM64 PCI Express development board', and even there, some of the PCI Express implementation on the Pi's BCM2711 is... weird. Therefore some devices have had a hard time getting drivers going on the Pi (case in point: graphics cards and Coral TPUs I've tested). I plan on testing Radxa's CM3 and Pine64's SOQuartz too.

I think some people still want a cheap(ish) CM4 motherboard that just has one x16 slot, front panel headers, and maybe a 24-pin ATX power supply connector. I think that could be made for under $100 and would also solve a lot of needs for debugging purposes.

mikepurvis• 5 years ago

For the sub-$100 price point, you often can't beat discarded thin clients. There's a steady of stream of units always hitting eBay, including ARM ones like the HP T-series and x86 ones like Dell Wyse units.

If you really just need something that comes with a case and power supply, might already have wifi, and has a few USB2 ports to plug things into, these are great options.

jeroenhd• 5 years ago

I can see someone making a very powerful open source WiFi router out of this. The Pi can run passively cooled unlike most Intel or AMD CPUs. You wouldn't be doing any DPI on the network traffic without slowing things down, but this thing with some networking hardware cards could become very interesting to many people.

The cost is too high because it's a niche product, of course, but such a board would be able to replace my home router/server quite capably with a good WiFi antenna and some ethernet ports.

timc3• 5 years ago

There are other platforms that are maybe better suited to that role though. I am not very up to date with the hardware but I remember PCengines being one.

NovemberWhiskey3• 5 years ago

How many networking cards? I know we tend to think of internal buses as infinitely-fast, but a single PCIe 2.x lane is saturated at 500MB/s, which begins to look a little skinny once we're considering 10GE or even Wifi 6.

opencl• 5 years ago

A 4 lane PCIe switch would allow for a much cheaper board and still be plenty to saturate the available bandwidth. Maybe 2x 2.5Gbe with one hooked to a switch, plus wifi.

Around this price point there are already ARM boards (and passively cooled x86 machines) available much better suited to this kind of use case, like the Honeycomb LX2K.

rektide2• 5 years ago

Alas PCIe switches were starting to get affordable before the great chipmaker consolidation of 2015-2019 ("and seemingly overnight the cost of these switches increased three-fold"[1]). Now most every PCIe switch is >$6/lane. Even just the 12 lanes here, on an old PCIe generation, is going to be a $72 chip, perhaps more (article: PEX8619 Broadcom PCIe switch, $129, 16 lanes, $8/lane, no stock. Compare versus Microchip nee PMC-Sierra Switchtec such as PM40028B1 at $6/lane[2]).

It's a cool idea. But wow, for $435, sure feels one ought consider going for an x86 pc. The form factor is cool though.

It feels insane to hope for but I'm really hoping that by the time 2025 rolls around we see some modest affordable arm chips that have Thunderbolt built in. Perhaps even multiple Thunderbolt. If nothing else, there better be some decent PCIe available by then- 8x PCIe 5.0 or better. ARM needs to start doing bandwidth, some day (in places other than server class chips); it's been on USB3.0 for a long long time. Thunderbolt + PCIe would be a great boon for general usability & connectivity. Now that Intel has made it royalty free, now that USB4 already incorporates most of the hard part (packet based switching), it might finally be the time for some real PCIe on chip.

[1] https://www.anandtech.com/show/15821/microchips-new-pcie-40-...

[2] https://www.mouser.com/c/?marcom=100722352

MisterTea2• 5 years ago

If nothing else, there better be some decent PCIe available by then- 8x PCIe 5.0 or better. ARM needs to start doing bandwidth, some day (in places other than server class chips); it's been on USB3.0 for a long long time.

Its not up to Arm to attach PCIe lanes but the chip makers who have been competing with x86 which already offer plenty of PCIe lanes. So far, only apple has bothered to make an Arm CPU which can compete with "PC" hardware. Hopefully AMD will offer something similar.

Thunderbolt + PCIe would be a great boon for general usability & connectivity. Now that Intel has made it royalty free, now that USB4 already incorporates most of the hard part (packet based switching), it might finally be the time for some real PCIe on chip.

USB4 is based on Thunderbolt 3. Given that Thunderbolt 3 is a mess, I don't have high hopes.

Look at RapidIO or 1394 for examples of better thought out interconnects and protocols. PCIe cant connect CPU's together in a NUMA system whereas RapiIO can. RapidIO also uses Ethernet physical layers so there's parallel development with Ethernet. I honestly don't like USB or contraptions invented by Intel that are invariably consumer grade with various extensions to satisfy other industries, bolted on.

NovemberWhiskey3• 5 years ago

So far, only apple has bothered to make an Arm CPU which can compete with "PC" hardware.

Also Amazon: https://aws.amazon.com/ec2/graviton/

rektide2• 5 years ago

i thought it was pretty clear i meant the consumer arm ecosystem in general but thanks for chiming in & clarifying. yup.

usb4's ability to transport a variety of protocols across the switched network seems perfect. reviews like the Plugable tb4 hub show video and data intermixing nicely[1]. the complexity seems scary but in my mind it feels utterly & completely justified, and it builds atop the world we have & makes it better, quietly fixes the underpinnings, rather than insisting on re-invention. thunderbolt hasn't been flawless but it's been a niche product, a luxury product, and we'll quickly gain the experience to make it as pleasant bulletproof & flawless as one might hope for. already, reviews like the one i cited don't come with caveats or speed bumps: they show a working, satisfying consumer experience. there's a lot of afraidness of thunderbolt & usb4, but i for one think we ought to dial down our resistances & afraidness.

that said, i also think the door is totally open to new interconnects emerging. i'd love to see cxl or gen-z (recently folded into cxl) or opencapi or rapidio or whatever become consumerized.

[1] https://www.anandtech.com/show/16964/plugable-tbt4hub3c-thun...

scottlamb• 5 years ago

It feels insane to hope for but I'm really hoping that by the time 2025 rolls around we see some modest affordable arm chips that have Thunderbolt built in. Perhaps even multiple Thunderbolt. If nothing else, there better be some decent PCIe available by then- 8x PCIe 5.0 or better.

In the shorter term, I'm hoping the RK3588 actually comes out, is as good as promised, and has a reasonable price. [1] 4 lanes of PCIe 3.0 + 3 lanes of PCIe 2.0 + 2x USB 3.1 + dual gigabit Ethernet total at well over 10X the bandwidth of the Pi's single PCIe 2.0 lane. Also things I care about for my NVR project: faster video decoding, built-in TPU, and SATA (which I didn't count above because it's multiplexed with the PCIe 2.0).

I don't think the 4x Cortex-A76 + 4x Cortex-A55 is in the same league as the Apple M1, but still quite a bit faster than the Pi4's 4x Cortex-A72.

[1] https://www.cnx-software.com/2020/11/26/rockchip-rk3588-spec...

CameronNemo3• 5 years ago

I don't think the 4x Cortex-A76 + 4x Cortex-A55 is in the same league as the Apple M1, but still quite a bit faster than the Pi4's 4x Cortex-A72.

IMO the node used is more important than the core design. Rockchip claims they will be using an 8nm node for the rk3588. And I think the rk3399 and Pi4 are using 28nm nodes right now. So if this chip materializes it could be much faster (compared to Linux oriented ARM boards of today).

bloopernova2• 5 years ago

Please forgive my lack of familiarity with PCI express hardware, but I have a question:

All those PCI express devices are all sharing the single v2 lane? Is the PCI express bus a network?

geerlingguy4• 5 years ago

Indeed, PCI Express can be thought of in many ways like Ethernet; you can have switches just like Ethernet, though in almost all cases, PCIe is a local area bus (local to one server/machine), whereas Ethernet is still the king of networks.

NovemberWhiskey3• 5 years ago

That's probably the way to think about it. PCIe devices communicate point-to-point over dual-simplex serial connections. If you have something like a Raspberry Pi CM4, which only has a single PCIe lane, then there's going to be a PCIe switch.

bloopernova2• 5 years ago

Interesting, thank you!

dragontamer3• 5 years ago

All I/O is a network these days.

* USB is a network.

* PCIe is a network.

* SATA is a network.

* Infinity Fabric / Hypertransport (AMD socket-to-socket communications) is a network.

* UltraPath Interconnect (Intel socket-to-socket communications) is a network.

-----------

People think their internet connection is Layer4 (TCP) -> Layer3 (IP) -> Layer2 (Ethernet) -> Layer1 (Wires), but that's wrong.

Its really Layer4 (TCP) -> Layer3 (IP) -> Layer2 (Ethernet) -> Layer 3 (PCIe) -> Layer 2 (PCIe frames) -> Layer 1 (PCIe wires) -> reinterpret PCIe on the Ethernet-controller -> Layer1 (Ethernet wires).

There's a bunch of "networks" in-between our networks that we use. That's why OSI model exists, to understand each different network and each of the different rules behind them all.

When a good network is abstracted away, you can ignore all the details.

EDIT: It might not even be Layer2(Ethernet) -> Layer3 (PCIe)... on a NUMA-system it could be Layer2(Ethernet) -> Layer2 (NUMA / MESI messages to CPU#2) -> Layer2 (Southbridge communications) -> Layer3(PCIe)...

agumonkey• 5 years ago

network all the way down

-- Alan Kay maybe

btw, slight sidestep HDMI CEC is also a bus/network, and even SCART was too.

colejohnson66• 5 years ago

One of the goals of PCIe when they were designing it was to avoid the “bus” architecture classic PCI had as that forced every device on the bus to run at the speed of the slowest device. PCIe, OTOH, takes a “star” network topology (more a "tree" though) with point to point connections. Thanks to the use of switches, each point can run at the maximum speed they support (limited by the switches speed).

Basically, it’s like USB (as @dragontamer said). Just as you can turn a single USB port into 7 more downstream ports through a USB switch, you can do the same with a PCIe switch.

zw1234562• 5 years ago

Some friends and co-workers of mine have an ongoing bet/joke... Who can find something that the Raspberry Pi can NOT do.

One more thing scratched off the list!!

542458• 5 years ago

Boot without a GPU - Pi can’t do that :)

zw1234562• 5 years ago

Good one. Putting it on the list.

It's an amazing little board, list is pretty short :)

sandreas2• 5 years ago

Things that are on my wishlist:

- Compute-Module should have a unified interface, that is not about to change for the next 3 generations of compute modules

- Someone create a "notebook" case having a board with mount for exactly this interface

- Case should have a decent keyboard and display without being to big

- How about having GPIO pins hidden

I would love to have an "upgradable" Notebook with arm architecture...

nine_k• 5 years ago

If you want a fast laptop, your two priorities are, IMHO, the thermal design and making as much room as possible for the battery.

I'm afraid that RPi level of performance in a laptop is going to be underwhelming, though still interesting for many cases.

If there was a widely accepted standard for laptop main boards, like there is for desktop and server boards, that would be wonderful. I don't expect it to ever happen though.

cjbconnor• 5 years ago

Have you seen the MNT Reform? There's a pi compute module adapter being built for it.

sandreas2• 5 years ago

Yeah, quite bulky :-) But interesting approach.

tambourine_man2• 5 years ago

I have no experience building PCs, but during the last decade or so I've been running RPis with 6-8 disks attached. It's great software and noise wise, but the cases were made by me with MDF (and I have no idea what I'm doing), so it's not as solid as I'd like them to be.

Every time I search for rackmount or 3D printed cases I'm floored by the prices, which seem completely out of proportion.

Any tips on cheep/custom computer cases are highly appreciated. Thanks

rblatz• 5 years ago

It’s probably easier to buy a regular case and then fit the raspberry pi into it.

hinkley2• 5 years ago

We should be meeting in the middle and creating SBCs whose bolt holes spacing is identical to some other device standard. For instance 2.5” HDDs, or 3.5-2.5 adapters. Then you can find trays and enclosures and caddies that can be modified slightly to purpose.

Expanding holes in sheet metal is easy. Tapping holes, welding captive nuts, and bending are a lot trickier.

imachine1980_• 5 years ago

its most a computer case(sound of drums) of production of scales the only ones who are mass produce are the ultra basic, the others are design for enthusiastic hobbyist who can pay a premium for their hobby.

jrockway2• 5 years ago

I agree with the advice that you should just use an ATX case. They are a commodity and there are a billion great ones. You'll have to remove the motherboard plate and drill 4 holes for M2.5 standoffs; the full Raspberry Pi dimensions are on the product page, so you can easily print out a 1:1 template, get the right drill bit, and go. No fancy equipment is necessary to drill a small hole in sheet metal, and the result will be very satisfying.

I tend to 3D print my own stuff, and have found it to be the be all and end all of electronics projects enclosures. I never had a good project enclosure until I got a 3D printer, and now I always have them. It's amazing! I have at least 3 different Pi projects, and I designed and printed cases for each of them. It's very straightforward and not a bad project for learning CAD and 3D printing.

8 hard drives are going to be heavy and take up a bunch of space, though, so that's why I'd recommend an off-the-shelf case that already has support for that. (Sheet metal bending is the right fabrication technique for this application, I think, and that's how the cases are made.)

tambourine_man2• 5 years ago

That's what I was afraid, you're convincing me of buying a 3D printer :)

They are still a bit expensive, but are so much fun that they maybe worth it.

dekhn• 5 years ago

I absolutely love laser-cut acrylic with tapped holes accepting m3 screws or standoffs.

For the MDF, it's mostly practice and time/care. It took me quite some time to master making square wooden boxes that stand up to abuse and nowadays I could design in Fusion and rip out a case on my CNC machine pretty fast (<hours).

Next project is to encase an entire RPi in clear epoxy: https://www.raspberrypi.com/news/epoxy-pi-resin-io/

marcodiego2• 5 years ago

Recent 4-cores, 2.0+ Ghz and 4GB RAM ARM SBC's are plenty powerful for most tasks a home user may need. Media consumption, browsing, light text editing, editing the weekend video and even compiling small code bases. People have been doing that in chromebooks for years.

Unless you need to compile large code bases, edit serious videos, process huge amounts of data or CAD/games, a simple cheap ARM computer should be enough.

Too bad the desktop was taken by a monopoly and monoculture of x86-64 with windows where you are locked if you need compatibility with some proprietary formats or tools/software.

alias_neo• 5 years ago

I wonder if there is a blog post in this somewhere, that I could write.

I have a bunch of Pi 4s (4GB). I just put one in an Argon M.2 case and was wondering what to do with it.

I have a "gut feel" for how my workstation should behave, I am a (backend) software engineer by profession, tend to work my day job on an i7 laptop (XPS 15, Ubuntu) but at home I'm on Ryzen 7 1800X (Manjaro) and Ryzen 9 3900X (Debian).

Is there some useful non-benchmark type "measurements" I could take by using a Pi as a workstation for a few days, that people would be interested to read about?

marcodiego2• 5 years ago

You can try subjective benchmarks. These ARM SBC's are as powerful as workstations from the 90's, so try to use them with as demanding software as what was run on a workstation from the 90's.

Trying to replicate this would certainly do a great blog post: https://news.ycombinator.com/item?id=27796821

3np2• 5 years ago

One significant bottleneck in a common desktop Rpi would be the SD card. You can improve a lot by putting the root filesystem on SATA-over-USB or use the CM with an appropriate board.

If you'd have been in the market for a new board, I would have recommended something that exposes PCIe, or at least SATA.

The best non-SoM option < USD 100 for that is probably RockPi 4 (but please fill in). You can fit it with an M2 NVMe (with limited lanes) or multiple SATA drives. RK3399 or RK3399pro have been holding that segment for some time now. Sadly RK3399 has max 4GB RAM. Armbian works great.

There is also Quartz64 and Rock 3A (max 8GB RAM, PCIe3). But it's going to take a while until community and mainline kernel support make the RK3566/RK3568 ready for a casual user.

You're going to have to be conscious about multitasking. Having 100 browser tabs open is not going to fly. And while 4k may be theoretically possible, you don't want to do that. Other than that I think it's all right.

-----

I have a similar profile to yourself. Currently drafting a small cyberdeck - maybe more as a replacement for smartphone than PC use.

lproven• 5 years ago

If you want to experience something really radically different from the rest of 21st century computing, try RISC OS on one of them.

https://www.riscosdev.com/projects/

RISC OS is the original native OS for the ARM chip. It's still alive, actively maintained and developed, and it's FOSS now. It's a 1980s OS that predates NeXTstep and Windows 3, let alone Linux, which means a unique (but very effective and elegant) GUI, and a shell, that are totally unlike anything else you've ever seen.

It's a 35 year old OS. Don't expect miracles. It doesn't support wifi or multiple cores, but it's very very fast and responsive on a single core of a RasPi, and there is a big software library with hundreds of freeware titles and some commercial stuff too.

ed25519FUUU• 5 years ago

Does this make building a large NAS with an RPI doable?

fnord77• 5 years ago

sorry if this is obvious, but how many CM4s can you plug into this board?

exabrial• 5 years ago

Is there a storage controller like a SAN that can do provisioned NVMe?

Something like you insert a RPI into a backplane, and it auto-provisions some storage and makes it available to the RPI, in addition to providing power/ethernet.

FpUser• 5 years ago

I am curious if by the time one finishes assembling all this stuff for Pi based IoT appliance the total cost (never mind time) will exceed one of some used laptop / small factor PC which can do the same thing but faster.

aasasd• 5 years ago

Curious how the author has a decent site, but on Youtube he automatically switches to the trashy clickbait tricks.

Mini-ITX Seaberry Adds 11 PCIe Slots to a Raspberry Pi

Related Stories