Crossing the 1 tb/s throughput threshold for object storage with NVMe drives

https://blog.min.io/performance-at-scale-minio-pushes-past-1-3-terabits-per-second-with-256-nvme-drives/

by edogrider • 7 years ago

54 26 7 years ago

blog.min.io

lettergram• 7 years ago

The key is “drives”, writing simultaneously across all you can get high throughput.

BubRoss3• 7 years ago

This doesn't say what minio actually is. It also says the benchmark cost $1000 to run. Also since this is terabits it is about 120 gigabytes per second using 256 drives.

estsauver• 7 years ago

Minio is an s3 interface compatible storage api.

vxNsr• 7 years ago

whats the use case?

mtmail• 7 years ago

I looked up the pricing: Each i3en.24xlarge (96 vCPU, 768GB RAM, 8 x 7500 NVMe SSD) costs $10.848 per hour. For the test they rented 32, so around 350 USD/hour or around 250.000 USD/month.

dx034• 7 years ago

At this cost, is AWS really worth it? Shouldn't colocation be much cheaper after only a few months of use?

mrb• 7 years ago

Fabric-wise, PCIe 4.0 offers 16 Gbit/s per lane, so a single PCIe x16 device would have 256 Gbit/s of bandwidth available (in each direction), and a single server could reach 1 Tbit/s with four NVMe devices on four PCIe slots. There is the overhead of 128b-132b encoding to be considered, but that's minimal (3%). There is also the overhead of PCIe packets to consider, but if the NVMe device supports large maximum payload sizes (1kB+) it should be minimal.

Are there PCIe 4.0 x16 NVMe devices commercially available?

amarshall2• 7 years ago

It’s worth noting that M.2 & U.2 both have 4 lanes, so anything more would need to be HHHL or FHHL. Anyway, the Liqid LQD4500 has a 4.0 x16 interface and claims 24 GB/s. I don’t wanna know how much it costs, though.

https://www.liqid.com/products/composable-storage/element-lq...

https://www.servethehome.com/liqid-element-lqd4500-32tb-pcie...

Rafuino• 7 years ago

x16 NVMe SSDs aren't very common. The most recent one I've seen is the Micron X100, but that's a x16 PCIe 3.0 device and it's not commercially available yet. Most consumer and enterprise SSDs are x4, though perhaps that may change as better media and controllers are able to saturate PCIe capabilities. Then it's a tradeoff on power requirements though to be considered...

jordanthoms• 7 years ago

Asus motherboards (and probably others too) support a mode where you can split the x16 into 4 x4 connections and use https://www.asus.com/us/Motherboard-Accessories/HYPER-M-2-X1... to mount them.

Nextgrid• 7 years ago

My question is whether this includes overhead of actually persisting the data and guaranteeing a certain SLA in terms of retrieving the data? Don’t get me wrong, this is a nice achievement and good luck to them if their objective is to build a storage company (I or my company has no need to store large amounts of data at high speed so no bias there) but there’s a difference between ingesting data at high speed and actually guaranteeing it’s going to be there in a year’s time despite potential storage drive failures and stuff.

algo_trader• 7 years ago

Are there any architecture papers/blogs on re-designing software for very fast storage ?

Once storage is as-fast as memory, things change.

Please note: i am NOT referring to cpu-on-the-memory-chip architectures

JulianMorrison• 7 years ago

There's a lot of programs of late, designed to run in pure RAM. I imagine it would be similar? Except perhaps with different retention-over-restart characteristics.

the8472• 7 years ago

P2P-DMA is making inroads to the linux kernel. Which means software can limit itself to orchestrating transfers between storage, GPUs, accelerators and network cards and avoid having the data touch system RAM or the CPUs.

natmaka• 7 years ago

Confining the CPU to the most noble processing, and flanking him with highly-specialized units directly exchanging... isn't it the mainframe all over again?

jamessb• 7 years ago

For databases, there was a 2016 VLDB paper that suggested using 'write-behind logging' as an alternative to traditional 'write-ahead logging' that is "designed for a hybrid storage hierarchy with NVM and DRAM": http://www.vldb.org/pvldb/vol10/p337-arulraj.pdf

amarshall2• 7 years ago

Once storage is as-fast as memory

Bandwidth is only one aspect, SSD latency (even Optane) is still quite slow compared to RAM.

mjb• 7 years ago

Lots of work, in both academia and industry. For example, check out Daniel Bittman's work at UCSC, and Tim Kraska's at MIT.

Two interesting things are happening here: memory becoming non volatile, and storage and network bandwidth catching up with memory bandwidth.

Iwan-Zotow• 7 years ago

Once storage is as-fast as memory, things change.

"Money can buy bandwidth. Latency requires bribing God."

SomeHacker44• 7 years ago

Can someone please fix the title? I presume it means TB/s or Tb/s but I am not able to load the article now. Then delete my comment.

LawnboyMax• 7 years ago

It should be Tb/s

Crossing the 1 tb/s throughput threshold for object storage with NVMe drives

Related Stories