Snabb Switch – A toolkit for solving novel problems in networking

https://github.com/SnabbCo/snabbswitch/wiki

by justincormack • 12 years ago

137 62 12 years ago

dclusin• 12 years ago

This pretty common in the financial trading space. It's usually referred to as kernel bypass. It's how HFT's are able to achieve < 100 microsecond tick to trade response times.

mikemoka• 12 years ago

this is a very interesting remark, it would be nice to find some more explorations of this concept applied to current web servers

chton10• 12 years ago

Indeed. I'd also like to know if there are any alternatives that offer similar performance in an OSS wrapper.

LinuXY• 12 years ago

There are a lot of parallels in serving hundreds of thousands of small HTTP requests a second and HFT. This, and the ability to instrument the onboard FPGA to mitigate against DDoS attacks is one of the reasons I speced SolarFlare for CloudFlare. http://blog.cloudflare.com/a-tour-inside-cloudflares-latest-...

fasteo• 12 years ago

The title is a bit misleading. This nice product is basically a programmatic wireshark in Lua, that is, a packet processor, so you are getting "40 million packets per second".

Once you do some meaningful work (say, HTTP protocol decoding), this figure will be a lot lower.

jvermillard• 12 years ago

For HTTP/TCP it's not helping much but it could be very interesting for the IoT protocols like CoAP (RFC7252) which is UDP based.

Anyway I suppose the main target of this project is to help developers of packet switching and load balancing software.

chton10• 12 years ago

If it's easy enough to use, it might be interesting for other too. To give an example: my own project is a message processing system as a service, intended for (among others) the massive amounts of data that come from IoT gateways or devices. While we're not quite there yet, we do intend to be able to handle loads that would require this kind of performance. In some applications, millions of messages per second aren't out of the ordinary.

If this kind of library can help us do that, that could save us a lot of time and effort. Time we could use to work on the next bottleneck :)

jsnell• 12 years ago

This is a very strange article, since the quoted text appears to have nothing at all to do with Snabb except for linking to the website. Instead it's describing some other closed source system that's using a similar networking layer, and that's also written in Lua.

dang2• 12 years ago

Yes, and the quote was from HN itself, which doesn't make for a new HN post. We changed the url to that of the project.

chatmasta• 12 years ago

Probably because the direct link was already submitted a month ago.

haddr• 12 years ago

Does it wrap around some sort of userspace TCP stack? I see that performance measures are really matching these kind of systems...

justincormack7• 12 years ago

There was some work done, see https://groups.google.com/forum/#!msg/snabb-devel/2yF5LZ-VS1... - its not the primary focus but I am likely to do some work on it later.

wingerlang• 12 years ago

As usual when there is a word I recognise, I go looking for a Swedish reference in the docs/founders/etc. And sure enough the guy is currently a Swedish citizen.

Snabb means fast/quick in Swedish.

sanityinc• 12 years ago

The author, Luke Gorrie, is a true hacker's hacker who built a good chunk of Emacs' Slime environment for Common Lisp. He ended up in Sweden to be part of the first successful Erlang-based business Bluetail, which got sold to Nortel IIRC.

(I imagine "Snabb" is closely related to "snappy" in English.)

signa11• 12 years ago

and for erlang folks on emacs, distel is god sent...

personZ4• 12 years ago

The disparity between what the OS can do and what the hardware is capable of delivering is off by a few orders of magnitude right now. It's downright ridiculous how much performance we're giving up for supposed "convenience" today.

What are they talking about?

Goopplesoft• 12 years ago

Yea, Im not sure thats a good/accurate bashing. This is a highly optimized tool for a very specific task. OSes have very very wide use cases they have to support.

fulafel• 12 years ago

It's a good and accurate bashing, irrespective of what this tool can do. Even more so once you remember that the networking equivalent of Moore's law tapered off about 10 years ago.

justincormack7• 12 years ago

The operating system APIs (sockets) basically cannot delivery to/from userspace anything like what the hardware can do. The classic example is 10Gb ethernet, where if you want to do something like packet switching, you cannot do anything like line rate, due to cost of context switches, memory copies, scheduling, cache misses in the stack layers and so on.

Given that servers are getting 10Gb, 40Gb or more, this is becoming more and more of an issue. The hardware is capable of it, but the abstractions are getting in the way.

personZ4• 12 years ago

There is an OS overhead for sure (although you can mitigate that with things like Zero Copy solutions), but it generally imposes a couple of percentage points of overhead. I was thrown off by the several orders of magnitude claim.

Reading it again, it sounds more holistic -- that a simplified, monolithic application with a local, lightweight database is much higher performance than, for instance, an n-tier, distributed database type solution.

lukego2• 12 years ago

Snabb NFV is the main software being built on the Snabb Switch base right now: http://snabb.co/nfv.html

nnx• 12 years ago

How does this compare to netmap?

http://info.iet.unipi.it/~luigi/netmap/

netmap / VALE is a framework for high speed packet I/O.

Implemented as a kernel module for FreeBSD and Linux enabling memory mapped access to network devices.

wmf• 12 years ago

Netmap provides an efficient syscall API and mostly reuses existing kernel drivers. DPDK and Snabb bypass the kernel and just take control of the NIC from userspace.

deadgrey194• 12 years ago

On of the serious drawbacks of Netmap is that it amortizes, but does not mitigate system call latencies. This means that although it can handle high packet rates, latencies for processing are dominated by the system call overhead (typically about 5us). This may not sound like much, but on a 10Gb/s network (1 bit = 0.1ns), 5us = 50,000bits or roughly 5KB. For small packets, this means that you have to batch very aggressively (about 150 packets at a time) in order to keep up.

bajsejohannes• 12 years ago

I watched the short introduction video [1], where they say that to get the fastest throughput, you need to pull up everything from the kernel and into user space. This is what they are doing with the network drivers. Won't you still need to talk to the kernel when you actually want to read/write from the network? Or is part of this that you take complete control of the NIC?

[1] Look for snabb switch here: https://cast.switch.ch/vod/channels/2i5k459xe3

justincormack7• 12 years ago

No you talk to the NIC directly, not via the OS.

There is also a podcast here http://blog.ipspace.net/2014/06/snabb-switch-and-nfv-on-open...

corysama• 12 years ago

(Disclaimer: All I know of Snabb is from reading the web site and a few discussion group posts)

Because Snabb is specialized around a few specific NICs and virtual IO interfaces with specific features, it is able to do things like set up the network hardware to be memory mapped. As in, after some setup, Snabb can bang on the bits of a specific memory range and that will change bits in the network card state without the OS or even the driver being involved. This means you are not calling send() or select(), you are dealing directly with the Intel NIC hardware interface.

It sounds a lot more like game console engine programming than node.js programming. As a game console engine programmer, that's pretty interesting to me :)

anarcticpuffin• 12 years ago

I'd like to know how the 26ns wire-to-wire latency was measured. As far as I know, just handling the PHY layer on NIC takes at least ~300ns. Likely the authors are inferring latency by using 1/<throughput>, which is mistaken at these levels.

deadgrey194• 12 years ago

Typical capture cards have a resolution of 6-7ns. 300ns PHY is an out of date value. These guys can do ~425ns wire to application (or 850ns round trip to software) http://exablaze.com/exanic-x4

bcoates2• 12 years ago

How does Snabb Switch compare to the Linux kernel's openvswitch/openflow combination? Are there any advantages to Snabb for forwarding/counting appliances or is it just for packet-consuming applications?

jeffreyrogers• 12 years ago

Shouldn't this link directly to GitHub instead? (https://github.com/SnabbCo/snabbswitch)

Also, just in case the author of Snabb Switch is reading this, it would be really helpful to provide a short example program of how to use this. A brief look over the documentation didn't show anything. (It's possibly I just missed it. In which case it would be great if you linked to it in the README or otherwise made it prominent).

lukego2• 12 years ago

loadgen is a good first example program. That is a load generator that can transmit an arbitrarily large amount of traffic (hundreds of Gbps) from a trace file using a negligible amount of CPU. This is practical and being used every day.

https://github.com/SnabbCo/snabbswitch/tree/master/src/desig...

Here is the driver hack that makes it possible: https://github.com/SnabbCo/snabbswitch/blob/master/src/apps/...

dang2• 12 years ago

Url changed from http://highscalability.com/blog/2014/2/13/snabb-switch-skip-..., which points to this.

Ecio78• 12 years ago

Relevant comment from author (lukego) in this thread of few months ago: https://news.ycombinator.com/item?id=7250505#up_7251722

"I'm the Snabb Switch originator.

The project is new: I and other open source contributors are currently under contract to build a Network Functions Virtualization platform for Deutsche Telekom's TeraStream project [1] [2]. This is called Snabb NFV [3] and it's going to be totally open source and integrated with OpenStack.

Currently we are doing a lot of virtualization work from the "outside" of the VM: implementing Intel VMDq hardware acceleration and providing zero-copy Virtio-net to the VMs. So the virtual machine will see normal Virtio-net but we will make that operate really fast.

Inside the VMs we can either access a hardware NIC directly (via IOMMU "PCI passthrough") or we can write a device driver for the Virtio-net device.

So, early days, first major product being built, and lots of potential both inside and outside VMs, lots of fantastic products to build with nobody yet building them :-)"

Snabb Switch – A toolkit for solving novel problems in networking

Related Stories