Show HN: Hydrogen, a multithread non-blocking Linux server framework in Rust

https://github.com/nathansizemore/hydrogen

by nathansizemore • 10 years ago

154 46 10 years ago

bradhe• 10 years ago

Happy to see projects like this pop up! As a long-time go user, these types of fundamental projects really are what drove the community early on.

nathansizemore12• 10 years ago

Thanks! Was lucky enough to be able to use Rust for a production application, and hope it can be useful to others in providing the tedious mundane parts of socket server building :)

dengnan• 10 years ago

Nice work! But you may want to remove src/#lib.rs# :)

cyphar• 10 years ago

I like the fact that you chose MPL. Not enough people are using copyleft these days. :D

loonattic• 10 years ago

Is it built such that a different event notification thing such as libev, libevent, kqueue could be swapped in without too much effort? Well, it's not even that much code.. looks nice

nathansizemore12• 10 years ago

Thanks, but it only uses epoll :)

mio is a crate that offers more platform agnostic async I/O.

idobai3• 10 years ago

addr: "0.0.0.0".to_string(),

Reinventing c/cpp? ಠ_ಠ

kibwen2• 10 years ago

String literals in Rust produce references to data living in static memory. The `String` type, which is dynamically-allocated and growable, is only created when requested due to the cost of dynamic allocation.

idobai3• 10 years ago

So, do you need to convert the string to a string whenever you want to use a mutable string?

nathansizemore12• 10 years ago

TBQH, I do not have the slightest clue why I decided to implement that as a growable String type instead of a literal. Definitely warrants a change.

onion2k• 10 years ago

It's great to see people talking about how they're using Rust in production. One of the biggest barriers to introducing new tech in to a business is a lack of successful examples.

kibwen2• 10 years ago

We just began working on an official site to showcase companies using Rust in production, here's an early look: https://www.rust-lang.org/friends.html (mouse over logos to see links to code and testimonials, we're still working on the UX).

ekidd• 10 years ago

As somebody who works with one of the companies on that list, I just want to say that, yes, using Rust in production is a great experience. (At least once I made friends with the borrow checker, which took some experimentation.)

We especially like the way that Rust code is simultaneously high-level, strongly-typed and fast. The code feels "crisp", and if the compiler doesn't return any errors, the program will almost always work correctly on the first try. The code refactors easily. Plus the tooling (cargo, unit testing, etc.) is well thought out. And we like the ability to build static binaries against musl-libc and copy them into an Alpine Linux container.

I would happily use Rust again, at least if I wanted C++-level performance and most of the right libraries were already available for a given task. (I can find most basic, useful libraries on crates.io, but if I'm looking for something like an HTTP proxy server using async I/O, I may not necessarily find a mature implementation yet.)

yazaddaruvala2• 10 years ago

May I ask,

Why not Hyper?

Why not Mio?

Don't misunderstand, I'm all for competing implementations. I just feel like it would be helpful to lay out a comparison for everyone else's understanding/decision making process.

nathansizemore12• 10 years ago

Of course!

Well, hyper is a lib for dealing with the HTTP protocol. If being cross platform did not matter, hyper could be build atop of this.

I did not go the route of mio because I wanted to learn the entire process of working with epoll by doing it myself. I've learned a lot, and would not have that knowledge if I just built atop of mio. Mio must also provide a unified abstraction across platforms, where I was free to experiment and change at will during the building process.

pcwalton3• 10 years ago

Is it possible to stay compatible with mio's interface, perhaps via something layered on top?

151553• 10 years ago

Why not Boost? / Why not libuv?

Why not EventMachine? / Why not Celluloid?

Why not Twisted? / Why not gevent?

This is a really sad situation unfortunately. I was hoping Rust wouldn't head down this road.

To preempt: a systems language shouldn't dictate these high-level matters, but there certainly should be one "blessed solution" vs. a sea of incompatible libraries.

Go doesn't have this problem, Node.js doesn't have this problem, Elixir doesn't have this problem. Sad deal.

pcwalton3• 10 years ago

To preempt: a systems language shouldn't dictate these high-level matters

But you cited a whole bunch of languages that do dictate this. Those languages don't have those problems because you can't write competing implementations of low-level async I/O in those languages.

And there certainly are competing, incompatible implementations of relatively low-level functionality in Golang, for example fasthttp: https://github.com/valyala/fasthttp

I'm all for standardizing on one solution when the time comes. Rust will have an async I/O story. But right now nothing is mature enough yet to begin that process. Rushing to standardize something that is not yet ready is the worst of all possible options.

nathansizemore12• 10 years ago

Why not X?

I wanted to use Rust and learn epoll through working with it directly.

I was hoping Rust wouldn't head down this road.

To my knowledge, Rust does not have a std async story. I hope it will, and I believe libs like mio will help flush out platform agnostic ideas and a tested API.

To preempt: a systems language shouldn't dictate these high-level matters

All of your cited example languages are _not_ systems language? Node.js isn't even a language? It could be listed above with the rest of your async I/O options across several languages.

Svenstaro• 10 years ago

Alright, when would I want to use this? It reads like I would use it for a high-performance and safe network daemon of some kind. So basically it could be used as the lowest layer of, for instance, an IRC server?

nathansizemore12• 10 years ago

Exactly. It is ideal for any type of high performance server that needs to handle many long lived concurrent connections.

Matthias247• 10 years ago

Just some comments from looking very briefly at it:

The send API lets me wonder how it copes when the socket only allows to send half a frame and then returns EAGAIN/WOULDBLOCK? There is not a number of bytes that were really sent returned, and the ownership of the data to send was given up, so it can't be retried later on.

On the receiving side it looks like the framework is allocating the memory and then pushing the received bytes instead of the user allocating the memory and fetching it (after readiness notification). That gives up some potential for memory reuse.

nathansizemore12• 10 years ago

Thanks for the feedback! This means I need better docs :)

So, the framework itself does not implement any of that, it is on the consumer. It just calls the proper methods on the trait object provided.

https://nathansizemore.github.io/hydrogen/hydrogen/trait.Str...

In the README example, I'm using it combined with another streaming crate I made, which does provide all of the internal buffering and state management for non-blocking I/O.

txutxu• 10 years ago

In software context, for me, the name is taken.

    Hydrogen: advanced drum machine for GNU/Linux

And a very good one, with a long road in the back.

http://www.hydrogen-music.org/hcms/

amelius• 10 years ago

How about "proton"?

gkya• 10 years ago

What about a proper, useful name that hints to the functionality? "libhttepoll" (because built on epoll)?

IIAOPSW• 10 years ago

How about compromise and call it "H+ ion"

nathansizemore12• 10 years ago

Bummertown. This is my first open source project I've really broadcast to the world and I should have taken more time to research the name instead of just going for the first one that came into my head. Thanks for pointing that out!

dberg• 10 years ago

So i assume this is similar to Ranch in Erlang ?

nathansizemore12• 10 years ago

From briefly looking across the README, it looks like Ranch is more for maintaining a pool of connections, where this _also_ handles the marshalling of consumer defined I/O (Non-blocking read/writes) operations, and the workflow associated with non-blocking I/O.

yarrel• 10 years ago

Loading drum samples into this will be hard.

Show HN: Hydrogen, a multithread non-blocking Linux server framework in Rust

Related Stories