Announcing the `http` crate

https://users.rust-lang.org/t/announcing-the-http-crate/12123

by JoshTriplett • 9 years ago

240 61 9 years ago

Would it not be better to call it http-types/http-base/http-common as this just provides common types and not the transport or security? Thanks and keep up the awesome work.

JoshTriplett6• 9 years ago

I think "http" actually makes sense here, for two reasons.

First, this crate provides a common layer for the HTTP protocol; if you're interacting with HTTP, you'd use this crate and its types, often directly. Note, in particular, that this crate provides builders to construct HTTP requests and responses, which is often a large part of client and server libraries, respectively.

And second, I don't think any specific HTTP client or server software should claim the crate name "http". A common layer like this, built around the protocol itself, seems to have a much more reasonable claim to it.

tadfisher2• 9 years ago

In addition, the "transport" and "security" portions are the least-HTTP-related parts of an HTTP stack. Far more than HTTP runs on TCP and TLS.

gregwebs• 9 years ago

In haskell it is called http-types

tyingq3• 9 years ago

One watch out here is patterns that encourage developers to do the wrong thing.

With golang, for example, it's common to see applications that don't set reasonable cache control headers, content type and length headers, handle head requests properly, compression, and so on.

It's not that golang can't do those things, but the libraries, documentation, and examples don't encourage it. So you see a lot of minimal http implementations that "work", but not well. It's like every (less experienced) end user of the library has to learn all the basics by trial, error, bug reports, etc.

dikaiosune• 9 years ago

This particular announcement is about defining shared API types between libraries for which (I think) the things you've listed would be a concern. It's not a full HTTP implementation AFICT.

taeric2• 9 years ago

The point is that pushing this as a concern for all implementing libraries is dangerous and likely not to end well. (At least, that is how I took the point.)

tadfisher2• 9 years ago

I'd argue that all but the most extremely basic of HTTP options are far from universal, and defining sane defaults is the responsibility of frameworks and not a core library.

JoshTriplett6• 9 years ago

Agreed. And in particular, these kinds of types would make it much easier to write a framework-agnostic library that helps you handle caching correctly, built on top of these types, like Request and Response.

tyingq3• 9 years ago

That doesn't rule out good examples though. And some of these are close to universal, maybe not in set value, but at least setting them to some value, versus omitting them.

RandomKid2• 9 years ago

> defining sane defaults is the responsibility of frameworks and not a core library

But...but.. Golang core team teaches us that "framework" is a 4 letter word and a core library is enough for everybody. No need to overcomplicate with extra abstractions, just use the standard library they say.

joosters• 9 years ago

As I understand it, this http library is meant to represent a 'low-level' representation of a HTTP request, and so I think it's right that it doesn't force lots of default HTTP headers or settings.

HTTP requests rarely live in isolation, you are probably going to be making several requests to one or more servers, so most language libraries have another layer on top of the raw HTTP objects. (e.g. in Perl, there's LWP::UserAgent, among others). Here is where the network code tends to exist, for instance. It's more appropriate to put some defaults at this layer - for example, keep-alives and connection re-use are vital concepts for efficient HTTP communication but apply across multiple requests. You can't sensibly control them at the base 'this is a single HTTP request' object layer.

angrygoat• 9 years ago

I think this is mostly a layering issue. If everyone that just wants to scrape HTML or talk to an API is using the basic HTTP API, then you've got this problem. That's where Python was about ten years ago - using the stdlib's urllib, it was very easy to talk HTTP, but you'd probably do it badly.

Then requests came along, aimed to make it easy for anyone to consume HTTP, with sensible defaults -- and it rapidly became the defacto standard way to get things done. Hopefully the rust ecosystem will have similar libraries, built on top of this new 'http' base.

jonreem• 9 years ago

Good news! The `reqwest`[0] library already exists and serves exactly that purpose.

[0] https://github.com/seanmonstar/reqwest

int_19h• 9 years ago

It's great to see this approach to core API design, and I wish other platforms adopted it. There are many advantages to having a rich ecosystem of third-party libraries, but composability becomes a problem when they define different types for the same concepts. This is the best of both worlds - you get multiple competing implementations of the interesting bits, but everything "just works" with any of them.

nly• 9 years ago

I couldn't agree more. Although, as someone mostly in to C++, I think data types like this should be standardised as concepts rather than added as concrete types. The HeaderMap described in this crate sounds like it is quite complex. I'd rather have a concept with some kind of trait mechanism for determining if e.g. the type preserved insertion order (which defaulted to false)

The Beast HTTP library that just got accepted in to Boost follows this approach. E.g. the following Fields concept specifies the requirements the library has on its any HeaderMap-like type you feed it. Writing an adapter for your own types then becomes straightforward

http://vinniefalco.github.io/beast/beast/concepts/Fields.htm...

derefr• 9 years ago

Defining the type as a trait would prevent link-time optimization of library-consumer-side manipulations of data of that type, no?

Really, you want a concept that's like a trait, but is explicit about the fact that there's exactly one implementation of the trait, and that that implementation is discoverable at compile time. Less like an interface, more like a C typedef in a third-party-library header file—just without the "header file" part.

cheez• 9 years ago

Can't wait for concepts...

barrkel2• 9 years ago

Counterexamples: XML DOM APIs, as interfaced and implemented (separately) in, say, Java; or OpenGL, with its extension functions.

One problem is construction. When you need to interact with a library at one remove using interfaces everywhere, you can't simply new up instances; you have to go via factories. And that leads to ugly alien code.

Another is extension (OpenGL). If your super duper implementation has nifty new features, how do you expose them if you're living behind an interface? Does the interface enable extension in a usable way? Or is it indirect and awkward again?

In practice, you code and test against one or two implementations (bug compatibility), and other implementations are only supported by accident or effort by the implementors.

(I'm fully on board with the idea that a language ecosystem is constrained by the highest level of abstraction in the type system that's shared across all ecosystem libraries. It's the strongest argument for a large standard library.)

icebraining• 9 years ago

That's mostly a Java problem, though, which lacks first-class types.

erikpukinskis2• 9 years ago

I think there might be a typo in your second paragraph ("at one remove"), but this is very interesting to me, so I would love to pick your brain.

Why is a factory more alien than simply newing up an instance? Shouldn't they have basically the same interface? Is it just a matter of giving people rope to hang themselves... because they can do an arbitrary interface, someone inevitably will make a weird one, and then someone else will think it's cute, and the fashion gets perverted?

That does seem like a problem. It could be solved with culture, but solving problems with culture is hard so I understand your decision to write it off as a bad direction.

I am writing a lot of code with this kind of structure lately though, so you've got me a little worried. One of the things I dislike about a library is when it exposes raw data structures to the user, and then presumes other libraries will understand that structure. For example, if I have a client library that is sending HTTP requests, and then another separate library on the server that handles them, there is an opportunity there for miscommunication in that data layer.

On the other hand, if I have one library which gives me both a factory and an interface which consumes that factory object, then the library handles both sides of the data structure management. I am forced to only work through the interface, which means as long as my libs are in order, everything should be able to communicate.

Sorry if that's vague, but I'm programming in JavaScript and this post is about Rust and you're referencing Java and OpenGL, so I'm not sure exactly where the ground is. Maybe I only like my approach because it's JavaScript, so I can't ever assume the type of anything is correct. #stockholmsyndrome

laumars2• 9 years ago

Go's interface{} gets heavy criticism, a lot of it deservingly too, but this is one area where I think Go works really well. You can have a standard API structure and use that across multiple domains.

I really do need to give Rust a try though. I'd been holding back because my impressions of it was the core libraries were still in flux so code that compiled today might fail 2 months down the line. Is this still the case now?

cytzol• 9 years ago

It's all good now! Rust has been stable with backwards-compatibility guarantees since 1.0 was released two years ago.

ented1• 9 years ago

No bullshit: interface{} is almost the same level hack as void*. And is not well, no matter how much Stockholm syndrome you have.

amelius• 9 years ago

It's a lot like the Boost libraries are for C++. They are curated, and easily composable by design. I don't see the big leap here, but you are right, it is a very good way of working.

On the other hand, this is the Nth time that somebody implemented a HTTP library on some platform. It seems that something can be improved still, software-engineering wise.

MrBuddyCasino• 9 years ago

I agree, Rust leadership continues to make well thought out decisions. Coming from Java, the Servlet spec is comparable (though broader in scope) and was really successful.

mixedCase• 9 years ago

I can't seem to find a parser for HTTP requests in the repo. I assume this is out of scope for the crate?

connorcpu• 9 years ago

Looks like this crate is merely trying to provide a standard set of types used in an HTTP pipeline to make different crates that want to have a part in that pipeline able to be combined together, without having to convert your types all over the place

ivanbakel• 9 years ago

To quote the announcement:

It does not cover transport, but is rather intended for use in libraries like Hyper, and to support an ecosystem of crates that can share HTTP-related types.

So I don't think they added any implementation details to the crate itself, to avoid people taking issue with something being too bloat-ey or not to their need, and rejecting the common types, which seem to be the real goal. It does seem like something you'd always need, though - I can only imagine that a proper parser would be pretty large, and they didn't want people to complain about it.

JoshTriplett6• 9 years ago

I wouldn't be surprised if an "http-parser" crate pops up on top of this. But unlike the http crate, an http-parser crate would have to make at least one potentially controversial choice, namely "which parsing library".

tatterdemalion• 9 years ago

Parsing would get into policy choices this crate doesn't want to make, like DoS protection (max request size, mid-request timeouts, etc), and often is intertwined with reading data from IO, which this crate definitely doesn't want to get involved in.

deathanatos2• 9 years ago

I really wish I could start a rebellious movement in HTTP libraries to call the constant UNAUTHENTICATED and for it to return

    401 Unauthenticated

…and then once rough consensus switches, get an update in the RFC.

(I'd likely be horrified to learn that something out there critically depends on the Reason Phrase…)

azdle• 9 years ago

(I'd likely be horrified to learn that something out there critically depends on the Reason Phrase…)

IIRC, the 8xxx RFCs have specified that you must read the reason and that the code itself does not fully specify the result because there have been some custom codes that overlap on number.

deathanatos2• 9 years ago

the 8xxx RFCs? Did you mean the newer 7xxx ones? If so, they say,

The reason-phrase element exists for the sole purpose of providing a textual description associated with the numeric status code, mostly out of deference to earlier Internet application protocols that were more frequently used with interactive text clients. A client SHOULD ignore the reason-phrase content.

I'm not aware of any highlevel HTTP RFCs in the 8xxx range.

JoshTriplett6• 9 years ago

You're not supposed to depend on the specific reason phrase given with the code, but I wouldn't find it surprising if something does.

See https://hackernoon.com/three-bytes-and-a-space-8f9fbd1c669b for a related debugging adventure.

Also, technically, you can return 401 for either "unauthenticated" (you need to pass authentication information) or "unauthorized" (you passed authentication information but it wasn't acceptable), depending on the nature of what you need.

BillinghamJ3• 9 years ago

401 Unauthorized means you aren’t authenticated, or your authentication is broken/expired/invalid.

403 Forbidden means that your authentication is valid, but that you don’t have access to the resource.

401 should not be used to indicate the latter.

majewsky• 9 years ago

"unauthorized" (you passed authentication information but it wasn't acceptable)

That's still "unauthenticated". "Unauthorized" means that you were authenticated (i.e. the server knows who you are), but you are not allowed (i.e. authorized) to execute the requested operation. So the correct names would be "401 Unauthenticated" and "403 Unauthorized".

awj• 9 years ago

Cf. "Windows 10", which had to a version number because people were looking for a "Windows 9" (i.e. Windows 98 & Windows 95) prefix in a version name string despite it being available as direct numeral values.

Feel free to be horrified, but you'd be a bit naive to be surprised.

arianvanp• 9 years ago

Seems like they took the hint from haskell :) good job. We have WAI (https://www.stackage.org/lts-9.0/package/wai-3.2.1.1) a common interface for multiple HTTP Web Servers. It's very useful, and a rich ecosystem of middleware has flourished.

gPH45• 9 years ago

Interesting micro-optimizations... For example the use of lookup tables and nested switches in uri.rs and method.rs.

Is this common practice in Rust? Shouldn't the compiler be smart enough to do that for you? Such code seems quite sensitive to typos.

DiThi• 9 years ago

I don't think the lookup tables is a micro-optimization. They're both clear and fast in this case (I can't think of a more clear alternative, just of a less verbose one). The "switches" are actually pattern matching, a very powerful feature of Rust. I'm not sure about nesting them being a micro-optimization either. I think it's also a choice that is both readable and fast. Can a more experienced Rust code shed light?

thijsc• 9 years ago

This uses pattern matching, the compiler will verify that all options exists and that the coverage of the enum is complete.

pimeys• 9 years ago

This is good news. Waiting it to be merged with hyper including the http2 support. With both of them it will make Rust the perfect choice for all my current use-cases.

Announcing the `http` crate

Related Stories