Tell HN: Please Stop Breaking HTTP Clients

https://news.ycombinator.com/item?id=705095

by tdavis • 17 years ago

124 30 17 years ago

news.ycombinator.com

For whatever ungodly reason, the news.yc web server doesn't terminate header lines properly when sending a response, which happens to break any HTTP client that actually follows the RFC. Please see RFC 2616 section 3.7.1 and use \r\n as God intended.

   This flexibility regarding
   line breaks applies only to text media in the entity-body; a bare CR
   or LF MUST NOT be substituted for CRLF within any of the HTTP control
   structures (such as header fields and multipart boundaries).

Thank you!

aristus2• 17 years ago

Voted up. But a lot of homebrew webservers don't adhere and you'll just have to deal. In this case "God" is W3C, and not every decision they made was good, eg the "Referer" [sic] header. :)

[deleted grumble about writing production HTTP clients in 2009]

tdavis3• 17 years ago

Neither, I was reading through a Twisted ticket and it was mentioned that this server was breaking the new HTTP client (and why). I then confirmed this ridiculousness myself, and posted it.

I wrote one in 2008, though. Okay, most of one. But in my defense, no comparable [python] client existed at the time and I needed the functionality (synchronous requests were out of the question; a couple threaded clients were available but not robust or fast enough). As a general programmer rule, I tend to avoid writing code wherever possible :)

paulgb• 17 years ago

I wonder if anyone has tried to calculate how many bytes of data transfer have been saved from w3c's spelling of the word "referer".

aristus2• 17 years ago

Fewer than that wasted by redundant CRLFs.

nir• 17 years ago

Voted up, as a good example why it's better to build on top of existing software rather than reinvent the wheel (or in this case, Apache)

blasdel4• 17 years ago

Apache is probably the worst example you could have possibly picked, it's such a dog.

davidw5• 17 years ago

Eh? Apache is a fine web server which has done a great job over the last 10+ years for all kinds of people and businesses.

kaens2• 17 years ago

Could you expand on why you don't like Apache? I'll grant you that it can be confusing to get used to configuring, especially if you're not really familiar with how the web works, but as far as I can tell it's a pretty damn good server.

What do you prefer over apache?

pufuwozu• 17 years ago

I had the same problem a while ago and ended up writing a patch to make PHP's HTTP library more flexible:

http://pecl.php.net/bugs/bug.php?id=15223

coderrr• 17 years ago

I had to make changes to ruby's em-http-request for this

http://github.com/coderrr/em-http-request/commit/8e6444fe472...

ivank• 17 years ago

This was (probably) fixed in Anarki a while ago:

http://github.com/nex3/arc/commit/6cb43b3a5977950a61bfd6ce5a...

visitor4rmindia• 17 years ago

Maybe this would help?

Section 19.3

The line terminator for message-header fields is the sequence CRLF. However, we recommend that applications, when parsing such headers, recognize a single LF as a line terminator and ignore the leading CR.

tdavis3• 17 years ago

A recommendation to applications is hardly as noteworthy as a written spec. Most clients (rightly) go by the spec, as you can see in this thread (PHP and Ruby have been mentioned; Python's urllib.urlopen also fails to properly parse the CR out of headers). So, no, that doesn't help the fact that the current production server violates the spec and breaks most mainstream http clients in the process, and does so with any oversight that is trivial to fix (which should never have made it into production and indeed already has been fixed in the repo).

blasdel4• 17 years ago

That guidance is for how best to "be liberal in what you accept"

news.yc's silly problem is in the area of "be conservative in what you emit"

herf• 17 years ago

The rather good Fiddler debugging proxy (http://www.fiddler2.com/fiddler2/) always flags HN for this. In a popup, no less. So it drives me nuts too, and if it could be fixed, I could click on fewer popups.

davidw5• 17 years ago

I wonder if this is the reason I get a proxy error when I try and browse HN from my mobile phone.

jacquesm• 17 years ago

So, you're crawling HN and you expect the target of your crawl to fix something so you have an easier time of it :) ?

kaens2• 17 years ago

I would think that HN wouldn't really care about someone crawling them as long as they were respectful about it.

Also, as far as writing crawlers/scrapers goes, a header-handling error is one of the least annoying things you can run into.

tdavis3• 17 years ago

I'm not doing anything with HN. I expect the server to emit the proper line breaks because that's what it should do, especially if it's going to be made freely available and used by others.

There's no reason that such a simple bug should have made it into or remained in the production source for so long. It's laziness for laziness' sake.

stefano• 17 years ago

It broke my http client too (written in Arc, btw) some time ago: http://arclanguage.org/item?id=8283

Tell HN: Please Stop Breaking HTTP Clients

Related Stories