Open Letter to Calendar Developers about HTTP

http://lists.w3.org/Archives/Public/www-archive/2013Aug/0021.html

by bpierre • 13 years ago

61 33 13 years ago

lists.w3.org

eksith• 13 years ago

I have a feeling many of these are from rubbish applications that have mostly been abandoned by their developers but still remain installed and running in the background for a lot of users.

codeka• 13 years ago

Except dataaccessd is how iOS's CalDAV sync process identifies itself, so I'd say it's not exactly a "rubbish" application that's been abandoned...

lazyjones• 13 years ago

What do we learn from this? If you provide a public-facing API or service for direct use by applications over the net, use a unique host name / DNS entry for it so you can point it elsewhere or delete it at leisure. DNS entries are cheap, millions of HTTP hits might cost a bit more.

shabble• 13 years ago

For even less fun, there have been numerous incidents of consumer accesspoints/routers hardcoding NTP servers by IP address[1][2].

[1] http://pages.cs.wisc.edu/~plonka/netgear-sntp/

[2] http://web.archive.org/web/20060408150213/http://people.free...

smackfu2• 13 years ago

I would guess the problem is that most sites are going to serve up a 404 in this situation, and handling 404s is a bit trickier, since they might be transient (like a proxy gets in the way, or the hosting service has temporary issues). So they just don't handle it, since polling a server and getting a 404 is going to work well enough for most use cases. Maybe put a little exclamation point next to the calendar or something, but still poll it.

And then they handle a 410 the same as a 404, even though a 410 has very little chance of being transient. It's just rare enough that they don't even realize they don't support it properly.

Wilya• 13 years ago

I'd wager 99% of codes have a big if(200) { // Do stuff } else { // Try again later }

Actually, a bunch of them probably don't even check for response code.

Edit: If dataaccessd is really the iOS native CALDAV integration, that's a bit annoying. Of all people, they should be able to handle that correctly.

pyre4• 13 years ago

  if(200) { // Do stuff } else { // Try again later }

That's like an "idiot-filter" interview question. "Is the following code valid?"

rav• 13 years ago

This is probably a case where an exponential backoff would be preferable. When the client is faced with a fetch error the first time, it should wait and try again in e.g. 10 minutes, and double the wait for each repeated error. This way, after a week of downtime at the source, each client would send less than four requests for the resource per month.

ZoFreX• 13 years ago

No, this is a case where obeying the HTTP specification would be preferable. 410 GONE means: Gone. Don't try again.

Millennium• 13 years ago

Agreed. Incremental backoff makes sense for the 5xx-class errors, where the error is known to be server-side. But 404 is the only 4xx-class error that it makes any sense for: in any other case, it's pretty certain that your side is doing something wrong. Even when you're not -i.e. if you're absolutely certain that this is the right URL for what you want- a 404 should still trigger some kind of notification so that a person can look into it.

Using incremental backoff for 410, in particular, is to miss the point entirely. 410 very explicitly means that you should not try again. The fact that it's so rare only lowers the chance that it's being used incorrectly: someone savvy enough to set up a 410 knows enough to mean it.

That said, I'm starting to think that if you're going to implement 410 on your site, you should also start logging activity. When a host keeps looking, either incrementally or otherwise, start refusing connections from that host at the firewall level. A program that sets 4xx-class errors to fail silently probably isn't doing anything about refused connections, so an IP-ban should wake the developers up. Even if it doesn't, you won't have erroneous requests clogging your server logs, so you still benefit.

MichaelGG• 13 years ago

Why should I believe a server knows a URI will never work ever again? If someone hijacks google.com and returns a 410, should we never try again?

That's the problem with writing specs that include all sorts of "useful" codes. Like SIP's 6xx "globally unavailable", as if a client is going to trust that some server knows the exact state of the known universe.

jstanley• 13 years ago

I have exactly the same problem with a project I run. I generate iCalendar versions of the timetables for each course at my university and let students download them. The timetables are static and do not change, yet I still get hundreds of hits every day from dataaccessd, even in the middle of summer when nobody has any classes.

EDIT: http://notlate.co.uk/

EDIT2: Not that it's enough load to be noticeable, but it's a little wasteful on the part of dataaccessd IMO.

ezxs• 13 years ago

This is another case for rate limiting

kogir• 13 years ago

Am I the only person who after about 3 months would have started returning random and numerous events? Then clients would voluntarily unsubscribe.

biot• 13 years ago

Nope. I'd be hard pressed to not do a little iCal trolling. I'm sure the moon would get destroyed by the Death Star at least a few times per year. Not to mention a new phase of the moon: Space station.

alexwright• 13 years ago

"I have a 2PM with 'Goatse.cx', I don't remember this, let me just click the info link. URGHWTF."

smackfu2• 13 years ago

Then clients would voluntarily unsubscribe.

Or write nasty blog posts about you that pollute Google results forever.

evmar• 13 years ago

A long time ago I published an XML file on my website for some ill-fated Google attempt at providing extensible search results. I took the file down some time later, but the Google crawler continued to hit the URL.

Amusingly, I asked around within the company to try to find who was responsible for the crawler repeatedly fetching the 404 but ultimately couldn't figure it out. (It seemed like the decommissioned service had maybe registered the URL with some still-running automated refetching service?) It eventually stopped on its own.

MBCook• 13 years ago

Fetching stuff over HTTP can be incredibly stupid. I there is a security list out there that some little firewall company decided to have their devices automatically download. The list doesn't change that often, but every single device that company made likes to check every 10 minutes or so. Nothing can stop them, they follow Wilya's algorithm [1].

[1] https://news.ycombinator.com/item?id=6268919

Side note: it's sad to see that Apple is a huge perpetrator in this one. I hope they fix that bug.

eli• 13 years ago

Google will continue to request 410 resources for quite a while too. As far as I can tell it treats them the same as 404.

charlesism• 13 years ago

I'd never even noticed there was such a thing as a 410 until this. And it's not surprising - what incentive is there for a team in the real world with a shipping product to honor it?

There's a pretty obvious incentive to ignore it: it takes developer time to implement, and the result of the 'feature' is that your customer may get permanently locked out of a net resource. What happens if a website admin accidentally puts up a 410, or - worse yet - if a previous owner of the domain once put up a 410 for a page with the same URI.

I suppose there's an RFC for the code somewhere, but it seems like a bad idea. I pity the tech support agent who has to explain why his/her customer's browser refuses to visit foo.com/bar when everyone else can view it and there's no longer any way of knowing that once, for an afternoon in 1997 the page threw a 410!

Open Letter to Calendar Developers about HTTP

Related Stories