Where did all the HTTP referrers go?

http://smerity.com/articles/2013/where_did_all_the_http_referrers_go.html

by Smerity • 13 years ago

110 77 13 years ago

warcode• 13 years ago

I already hide/change my referrer through browser extensions, though I guess this might finally get those few sites who have functionality based on refs to stop using it.

Smerity5• 13 years ago

Whilst I still think referrers should be used where appropriate, there are certainly places or reasons you want to nix them. HTTPS not passing along referrers by default is a sensible decision, as is the addition of <meta name="referrer" content="never"> for sites stuck in HTTP that may want to go under the radar.

I still feel that removing referrers entirely destroys many useful tools and analytics that we've traditionally been able to use. It removes the core way in which we understand connections across the Internet. By removing referrers, the best we can do is use link graphs, falling back to the original PageRank algorithm where we assume people are random bots that click on one of the links on the page.

Edit: Can't reply to you hnriot due to comment depth limit. My reference to PageRank is as links and backlinks could be used as a poor referrer substitute, though they aren't currently used as it's a lot more work and less accurate. I'm simply saying that, in the event that referrers all disappeared tomorrow, you'd see normal websites trying to estimate where their traffic comes from by using a PageRank inspired algorithm, or more naively by looking at who links where.

hnriot• 13 years ago

Referrers have nothing to do with pagerank. It's used in website analytics of course, and, in general is a very bad idea leaking information between websites. Search engines don't see your site's referrer data so the association you tried to create with pagerank is misleading.

anonymouz• 13 years ago

Referers are a privacy trade-off though, and personally I have never seen a good reason why the desire of some website to track where I am coming from should trump my desire to not broadcast this information to the world. Therefore I like to switch of referers anyway.

claudius2• 13 years ago

I am doing the same. Fortunately, the websites requiring the referer to be set are getting fewer and fewer – some years ago, I regularly got placeholder images ‘THIS IMAGE WAS STOLEN FROM XYZ’ when browsing XYZ without referers.

joosters5• 13 years ago

This is one of the good uses of HTTP referrers!

I host a small, very low traffic website. One day, the bandwidth shoots through the roof and stays high. The reason? One of the images on a page got added to the .sig of someone in a popular forum. Suddenly thousands of people are fetching the image.

The solution was to filter by referrer header, letting the image be seen by visitors to the actual page, but linking from other sites gets blocked. Note that usually it's best to allow requests that have no referrer header at all, otherwise you'll be blocking some legitimate viewers of your site.

End result: bandwidth back down to the usual, tiny levels. It's not that I cared about people copying the images, I just didn't want to foot the bill for the traffic!

huhtenberg• 13 years ago

Typekit and other @fontface providers rely heavily on them. So, with referrers off, every fancy designer website renders in Georgia.

NameNickHN• 13 years ago

In our case referrers can sometimes be really helpful. More than once a (potential) customer has posted a link to our website with questions or comments about our products. This allowed us to head over there and answer those questions or generally get in touch with your target audience.

Nursie• 13 years ago

Yup.

It's just another way that (for some reason) my browser leaks information without asking me if I want to.

It gets switched off now, along with cookies, analytics and most other stuff.

monkeyspaw• 13 years ago

I respect your desire to not be tracked via referer headers, and to turn them off/use SSL/use extensions. You are a person, and your privacy is important to you.

I also respect a site owner's desire to track your referer, and to decide the rules for who gets to access their content. They are (often) a business, and knowing where to focus their efforts to make money is important to them.

I don't respect either side feeling they are owed tracking information, or content, if they aren't willing to respect the other agent's rules and preferences.

lucb1e• 13 years ago

Oh I did not know this. Very good to know indeed! I too am missing referrers from websites, and since my website also supports https I suppose others are missing them from me as well. This way I can make the internet a very tiny bit better ^^.

mooism22• 13 years ago

So my understanding is that the default behaviour is:

1. Follow link from https://example.org to http://example.com --- referrer is not sent

2. Follow link from https://example.org to https://example.com --- referrer is sent

I don't understand how the same referrer can be too sensitive to be sent as plaintext, but harmless enough to be passed to a not-necessarily-trusted third party.

kyrias• 13 years ago

It's more about who can see it along the way than it is about who receives the request.

ctz2• 13 years ago

I'm willing to bet that it gets sent even if the second HTTPS connection provides no effective security.

(If this is the protocol we're building economies on these days, I feel computer security is going to get much worse before it gets better...)

mAritz• 13 years ago

That may be, but it really shouldn't be. You cannot have sufficient trust in all parties involved to be certain that no sensitive data is going to a third party (the website you're going to) that is malicious.

Smerity5• 13 years ago

This is the default behaviour, thought it indeed likely means sensitive referrers are being leaked from website X to third party website Y. In most cases I'd wager people aren't aware this is the default behaviour.

An example where the default behaviour may be appropriate is Facebook interacting with third party apps.

Facebook may be happy to pass referrer information across to these third party apps as long as they handle it securely. If the referrer goes across HTTP, it goes across the Internet in plaintext (unsecure). By ensuring it travels over HTTPS, you're at least ensuring a minimal level of security.

buro92• 13 years ago

Let me fix that for you:

1. Follow link from https://example.org to http://example.com --- can be read by a third party if referrer were added

2. Follow link from https://example.org to https://example.com --- cannot be read by a third party so referrer can be added

The assumption is that secure pages are secure for a reason, and that the author of a secure page is linking to other secure pages and has some basis of trust by which the link is provided.

mooism22• 13 years ago

Example.com is the third party. (Example.org and a human user being the first two parties.)

Let me rephrase my question: why the default assumption that example.com is trusted not to misuse referrer information merely because example.org provides a link and the human user follows that link?

biot• 13 years ago

By third party, do you really mean eavesdropper?

dlsym• 13 years ago

But why is that a bad thing? Just count your page impressions to see if you blogcontent is popular.

What websites your visitors consume is IMHO not your business.

josteink• 13 years ago

> What websites your visitors consume is IMHO not your business.

But if your product is the subject of discussion on another page which links to yours, you may want to be able to join the discussion.

As a site owner, I don't think it's unreasonable to ask my visitors how and where they found out about my website.

jonknee• 13 years ago

It's two very different things if the sudden spike in page views is because of nytimes.com or reddit.com.

w00kie• 13 years ago

Thanks for the info, I was not aware of this meta tag. My blog is all HTTPS (with SPDY) and I added content="origin" to all visitor facing pages, leaving the admin pages dark.

cnahr• 13 years ago

When I recently tried to purge my server logs of referrer spam, I found that a whopping 80% of all visitors had either no referrer or referred from my own website. About a third of the rest were spam. So judging from my small sample, referrer headers are largely useless except as spam machines. They should probably be removed from HTTP anyway.

kmasters• 13 years ago

I fail to see why referrer is part of any spec. Referrer is a 90's concoction when we were all so naive of future uses. I'm sure it's made a lot of porn sites some money, but affiliate marketing is not the purpose of web standards.

As far as analytics its all done with tracking cookies now, which is another issue that needs to be addressed, but Im not going to miss referer should it ever really go away.

jacques_chester2• 13 years ago

My honours project was on the problems of tracking users visiting different websites using cookies.

The problems are numerous. I think I went through 9 revisions of the "naive protocol" (including, at one point, ditching the referer header in favour of HTTPS). Subsequently I realised that my tracking protocol was broken anyhow.

My conclusion is that there's no reliable way to track users visiting multiple websites using the standard features of HTML/JS/HTTP in the face of malicious users or publishers.

You have to fall back on traffic analysis.

I developed a successor technology which works better in many respects (but not all). As it's the subject of a current patent application I can't really go into much detail.

mayank• 13 years ago

I hope you've looked at evercookie before you go about filing patents on user tracking...

Angostura• 13 years ago

I see it rather differently. Referrer is a 1990s concoction based on the idea that the Web is a collaborative effort that we all engage in. I don't personally have any problem in a Web site knowing where I arrive from in most cases. It's harmless him of information (in most cases, as I say) which makes the Web site owner's life a little better.

mindcrime• 13 years ago

Same here. Referer is part of the way the Web works, and I, for one, don't give two shits if Site A knows that I arrived via a link from Site B.

That said, I can see why certain people might see it as problematic... if you're browsing http://www.anarchistsbombmakingforum.com and somebody posts a link to http://www.fbi.gov, then maybe you don't want the FBI knowing you were at anarchistsbombmakingforum.com. But, still, barring other privacy problems, the FBI don't know who you are when you visit their site, just that you came from anarchistsbombmakingforum.com.

I have a hard time getting worked out about this though... for one, if you're surfing anarchistsbombmakingforum.com, common sense would dictate that following a link to fbi.gov isn't such a good idea (and a forum that fosters discussions of anything controversial should probably munge links to go through an anonymizer anyway) AND the people for whom this really matter are the people who have a referer blocking plugin installed in their browser.

kvi• 13 years ago

Yeah, that's a long known thing. I use HTTPS on parsebin.com specifically for stripping out referrers.

nathell• 13 years ago

This is how I discovered HN went HTTPS some time ago. I never noticed until now.

sp332• 13 years ago

HN started using SPDY recently, and SPDY requires HTTPS. https://news.ycombinator.com/item?id=5660797

Skalman• 13 years ago

HN has been available over HTTPS for longer than that.

happyshadows• 13 years ago

I wish Google would change their meta-tag from "origin" to "always". It's getting harder and harder to see which keywords bring traffic to your site these days.

kijin3• 13 years ago

The article makes it sound like every webmaster is entitled to see Referer headers. Really? How does enabling referers "save the world"?

A few weeks ago, I ran across a website (can't remember their domain) that refused to show me any content unless I enabled third-party cookies, and even contained a lengthy argument explaining why disabling third-party cookies hurts the web. IIRC the whole argument was 24K bullshit, written by people who feel entitled to keep making money with their outdated business models and who were obviously alarmed because modern browsers were doing sensible things. And now we're seeing a very similar argument, only this time it's about referers. Why do you think it matters whether I came to your site via Google, Reddit, HN or someone else's blog? What makes you think gives you the right to know that?

Webmasters never had the right to know where your visitors were coming from, any more than the owner of a random gas station on the Interstate has the right to know which city his customers are driving from. If SSL is making Referer headers disappear, good riddance. We just closed a privacy hole, 99 more to go. Next in the TODO list: get rid of referers even when the referring website doesn't use SSL, because as the article correctly points out, we've got a bit of inconsistency there.

arb99• 13 years ago

I use referrers extensively (see how sites are doing traffic wise, traffic trades etc) so I am definitely glad they exist, but i am always surprised it was ever added to the spec. think it was added in 1.1 (so 1999). Without it I guess a LOT of sites nowadays would link out with urls like ?from=mysite.com so people knew where traffic was coming from.

It is, like you said, basically a massive privacy hole. But a very handy one

kijin3• 13 years ago

I think search engines like Google should just standardize on a query string like your example. It always works, it's clearly opt-in, it's visible to the user, it's just as straightforward as a header (if not more) for web apps to process, and it's no more spoofable than a header anyway. If you think it's a bad idea to pollute URLs with such things, an alternative would be a URL fragment that is detected by JS.

pi18n• 13 years ago

And there are occasionally similar articles where the author informs us all that AdBlock and NoScript are literally destroying the internet. And that misguided blogger this week who wants Apple to force users to accept push notifications from any app they've installed to make his life easier.

Smerity5• 13 years ago

If you reread the article, I actually say you should add a meta header regardless of whether you do or don't want to send headers.

Why? It actually solves your "next in the TODO list". Most web sites that shouldn't send referrers don't use <meta name="referrer" content="never"> so will be leaking referrers to other web sites. Adding this meta tag will eliminate referrers in both HTTP and HTTPS.

So yes, my own preference is to keep HTTP Referrers, but I also explain how to kill HTTP referrers for webmasters who would like to as well.

untog3• 13 years ago

That's still taking the decision out of the hands of the user, though.

I'm not saying that I totally agree with kijin, but I don't think your answer addresses his point.

kijin3• 13 years ago

I disagree with your suggestion to use such a meta tag, because it's an opt-out scheme. People should not have to opt out of potential privacy leaks. If a web page does not specify any referer policy, I think the default should be no referer.

Unfortunately I still can't remember the website where I found the "bullshit" argument, and it's not in my history because I probably ended up using a different browser to comply with their no-access-unless-you-accept-3rd-party-cookies policy. But if you understand why people like me want third-party cookies to be disabled by default, I think you'll also understand why I want referers to be disabled by default, too. It's not about user control as @untog suggests, because the user is always ultimately in control when it comes to HTTP headers. Rather, it's about having secure defaults.

untog3• 13 years ago

There are some legitimate (IMO) uses of referrers, though. Preventing hot-linking, for example.

vidarh• 13 years ago

There's many legitimate uses for them, but while they're convenient for preventing hot linking, they're not necessary. Preventing hotlinking can be done easily enough by appending a url argument that changes regularly, and can be made performant by adding that argument checking to the configuration of your frontend/caching servers.

tomjen3• 13 years ago

No. The web is about links, if you don't want things to be linked too, don't make them accessible to the general public.

liotier• 13 years ago

> The article makes it sound like every webmaster is entitled to see Referer headers. [..] Why do you think it matters whether I came to your site via Google, Reddit, HN or someone else's blog ?

As an occasional small-time blogger, I like to know where the conversation happens, where people are interested - it enhances my engagement with my readers. As a reader, I'll gladly grant webmasters that courtesy.

pbreit• 13 years ago

Uh...I don't think anyone, OP included, is suggesting it's a right. Just that it used to be almost always available. And it's going away, sort of accidentally. On the scale of "useful things" vs "privacy invasions", it's pretty safely on the former side.

tarkin2• 13 years ago

I get referrers from httpS://www.google.com quite often in my logs.

And I'm running my blog on HTTP.

How is this so?

slig• 13 years ago

Looks like someone didn't RTFA.

ZoFreX• 13 years ago

Read the article. This is explained about 80% of the way through.

leephillips• 13 years ago

You can selectively block referrer information if you are using a webkit browser (Chrome, Safari, etc.) by using http://lee-phillips.org/norefBookmarklet/

I've found the information in my referrer logs quite interesting and useful, despite the Russian referrer spam, and am sorry to see it going away.

zapt02• 13 years ago

If you enable SSL for your site, will you start seeing referrers again? Mainly interested in the case of Google searches.

j_s2• 13 years ago

No. You have to pay if you want referrer info from Google.

http://searchengineland.com/google-puts-a-price-on-privacy-9...

tacticus• 13 years ago

Or run https on your server.

jmillikin• 13 years ago

Yes. I just tested it by searching for [haskell dbus] and clicking a result link to my site. Here's the referer in my nginx log:

  2620:0:1000:3509:ed78:b06d:4942:90e1 - - [28/May/2013:17:35:00 +0000] "GET /software/haskell-dbus/ HTTP/1.1" 200 5819 "https://www.google.com/search?client=ubuntu&channel=fs&q=haskell+dbus&ie=utf-8&oe=utf-8" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:21.0) Gecko/20100101 Firefox/21.0"

Where did all the HTTP referrers go?

Related Stories