Google's new Ajax-powered search results breaks search keyword tracking for everyone

http://getclicky.com/blog/150/googles-new-ajax-powered-search-results-breaks-search-keyword-tracking-for-everyone

by imp • 17 years ago

88 80 17 years ago

This is a potentially huge problem. Google is effectively cutting off all other search analytic packages from their information.

jorgeortiz852• 17 years ago

No, they're not. It's not like they're deliberately obfuscating the information in order to shut everybody else out. They've made a technology decision which will probably significantly speed up searches and result in increased revenues for them. Google shouldn't be beholden to other people's business models.

That said, given the amount of referrals that Google generates, and how important referral information is for a lot of things online, as a courtesy Google should probably communicate this to the web-at-large before rolling it out on a massive scale. However, they're not "cutting off" other search analytic packages. At worst they're forcing a rushed bugfix.

cakeface2• 17 years ago

Yes they are cutting off other search analytic packages. If there is no tracking of the search term in the referring url then the analytics package has no way to determine what keywords were used to bring a user to the page.

Since google provides one of the most commonly used analytics packages, Google Analytics, and owns another, Urchin, I would think that they would be the last ones to try this.

sant0sk13• 17 years ago

They aren't obfuscating the information, they're removing it altogether. As an analytics provider, you cannot 'bugfix' a lack of information. You're just hosed.

Probably what would have to happen is for the browsers to allow for passing data after # symbols the same way they do after ? symbols. That means updating every major browser to support this. At the speed of browser development, many analytics providers may be out of business by the time that happens.

sam_in_nyc7• 17 years ago

Who is this a problem for? It's going to improve search results because now SEO gamers have less data to work off of. If you're a website, you should know damn well what keywords your website attracts... and you shouldn't be tweaking things around just to trick Google into sending you more traffic.

Now web sites can focus on creating content, and google can focus on creating good search. Both parties will benefit at the cost of spammy SEO gaming websites, who are now blind and will be pushed to the bottom.

sant0sk13• 17 years ago

Just wow. I'm really starting to anti-trust Google.

modoc2• 17 years ago

Really? I mean how much should they hold themselves back in order to maintain support for other products?

I know they're huge, but it's still a free product/service, they shouldn't have to retard their progress to avoid breaking other products. If they decided to stop supporting IE (while potentially bad business) I'd support their right to do so.

What exactly do they owe you?

sant0sk13• 17 years ago

Yes they have the right. No they don't owe me anything (save the service I pay them to provide, which is moot to this discussion).

But it is my trust and they can still lose it. The fact is that I believe this to be shady, and for a company whose motto is "Don't be evil", they raise the standard for trusting them. A standard which I don't believe they are meeting.

axod• 17 years ago

They don't owe you that information, sure it's nice I guess...

In any event, the only way I can think to do this with an ajax search is to pass every outgoing link click through a redirection script, so that the referrer is rewritten to something useful.

However, whilst improving things for webmasters, that would seriously negate the point of using ajax, and be a pain for end users.

Google are serving their end users. I don't think trust comes into it TBH.

What would you rather google do? not progress and use ajax search, or use a redirection and make a worse experience for end users?

ryanwaggoner• 17 years ago

Unless I'm way off, Google is already doing a redirection script on all outbound clicks. The endpoint url shows up in the footer of the browser on hover, but if you right-click and copy the link location, you get something like:

http://www.google.com/url?sa=t&source=web&ct=res&...

coderrr• 17 years ago

Here's a crazy hack they could do to use ajax queries while preserving the current referer url:

Whenever the ajax search is performed an invisible iframe is loaded at: http://google.com/search?q=search+term&donothing=true.

The donothing=true tells Google's webserver to perform no actual query and return a blank page. This should happen extremely quickly. (It could be performed after the ajax results have been fully loaded as to not affect response time)

Then when a user clicks a search result, Google does a javascript redirect from the iframe (use javascript to add a new <script> tag with a window.top.location= in it) and you get a referer just like you would before... with a negligible performance hit.

th0ma52• 17 years ago

the standard is the referrer being in the HTTP headers, not specifically that if the referrer is google, that it must include the search terms used in their search

we seem to be putting more reliance on these flimsy things... in the future will i create an outage because i @somebody instead of #something on twitter?

ComputerGuru• 17 years ago

In the case of web browsers, we don't even SEE the stuff that comes after the #

So it's not like putting "@someone" instead of "#someone" but rather like putting "" instead of "#someone"

See the difference+

mbreese• 17 years ago

Not really no. If the web browser isn't sending the information, then take that up with your browser... it's the one that has the information and is dropping it.

Then again, I should point out that I'm not seeing this behavior, so this may be much ado about nothing.

th0ma52• 17 years ago

sorry, maybe a bad example on my part, my point is that querystring parameters being passed should include the search terms used on that search engine is something of a rule of thumb more so than a standard... but i don't know that there are any standards built on top of the http referrer other than this is how google and many others operated, so these reporting companies wrote code that relied on that, rather than coding to a standard.

ewiethoff• 17 years ago

Hmmm. I'm not seeing this new "feature" when I search Google. I've turned on Javascript, I'm accepting Google cookies, I've even logged in. And I still see the usual http://www.google.com/search?q=ajax+crap results URL. I'm running Firefox 3.0.5.

Anyone know what triggers the new search results?

ivank• 17 years ago

Google selecting your cookie for the test run.

raquo4• 17 years ago

Probably (and hopefully) it's just being bucket-tested...

dchest• 17 years ago

See how there's a hash mark # in there now, and the "q=test" is after it? The problem is that web browsers don't send anything after the # in the referrer string.

I wonder why they don't send it in referrer string. Are there any technical or security reasons?

lacker4• 17 years ago

Originally the content after the # was only used for automatic scrolling to particular tags within a page, so there was no need to ever send it to a server. Once Ajax came around, people realized you could use javascript to read the url after the # and do arbitrary stuff based on it. So you can show different pages at

  foo.com/
  foo.com/#bar

but since the server doesn't treat them differently, when the user transitions from one to the other, there's no page reload. So you can move from page to page, but cut n pasting links still works.

There are libraries like RSH that let you do this to speed up your site.

sqs• 17 years ago

Seems to me that browsers ought to start including the fragment (the part after the #) in the referrer, now that different fragments can refer to entirely different pages. I can't see any security issues, as the previous poster was wondering.

peregrine• 17 years ago

Google is a business who wants you to use their analytical service over others. They have no obligation to you send you the information their users are using to find your site.

gojomo2• 17 years ago

Google is also arguably close to being a effective monopoly in web search. Antitrust law prohibits using monopoly power in one market (eg search) to acquire a monopoly in another market.

So there's a bit of a bind for Google: strategies that were perfectly legal when they were up-and-coming become illegal at some point of dominance. But there's no 'bright line' for when the rules change -- market definition, thresholds, and acceptable practices can all be argued after the fact.

A naturally competitive company is almost certain to keep the 'pedal to the metal' until it's too late.

madh• 17 years ago

Is there ever really a "too late"? Shouldn't Google push for dominance as far as it possibly can? Isn't it some other entity's role to stop them?

patio11• 17 years ago

Google is also arguably close to being a effective monopoly in web search

Yeah, I think we're past arguably these days. I've thought that for a while.

I think arguably it was thrown into sharp relief when a bug at google.com required me to stay up past midnight trying to explain to my customers that

a) yes, I understand, the Internet is broken

b) I understand and respect that you are pissed off at this occurrence but I didn't do it

c) I empathize with your desire to have this fixed RIGHT THE '#($ NOW.

Google, you set out to dominate the Internet. You succeeded. You are now indispensable. Please do not break the Internet.

sam_in_nyc7• 17 years ago

A change of this magnitude is not to plug their analytics service. It's to improve search results by cutting off the feedback they're currently giving to SEO gamers. Read my other comment below.

three14• 17 years ago

Many of the comments seem to assume that websites are entitled to know what I searched for. I think that it would be a great step to give millions of people a little more privacy, even if it's just an accidental side effect of a change to Ajax.

dejb2• 17 years ago

I feel that the benefit in letting webmasters know what their users are looking for outweighs any privacy gains to be had. If people are that concerned about their privacy they could always set their browser to not send the referrer. I doubt that this move has anything to do with concern over user privacy.

tjpick2• 17 years ago

making it about privacy is playing an emotional card when it should really be a technical discussion. Hiding a query string isn't "privacy" in any meaningful sense. It's beneficial for you to tell a webmaster what it is that you have come looking for.

I'm in two minds about the issue though. Google don't have a contract to supply the searchterm on outgoing links (via the referrer or otherwise) but for many applications it was a pretty useful thing for them to do. Arguably their only resource there is a single search page which is dynamic depending on your query string - to me this says ?q=searchterm is the semantic way to represent it, but hey it's their application so they can "design" it however they want. On the other hand, urls are about as explicit as you can get on the web and google's are more publicised than most. Changing your public API is asking for trouble.

semiquaver3• 17 years ago

It's beneficial to the webmaster, not the user. If most people were presented with a choice of whether or not to provide this information, they'd decline.

fhars• 17 years ago

I would even support this argument if google wasn't evil and wouldn't store personalized search logs themselves. Then this would indeed be a privacy win. The way things are now this smells suspiciously like an attempt to further monopolize useful data that were available to webmasters before and kicking out competitors in the web analytics space. You might argue (with good reasons) that these data should have remained private from the beginning, but then google should stop logging them first and only then remove them from the results. The other way round it's "do as we say, not as we do."

rgrieselhuber• 17 years ago

Surprised to see this as the keyword reports are a big part of Google Analytics as well.

raquo4• 17 years ago

I can't really see how google can easily pass the search query to Google Analytics. GA requires atomic data to perform all this analysis and custom segmentation magic.

The only way I see is using a cookie (with a user id) on the google.com domain, which ga.js must interpret in backend on every request. But this means that ga.js will have to be dynamically generated, rather than being a simple static file (and yes, it won't be cached by users visiting your site, making tracking even slower). I don't think google wants this additional load every time anyone loads some GA-enabled page on the web. Besides, ga.js is currently hosted on google-analytics.com, not google.com, so all users will need to upgrade to a new tracking script. And it will not be possible to self-host the script anymore.

Anyway, this does not sound like anything reasonable to me. I hope it won't be implemented - it would destroy an entire industry, and not a redundant one.

schammy6• 17 years ago

When you click a link from a google SERP, it goes through a redirect script. They could easily be using that to feed Google Analytics.

raquo4• 17 years ago

Yes, but in that case other web analytics packages will also have access to referrer that contains the search query and there is no problem.

I wasn't clear enough, sorry - I was concerned with how Google could be able to share the search query _only_ with GA but not other analytics tools, because it is clear that Google will not sacrifice GA usefulness to some ajax SERPs.

sam_in_nyc7• 17 years ago

ga.js can be static, and can read cookies.

You go to Google and do a search, Google stores your search sessions on the server, and assigns you some unique ID. If you click on a search result, google logs that in your session.

You visit site with Google Analytics. ga.js reads the Google.com cookie to get your session ID in some clever way (perhaps embedding an invisible iframe in the Google.com domain, which can read the Google.com search cookie). Now, Google Analytics knows that you're the same person who just clicked on that search result .2 seconds ago... a match in my book.

They could also tell you things like "visitors to your site also search for XXX" but that's clearly a violation of privacy. They get this data, but we don't.

braindead_in• 17 years ago

Nightmare scenario. What if analytics goes paid?

redorb• 17 years ago

I just looked at the Source for google.com's new search result page, You can't read 1 word of english off it - totally Ajax. Looks amazingly complex.

parenthesis• 17 years ago

Is anyone not getting this on google.com ? It looks like it might be happening for everyone on google.com, but not on, e.g. google.co.uk or google.it .

raquo4• 17 years ago

I don't get anything ajaxy on google.com

Even if I go directly to http://www.google.com/#q=test it redirects me to http://www.google.com/search?q=test&cad=h

semiquaver3• 17 years ago

http://www.google.com/#q=test sends me to a blank google search with no results.

tlrobinson• 17 years ago

Google could easily fix this by redirecting through some sort of shim script. But will they?

andrewf• 17 years ago

Google has ~65% of the search market.

The other 35% is probably still representative enough for you to figure out what the population is searching for that leads to your website.

imp• 17 years ago

The problem with that is that your search ranking for any given key word is probably different in each search engine.

randomwalker• 17 years ago

I believe the problem is bigger than what the article points out. Under the new scheme, all search results pages have the same URL. The part after the hash not part of the URL, it is a fragment or anchor.

It breaks my firefox extension that nukes searchwiki. It breaks my custom CSS for the search results page (see userstyles.org). I often bookmark google search results in del.icio.us. Depending on how such services are implemented, different search results pages might show up as the same page.

I'd guess there's just a lot of code out there that assume search results are separate URLs, and that's going to break.

Those two breakages are simply due to the regex identifying the search results pages not matching anymore, and are easily fixable.

sam_in_nyc7• 17 years ago

Search results to the old ?q= google will still work:

http://www.google.com/search?q=test

sam_in_nyc7• 17 years ago

There are a variety of things which I don't think anybody has noted. First of all, for those without Javascript, the old Google page is used. So, you'll still see some sampling of keywords coming in... albeit only 2% of what it originally was. So it's not totally ruining things. You'll still get some sample of referring keywords.

The benefits to Google are more than anybody is reading into. It is definitely an intentional move to cloak their redirects and keywords. And it definitely increases performance for them.

Performance gains:

The back/forward buttons do not cause anything to be reloaded. This should save a nice percentage of page loads which were along the lines of "what was my last search?" and going back and forth between pages.

If you look at Firebug, the Ajax request is actually going to the proper google search page... google.com/search?q=... Google is taking the entire search page and essentially replacing it without a page refresh. This doesn't really mean anything significant, it's just a neat observation.

Strategy:

More than anything, in the long run this will improve search results. It's not about locking down the "analytics" arena... as one poster noted, they probably won't be able to show Google search terms either (unless they set a cookie on the Google search page, and read it on GooAnal [yes, GooAnal] pages, which might actually happen). The important thing is: SEO gamers now do not have access to which google keywords work and don't work. Without giving SEO gamers which keywords are working and which aren't, it's going to be a lot tougher to game the system. These SEO firms that manage to get the spammiest crap on the top pages won't have much to work off of other than the raw amount of visitors coming from google. I imagine that Google has secretly rolled out quite a few significant changes in their search algorithm as well, to work in tandem with keeping analytics in the dark.

My prediction is, in a few months, you'll be thanking Google for the much better search results. I welcome this change with open arms, because I'd much rather see quality content rise to the top, rather than gamed content targeted at keywords. The trade-off that people running websites won't be able to see referring keywords... well I guess Google might lose .001% of their market share.

schammy6• 17 years ago

"My prediction is, in a few months, you'll be thanking Google for the much better search results."

I don't really think this is going to change anything. How is this going to affect SEO? I doubt any of them use referrers logs - waste of time. They scrape the SERPs. The old SERPs are still available. They can tell when a rank goes up or down.

How is this going to change that?

sam_in_nyc7• 17 years ago

It's one thing to tell what rank you are on what keyword, but quite another to track how much traffic you are getting from a specific keyword. Though I guess there's ways to estimate that.

You bring up a good point, though. Guess it's not as grand a scheme as I thought.

lacker4• 17 years ago

Disclaimer: I work at Google.

This is not about analytics. This is just an experiment to test if using ajax instead of the typical form submit can make the result page load faster.

Here's an official statement -

  We’re continually testing new interfaces and features to enhance the user experience.
  We are currently experimenting with a javascript enhanced result page because we believe
  that it may ultimately provide a faster experience for our users. At this time only a
  small percentage of users will see this experiment. It is not our intention to disrupt
  referrer tracking, and we are continuing to iterate on this project.

http://searchengineland.com/google-ajax-search-results-death...

YoavShapira• 17 years ago

Nice to get a comment from Google right away, though.

Google's new Ajax-powered search results breaks search keyword tracking for everyone

Related Stories