Google Has Ended a Bunch of Free Data

http://finance.yahoo.com/news/google-gone-dark-search-giant-142955454.html

by Brajeshwar • 13 years ago

55 76 13 years ago

finance.yahoo.com

mfincham6• 13 years ago

If this discourages "marketers" from using the web it sounds like a win to me.

clavalle• 13 years ago

I generally don't like SEO and I'd agree with you if I didn't know so many clueless small business owners (I mean that in the least offensive way possible -- marketing is just not their strong suit).

So many people have something really cool to offer but their message is unfocused or overly complicated or just not elegant. Keywords offered a relatively simple and cheap way to see what people who might be interested in what they have to say/sell are looking for and hone their shop window to the world.

Now they get to pay Google for the privilege.

It is nice that Google is trying to get rid of the disease I just hope they don't take too much of the good flesh with it.

mfincham6• 13 years ago

Did having the visibility of search keywords ever really help people "optimise" their page text for increased business?

I am far from an expert in this, but wouldn't you only get to see the search queries of people who found the site already? How do you get from that to knowing what to change? :)

I did always enjoy looking at the AWStats output for my old web pages to see what hilarious pornographic keywords people were using to accidentally find me.

vdaniuk11• 13 years ago

Not again. If marketers stop using the web there is no web. There is no Google, bing, twitter, Facebook, reddit,stack overflow, hordes of startups that use ads to monetize, plethora of blogs and therefore VC companies and whole startup ecosystem would bust.

Do I want a more efficient system of transfer of commercial information? Yes. Is there is better system than marketing at the present moment? No.

One of my hats is a marketer and I am shocked how the industry is misrepresented here on HN and on reddit. The level of arguments is comparable too.

quesera6• 13 years ago

If marketers stop using the web there is no web.

There was a web before marketers.

I liked it.

mfincham6• 13 years ago

Those big centralised choke points for information (Google, Facebook, Twitter et al) we could do without. I'd strongly favour a non-commercial distributed solution to those problems.

jezclaremurugan• 13 years ago

People are freaking out -> Marketers are freaking out. Ended a bunch of free data -> Increased users' privacy.

Seems like a win to me.

vdaniuk11• 13 years ago

Yeah, marketing is bad right? Google is evil unless it takes something from marketers, then its good. The fact that startups use the same data, who cares? Actually, I think it will fuel some interesting innovation on traffic sources attribution, but I just can't comprehend this hate for "marketers".

dmix2• 13 years ago

My side business depends on SEO, but I also care enough about privacy to use DuckDuckGo (which also doesn't share keyword data), so I'm ok with the decision overall. Webmaster tools is pretty sufficient.

Privacy has trade-offs. Marketers/data miners will have to adapt to it if their customers demand it.

ams61102• 13 years ago

I almost never click on links, I retype them or copy/paste them into the search bar. You really have no idea where you're about to go or what extra data will be tacked on otherwise.

Edit: sorry this is unclear; I am talking about search engine result links. Of course within a site or app I do click links though even then I often glance at the status line to see what's about to happen.

chc2• 13 years ago

Yeah, I hate those dirty marketers who wrote the HTTP spec. I'm sure glad Google is thwarting the evil IETF's schemes.

Let's be real here: the biggest marketing company in the world still has all this data and way more. They are simply forcing site owners to pay to get at it now.

vdaniuk11• 13 years ago

Yep, while I have objections to most Google criticisms, I totally agree that killing off search referrers is bonkers. I guess there will be some starttyups springing up that will try to correlate your search rankings and page visits, creating fuzzy but actionable stats concerning an search keywords.

AJ007• 13 years ago

How are people finding your web site? Who knows.

Just because you own a web site doesn't make you a marketer or even looking to turn a profit.

Google is going dark because they are scraping web sites and keeping their visitors on google.com. Take a look at Google Images today verse a year ago. You can also see glimpses on certain mobile searches related to wikipedia keywords where they outright scrape the page, and the information boxes they show on the right hand column for certain terms on desktop.

Good business? Most likely.

DougWebb3• 13 years ago

How are people finding your web site? Who knows.

Just because you own a web site doesn't make you a marketer or even looking to turn a profit.

Owning a web site also doesn't give you any right to know how people are finding your site without you having to ask them to volunteer the information. That's what businesses in the real world do: they ask you, and you choose to tell them or not.

x0x0• 13 years ago

perhaps, if you are so concerned about privacy, you shouldn't be using the site period; are you now going to claim that the site owner has no right to see how people use their website?

but oh! google is happy to sell that data

logn2• 13 years ago

That's good news.

I'm not sure which scares me more. Marketers amassing surveillance data or the government. At least with the government, we might reign in the practice. Private industry seems to have some blessing from the public to do anything and everything profitable.

brudgers2• 13 years ago

Governments have an exclusive sanction to employ violence. Private entities do not.

001sky3• 13 years ago

This is true, but not really. Governments only have sanction to enforce the law. Anyone can behave illegally or unethically. Increasing your surface area is not street smart.

Just the opposite is true. People that live in large cities (and subject to random crime) adopt a strategy of blending in, not one of "sticking out and relying on their lawyers"...

uptown• 13 years ago

Blackwater (and other similar entities) would probably disagree.

joelrunyon2• 13 years ago

How do you feel about google still having that data?

They're not getting rid of it - they're just selling it now (most likely to the highest bidder).

thisisnotatest• 13 years ago

they're just selling it now (most likely to the highest bidder).

[citation needed]

logn2• 13 years ago

The article said they're making some attempt to anonymize the data. Anyhow whether that's wholly true or not, I don't like google having the data either. However, I do get that it's fair on some level since I'm directly using their service and it's been always apparent they monetize it through the data we inherently give them.

Where marketing and private intelligence bothers me is when it's not obvious to consumers when they're providing data. And also it's not always expected the ways in which it's used. And in the increasingly interconnected world of marketers selling and sharing data, the full implications won't be realized for some time.

edit:

Recently it's bothered me more and more that the software world tries to find hidden ways to charge users. It would be like highway construction companies tracking traffic patterns by embedding data sensors in their roads to collect consumer data and while they're at it, share it with the government to win some favors. Or like your carpenter going ahead and embedding sensors into the drywall to collect your data to resell and help them generate more revenue. People have paid for stuff directly for a long time and been happy to. But internet companies have this thirst to get millions of users by offering something for free, gambling investor dollars, and hope it pays off huge (which it occasionally does).

anaphor3• 13 years ago

It looks like they just enabled SSL for all users, which is defnitely a GOOD thing. These SEO people are confused because I guess it limits some data they would've normally got, but it's a byproduct of them turning on SSL. It's not some campaign by Google to limit SEO data except insofar as it gets limited by SSL being on.

dmix2• 13 years ago

No I believe SSL is a separate feature that was always-on a while back (or at least optionally so).

This feature creates a redirect which hides the original keyword in the referring URL. Which has always been available at https://encrypted.google.com but now has become default at google.com.

The encrypted subdomain confused people, considering it's only adding a redirect, not more encryption.

dangrossman6• 13 years ago

This isn't a byproduct of turning on SSL. If both Google and the site linked to by Google are secure, then the referrer would be intact, and contain the search terms, which is where Google Analytics and every other tracker has picked them up for 15+ years.

Google purposely added another level of indirection between their search result page and the search result links themselves. When you click, a bit of JavaScript rewrites the link to a script at google.com/url in order to wipe the real referring URL. Should Google want to do that for reasons other than masking search terms, like recording clickstream data from their SERPs, they could easily add the search term to that google.com/url link's query string. That's exactly what they do for ads -- they transfer search terms through the redirect's referrer even on SSL connections.

anaphor3• 13 years ago

It's been this way for a quite a while though, now the only difference is that SSL is enabled for a larger number of people. Also adding the search terms to the url would defeat the whole purpose of using SSL for search at all.

nwh• 13 years ago

The /url redirect has been there as long as I can remember. It's just well hidden from view.

mikegirouard• 13 years ago

and contain the search terms

This would be amazing. Can you cite a source I can use?

I made the mistake of launching a [previously heavily optimized] site on the very week the secure switch took place. Explaining this to the client has been a complete disaster for me.

gabeio• 13 years ago

Plus it's not like it would have really been helping the SEOs all that much. Though they really shouldn't have been relying on it as mush as they were clearly.

lstamour2• 13 years ago

"If a website is accessed from a HTTP Secure (HTTPS) connection and a link points to anywhere except another secure location, then the referrer field is not sent." http://en.wikipedia.org/wiki/HTTP_referer whose source is "Hypertext Transfer Protocol -- HTTP/1.1: Encoding Sensitive Information in URI's (RFC 2616 § 15.1.3)". IETF. June 1999. Retrieved 2013-03-20. "Clients SHOULD NOT include a Referer[sic] header field in a (non-secure) HTTP request if the referring page was transferred with a secure protocol" [1]

Easy fix: Enable SSL everyone, stop "freaking out".

[1] https://tools.ietf.org/html/rfc2616#section-15.1.3

Edit: I see comments further up indicating that Google redirects from SSL to non-SSL, thereby obfuscating the referrer and that this is new. If so, I'll have to take back this statement, sigh.

dangrossman6• 13 years ago

The search term is not in the referring URL, SSL or not. Google removed it from organic search, by a combination of moving it out of the query string into a URL hash, and by adding a level of redirection between the search result page and the linked website.

lstamour2• 13 years ago

Ahh true. Instant did change things, didn't it? I'm still used to omnibars that hide the URL from me on Google searches that I forget it's not all /search?q= any more. Good point, good point.

Though, given Google's native apps, perhaps this was inevitable? I mean, there was always a percentage of traffic that would seem organic yet be from Google...

gfodor3• 13 years ago

It's kind of stupid to react positively to this news just because you hate marketers or ads. The truth is that incoming keywords from Google provide a whole host of functionality and analytics for site owners as well. (I realize they offer Webmaster Tools, but the fact is that getting incoming keywords for individual visitors provides all kinds of leverage for personalization, improving SEO, and understanding user behavior that aggregated, offline reports will never come close to doing.)

The fact Google is still providing this data to people who do paid advertising shows this isn't about privacy at all, but is about closing off a valuable source of information for some other reason, most likely just under the guise of privacy.

At a high level, it's not clear that this type of information being "leaked" to sites was ever anything other than a function of the way Google worked from v1, where the query was passed as a GET parameter. So it's debatable if this was a good design decision in the first place. But Pandora's box was opened. Due to this design decision an absolutely immeasurable about of code has been built around this source of information (for good and evil) and calling this a "data apocalypse" isn't really that hyperbolic for people relying upon it.

mortehu• 13 years ago

The fact Google is still providing this data to people who do paid advertising

By "this data", I presume you mean "traffic data grouped by keyword"? Or are you saying that the search query is passed unfiltered along with the request for individual ad clicks?

AdWords can provide aggregate keyword traffic data easily without compromising privacy.

In order for Google Analytics, and similar software, to provide aggregate keyword traffic data, the data would have to be bundled with each request. The more data a website receives about you simultaneously, the less privacy you have. Previously, a website would get your IP address, your user agent and your search query, and at the same time have the ability to inject a tracking cookie. The IP address I use is registered to me in RIPE, so in my case it would be trivial to connect my search term to my name and address. If I visited unencrypted sites, secret agents monitoring the Intertubes would be able to store my search queries. Is this kind of data really available in AdWords?

gfodor3• 13 years ago

I actually haven't verified personally if the keywords are passed along with ads, I was going by the claims made in the article.

If you are concerned about your keyword data being aggregated and stored based upon referrals and associated with IP, I have bad news for you. This data has been being aggregated and cross referenced not only by sites and advertisers but also third party aggregators for more than a decade.

woah• 13 years ago

It's a contradiction, according to Rishi Lakhani, a search consultant:
... their idea of privacy is ridiculous to say the least. You can't offer privacy, but still SELL the data to AdWords advertisers. It's the same user. It's the same action.

The idea that clicking on an ad is the same thing as clicking on a link is where I'll have to disagree with this guy.

gfodor3• 13 years ago

This is a pretty shallow analysis of his point. His point is that insofar as it related to the privacy of propagating keyword information, the action of a user clicking an ad or an organic search result doesn't seem any different. If the keywords a user searches for is private information, then why should this privacy be off the table when they click on an ad?

mwww• 13 years ago

Google wants website owners to use Webmaster Tools so that it can connect websites with individuals. The end goal is to make it harder for people to manipulate search results in a way that would result in an inferior user experience for searchers.

For many years, Google has been a gold mine to a profession called "SEO". It still is, but every single day it is getting harder to extract value from it if all you do is trying to get your site to the top of the SERPs. Eventually it will be very expensive to trick Google and this is why I see many professional SEOs starting work at big brands or doing consulting for them. It's easier to get good rankings for a site that provides great value to users than for another spammy endeavor.

I was also involved in this business many years ago, but mainly to rank my sites that were getting traffic anyway because of the value they provided to users. Instead of being angry, I think that SEOs should be thankful that Google made it possible for them to take advantage of their service and even helped them along the way — I am. There is nothing wrong in charging people money for services that they use to make money themselves.

SEO should always only be a tool that helps you reach users with your quality product, not your main way of making a living because in itself it creates no value to society.

vdaniuk11• 13 years ago

Ugh. Someone is wrong on the internet and I just can't ignore that.

SEO doesn't create value for the society? That is an extremely bold claim. Guest posts, infographics and other form of viral content are a very common tools for so these days. They create immense value for content consumers and that's the point. SEO is very much white hat these days in the anglosphere due to superior Google search spam detection algorithms. There are black hats, of course but they are not a defining market force for a long time already.

guiambros• 13 years ago

If this is about privacy, what about stop capturing search data at all? Eliminate all user personalization and search history, as if you weren't logged in. No more privacy issues, NSA court orders, etc.

Of course, this won't ever happen. This is not about privacy. This is about posing as a company that cares about privacy. It's easier to deal with some upset marketers rather than governments pushing for stronger regulations and anti-tracking measures, given the total lack of self-regulation (third-party cookies, anyone?).

r0h1n• 13 years ago

So do unto Google as Google does unto others, in a way at least.

I perform all my Google searches while remaining logged out from Google. More importantly, I use Firefox/Chrome plugins that rewrite Google search results minus their tracking. In a way, I've gone "dark" to Google too.

samsolomon• 13 years ago

FYI search terms have been disappearing since a Google update in October 2011. More than 95 percent of keywords have been marked as "Unknown" on my sites for the last 6 months.

You can still make a good guess at what terms are leading people to your site by the number of impressions on Google you get for a certain search term. To discover this go to Analytics > Acquisition > Search Engine Optimization > Queries.

Google Has Ended a Bunch of Free Data

Related Stories