Why We Desperately Need a New (and Better) Google

http://techcrunch.com/2011/01/01/why-we-desperately-need-a-new-and-better-google-2/

by vyrotek • 16 years ago

222 184 16 years ago

The argument being that Google is loosing the war against spam. A new and better Google will likely be Google itself. What we really need is a way to discover content that's not search.

meterplech• 16 years ago

For many things there is utility to social content finding. Sites like HN work for news/discussion, and Twitter and Facebook work for a random amalgam of things. I'm interested in the future of social search startups that somehow curate content from friends. In the articles example- asking friends if they liked their Dishwasher, and if yes what brand it is. That's the most like how people IRL make these decisions. I know there are some startups in this space as well, hope they do well!

nhangen2• 16 years ago

Furthermore, I'd love to see a simple thumbs up or thumbs down system integrated into the browser or Google search.

Did this page help you find what you were looking for? Was this page useful?

etc

ktsmith3• 16 years ago

Having just bought a new washer and dryer I can honestly say that my friends opinions were not a factor in the decision at all even though we did speak about it before I made the purchase. The main reason being other than some superficial elements of the appliances my friends can't really speak intelligently about their own appliances. They tend to not remember beyond the initial purchase point what many of the features even are. For example I asked a friend if his washer contained a heater and while it does he didn't even have a clue that it did.

I have a relative working at Home Depot and we just discussed this topic at length given my own recent purchase. Most of their appliance customers come in with some idea of how much money they want to spend and what basic feature set they would like. Then they are looking for a knowledgeable sales associate to explain to them the differences and benefits of the various models. They may have "heard from a friend" that a particular model was good or a particular feature was good but even that information is usually incomplete or incorrect. I would fully expect "social" results for these types of queries to result in even more misinformation.

binomial• 16 years ago

Working on it - http://jellly.com/

Also, just got a Firefox addon working that provides automatic related links to most web pages. Unfortunately, it relies on search now.

But I'm sure that any sufficiently popular content-discovery tool will have a lot of spammers trying to game it. It's not easy to fight that.

javadi82• 16 years ago

I am sorry; I do not understand how this is different from the "Similar" link that is placed beside the "Cached" link on Google's SERPs.

Could you please elaborate?

dawgr• 16 years ago

How would it be possible to discover content without searching? If I want to discover content about something I have to tell the computer what that something is about, that would be searching. If you are talking about directories, then I wouldn't use that.

replicatorblog2• 16 years ago

It will be interesting to see how this impacts the Android/iOS battle. Search revenue funds almost all of Google's other activities so if people start using other search engines or find alternate ways to get their content it could impact the level they can spend on phones.

With a push to a mobile first world the Android model is especially sensitive to spam. On a full size browser you have a lot more context and results for a given search. 5 Results may be spam, but you can work around them. If the average phone screen shows 3-5 results and all of them are spam you will quickly find alternate tools.

Google ignoring spam is like Microsoft ignoring the cloud.

petervandijck3• 16 years ago

"Google ignoring spam is like Microsoft ignoring the cloud." Great line!

zachrose• 16 years ago

That's the kind of line that I wish I could email to people in 1960 as an important business insight from 2011.

Matt_Cutts7• 16 years ago

The article calls out two specific companies as "landfill in the garbage websites that you find all over the web." Reasonable people can disagree over whether such content is truly spam or low-quality content, and thus how to respond.

replicatorblog2• 16 years ago

What is the difference and how should they respond? It seems to be a rising frustration among power users that Google is increasingly becoming a wasteland populated by spam. For example, Marco Arment recently commented on his podcast how hard it was to find answers to simple questions on Google these days. He was saying that the content farms have basically created a page for every PHP function with thin content and rendered it useless. For a company whose goal is to index all human information it is a pretty big warning flag.

What is the appropriate user response? Go to Stack Overflow? Find a branded knowledge base like O'Reilly's Safari? I'm genuinely curious to know what we can do.

ddemchuk6• 16 years ago

Sounds more like a semantic excuse to justify the proliferation of adsense riddled spam content mills that are chasing keyword search volume and making Google a hefty sum each year rather than a legitimate conundrum.

gojomo2• 16 years ago

If we were to pick a random Demand Media or Associated Content page, what are the odds a reasonable person would prefer its writing to a less-cynically-optimized, less-ad-drenched alternative page that the test page has bumped out of the top results?

fanf22• 16 years ago

Neither spam nor low quality results should appear on the first page of results. I don't see how the distinction necessarily changes how you treat the useless sites.

tomjen3• 16 years ago

It won't. Whoever comes up with a better search engine is going to get a large check from Google, and a monster check if he can make MS and Google go into a bidding war over the business.

A better search engine is not what will do Google in, because they would understand the danger it posed to them. What will do Google in is a business which they don't understand that can kill search engines as a place to do business.

EGreg5• 16 years ago

<cough> facebook </cough>

klbarry• 16 years ago

Isn't the issue, of course, that spammers have no incentive to game other search engines since they're not worth the time? Any search engine that gets big will have the problem.

bambax• 16 years ago

Came here to say this but you said it before! ;-)

Also, "crowd-sourced curated lists of websites" sound like the old Yahoo directory of yore. They will either become obsolete very quickly or spammers will find a way to penetrate and dominate them.

jerf• 16 years ago

I believe the Yahoo lists were actually centralized. Hypothetically a fully-distributed approach could work better, in the style of Wikipedia. Realistically, well, I have no idea but it's worth a try.

jshen3• 16 years ago

I've been wanting to build a decentralized curated social search (buzzword soup I know). The idea is that security has to be based on trust, and adversarial search is a security problem. Here's my idea, it starts with delicious style bookmarks and a social graph. the search engine indexes the pages you've bookmarked so your bookmarks are searchable. AND, you can expand your search to include your friends bookmarks. Obviously it wouldn't be comprehensive, but it would work really well for shared interests like programming.

didip2• 16 years ago

It could have been del.icio.us with its army of users tagging keywords on every URL.

The way del.icio.us does curation is smart in many level. Spammers need to create many accounts to affect tags they care about.

cdr• 16 years ago

Sure, it's an arms race. The issue is that Google hasn't been trying to keep up.

SoftwareMaven2• 16 years ago

Google's current incentives align with those content sites more than the searcher. Those sites provide a lot of revenue for Google. Until Google starts losing searchers as a result of those sites (unlikely, because my mom is unlikely to care that eHow is massively gaming things), those sites are going to stick around.

protomyth• 16 years ago

True, but another part of the issue is a pure algorithmic approach can be gamed, inside a search company human curation doesn't scale, and community curation has scaling problems / seeding problems / can be gamed.

I do think think that some human curation, if for nothing else to mark sites as copy spam, might be workable. Pattern matching is still humanity's territory.

Matt_Cutts7• 16 years ago

"Google does provide an option to search within a date range, but these are the dates when website was indexed rather than created; which means the results are practically useless."

I believe the author is mistaken on this point. Quick proof is to do a search for [matt cutts] and you'll see the root page of my blog. Click "More search tools" on the left and click the "Past week" link. Now you'll only see pages created the last week, even though lots of pages on my site were indexed in the last week.

amichail2• 16 years ago

BTW, why is it that Google has not opened up its core search engine to third party developers so that their code can be used to bring up some of the search results by default (without requiring the user to subscribe to third party features)?

Most people have never used Google's Subscribed Links feature because it is not enabled by default.

User feedback can be used to determine which third party code to use in which contexts. Spam/unhelpful features would be detected quickly. There would be intense competition among third party developers for highly desired features.

Something like this could give you Wolfram Alpha like features among other things such as custom UIs for various searches (e.g., travel).

The Google App Engine could be used for computation. You could pay third party developers by how often their code is used in search results.

Matt_Cutts7• 16 years ago

Just speaking for me personally, I'd be a fan of trying ideas like that. The potential danger would be that some code could introduce latency or other things that would make the search experience worse instead of better.

mark_l_watson2• 16 years ago

re: search tools search by last week, last month, etc.: very cool

acangiano• 16 years ago

Indeed, this is true when using the search tool in the sidebar which sets the cd_min and cd_max parameters in the query string. However, if you were to use the daterange operator directly in the search input box, you'd see all the content indexed in the last week.

jackolas• 16 years ago

The down side of this is that google doesn't hold them honest. I.e. reporting on 2001 events in 2000.

RP_Joe• 16 years ago

The date range is one of Google greatest features. Its not perfect but it is great. I would like to see past 2 years added to the list. Sometimes 1 year is not enough and it is to time consuming to type a custom date range. But it is wonderful at filtering out 5 year old content. This is a feature I use every time I am looking for code samples on the Internet.

DanielBMarkham3• 16 years ago

This is exactly what blogger Paul Kedrosky found when trying to buy a dishwasher. He wrote about how he began Googleing for information…and Googleing…and Googleing. He couldn’t make head or tail of the results. Paul concluded that the “the entire web is spam when it comes to major appliance reviews”.

So I happen to know somebody who is taking a small section of the home appliance market and creating content around it -- reviews, news, advice, a place for other consumers to talk to each other.

Of course to do this you need to have income, so they are going to use some sort of ad-supported model.

My question is very simple: is their project a spam site or not? To some, I guess it would qualify. To others, not.

You see, there are two questions when it comes to search results: 1) Am I being presented results that match the query I entered? and 2) Am I being presented results that match what I want to know?

These are two entirely different things. A third-grader looking for information on a movie star might find a games page with all sorts of information on that star -- all sponsored by some kind of adsensey stuff. And he's very happy. A researcher typing in the same question gets the same page? He's pissed.

There is no universal answer for any one question. It's all dependent on the culture, education, and intent of the user -- all of which are not easily communicated to a search engine.

Look -- this is a real problem. I hate it. Sucks to go to pages you don't like. All I'm saying is that it's more complicated than "we need a new Google" Finding what you want exactly when you want it is a difficult and non-trivial problem. We just got lucky in that Google found a simple algorithm that can be helpful in some situations. It may be that we're seeing the natural end of the usefulness of that algorithm.

</soapbox>

ktsmith3• 16 years ago

> is their project a spam site or not?

To me that depends a LOT on how they present the advertisements and what they do on pages where they don't have information for a product. My biggest complaint with so called review sites is that they are presented more advertisements than content. They also tend to have automatically generated pages for every model number you can imagine, including incorrect ones that they receive searches for. On those pages there tends to be links to shopping sites and prices for unrelated appliances and products. I absolutely consider that to be spam because they are content free.

The problem with google searches lately, and especially for things like appliances, is that the spammers and content mills are clearly winning. In my current search for a washer and dryer the manufacturers page was on page four or five or the results. The first few pages were flooded with bogus content pages, sale pages and unrelated pages. Trying to filter out shopping sites and explicitly target specific keywords and filter others doesn't help. There are a growing number of search topics for which google is simply broken.

DanielBMarkham3• 16 years ago

In regards to the style of the pages, I completely agree with you -- probably because we have similar tastes in content. Others, however, might not. I'm not saying that flashing text and floating images are okay, just that the threshold between "mildly annoying" and "total shit" is a personal thing.

The problem is, as you point out, that most people, most of the time, are beginning to see results they don't need or like when they type a search. This is a big problem for both searchers and the companies that provide search. If you create an algorithm for directing people's behavior (a search engine) folks are going to game it. You and I might not like it, but "gaming the way people do things" is called marketing in any other context and has been around for hundreds of years.

This leads me to suspect that no simple (or even complex) system of finding things for people is ever going to work for an extended period of time. It's a radar vs. radar detector problem. It's a natural competitive situation.

But it doesn't have to be all bad. From competition and fitness criteria comes evolution. Spammers and search engines will probably be a key part of how AI evolves. It'll be neat to see if we move beyond Bayes -- and if so, how would that work?

The one thing you bring up that's interesting is what to do with bad searches. How do you deal with a mis-typed part number? Should a system know which part number you have? If so, how would that be done?

I think the spammers covering all the misspellings are doing a service -- as long as the site isn't obnoxious and provides the user with the information they are looking for. We think of it as a failure of Google, but in fact it looks like a win: thousands of little spammers trying to find all the mistakes I make and providing content for them -- as long as they have my best interests in mind (and are not trying to trick me). I'll happily look at an advertisement for a Ford Explorer in return for valuable information on my 1978 dishwasher that I couldn't read the entire part number for. And I hate ads. I like that scenario a lot more than looking for a favorite mp3 for a cell phone ringer and spending the next 3 hours in spammer hell.

jshen3• 16 years ago

you failed to address the main point. Google is filled with duplicate content, aggregation sites, and content farms with shit content that has the right statistical profile.

this is not the same problem as different people perceiving results differently.

rhizome2• 16 years ago

Sure, but it can also have to do with different people perceiving inputs differently. While I lament the content duplicators with a passion, I deal with it myself with longer search queries. Grouping and requiring, excluding particular domains (like efreedom), and so on. I know we all probably do this to some extent, but I've found that, sure, you have to exclude 10 different domains, but it does winnow down the results in a useful search. I'd rather be able to search with three words, but for now those days are over.

spenrose• 16 years ago

"Unfortunately, it isn’t just appliance reviews that are the problem. Almost any popular search term will take you into seedy neighborhoods." Quantification? Detailed examples? Link to his research? All missing in the original article.

mattmcknight2• 16 years ago

My thought when I read the Kedrosky thing months ago was very similar. Many times the reason you aren't getting good Google results is that there isn't good indexable content. Consumer Reports is a leader in this space and they are behind a paywall. I started exploring the possibility of creating a site for appliance reviews as well.

mark_l_watson2• 16 years ago

I just tried two test queries on blekko and google. Small sample, but there did seem to be less link-bate results on blekko. The issue is whether their results are close to being as up to date as google's results.

I was interested that blekko seems to have done a lot with a modest amount of funding.

Also, I wonder if they are getting some monetization with the association with Facebook.

meadhikari• 16 years ago

Professor, you could've proved your point by linking to at least one example of how Blekko found a founders work and listed it by date (as the task required), instead you have hashtags on health, finance, etc. The truth is that nobody has arranged that information in the way you want, if it existed at all, that venture database where you found the 500 companies would've been the natural place to look.. CrunchBase maybe?!

Thought Worth mentioning

apollo• 16 years ago

This may be a bit of a tangent, but I want to see the results of the VC system survey.

buro92• 16 years ago

I love Google products, but I can't help but agree. I'm currently trying to find a colour laser printer that has good performance (quality vs speed) with a reasonable running cost over the life of the printer (at least a few years).

All I'm getting is either the manufacturers slant (PR) or spam sites all harvesting the same reviews.

To solve this I now look for vertical based search sites. In this case http://www.printershowcase.com/small-officecolorlaser.aspx is the best I've found... but it's hardly to printers what dpreview is to cameras.

I stick with Google because it largely works well, but when I know what I want to see and that it must exist but cannot find it... then I find myself looking elsewhere all the time. DDG and Blekko I use in these cases, but even they're not solving these kinds of needs.

stephenbez• 16 years ago

I trust Amazon for my product reviews.

buro92• 16 years ago

Beyond books and music... I don't.

Too many products without reviews. No control for the reviews.

Example... the HP CP4025 appears to be good (I have a HP Z800 workstation so thought it was worth checking HP for the hell of it)... yet I haven't found a balanced and comparable review for it. I'd like to see it set against other small office printers and to see the average cost per print.

It's on Amazon twice:

http://www.amazon.co.uk/Color-LaserJet-Enterprise-CP4025n-10...

http://www.amazon.co.uk/Color-LaserJet-Enterprise-CP4025dn-1...

But no reviews.

I'd like Google to take my search terms and help me find stuff... you know, that "sort the world's data" thing, to make sense of my terms and show me the results I'm looking for.

Google isn't working for me. Amazon isn't working for me. It takes hours and hours of searching in circles to research even the most basic purchasing decision.

tedunangst2• 16 years ago

I've found them nearly useless. Way too many people giving one star reviews because their 15" laptop doesn't fit in a 13" bag. If you take the time to read all the reviews, you can weed out the idiots, but sifting out the haters and the astroturfers can take longer than just going to a store and fondling the merchandise.

cletus• 16 years ago

This issue in a roundabout kind of way touches on Facebook.

The issue of social search has a lot of mindshare. Some think it is the future of search. I disagree.

One of the things that made search successful anduseful early on was scale. Instead of having to go to the librar or ask your friends you can effectively canvas the connected world.

I find the notion that friends' recommendations will replace that as nothing short of bizarre. It's like a huge step backwards. The argument is that you can filter out the garbage as your social graph will provide a level of curation.

Let me give you a concrete example. If I wanted t buy a camera I'd stil need t go to dpreview and other sites. It's highly likely that my friends don't really know a lot about this (but some will have an opinion anyway).

This same idea of human curation is behind such sites ad Mahalo and the garbage sites themselves to a degree. Of course at some point computers will be powerful enough to generate this garbage content.

Blekko's idea of slash tags s interesting (to a degree) but if it's successful its easily reproducible. Google is still in the box seat here but of course that's no barrier to a link-baiting TC title.

Personally I'm an optimist. I believe that, much like email spam, the garbage from AC, DM and others I'd a transitional problem (email spam is basically a solved problem now if you use a half-decent email provider). If they succeed we won't be able to find anything. I don't believe that'll happen so these services are therefore doomed.

So betting on Demand Media is (to quote Tyler) like betting on the Mayans (meaning betting they're right about the world ending in 2012: it doesnt really matter if you're right).

So my money is on Google being the better Google.

richcollins2• 16 years ago

How does filtering based on a trust network imply that the network will only include your friends?

stcredzero4• 16 years ago

Indeed. For such filtering to work, a significant portion of your friends should be searching the entire network and not just filtering through friends. But this might just work out to an early-adopter/follower divide. Friend network filtering will work better for the followers. The early-adopters/leaders will continue to use other means of search and will feed the friend-network.

beoba2• 16 years ago

Well, at that point, your 'trust network' may as well include the blog/review site that you would have found through a normal search to begin with.

joe_the_user• 16 years ago

Interesting question.

I'd divide the possible filtering processes into three approaches.

* Throw all your questions to all your friends and all the review sites you "trust". This means any third party site you trust gets a rather excessive ability to spam you - even a site with good user reviews on them are trying push pop-up windows on me. Just because I've gotten good info from X once doesn't mean I want any more from it.

* Throw friend recommendations and trusted sites to a third party "meta-filterer" who organizes things for you. You'd have to really trust that site and essentially there's no reason they'd be better than Google.

* Do the filtering yourself. Most people essentially do that now. I'm working on a project to create tools to automate and improve this this process. Create your own relevance and topic-weighting system that adaptively filters all the other filters. I believe that this kind of approach eventually going to be needed. Not so much because each person can or should do all their topic-relevance-weighting but because this approach would keep the other systems honest.

-----------------

Eventually, people are going to realize that both their social graph and their content-relevancy-weightings/algorithm are far too personal to farm out unquestioningly to a third party. The present social networking system is like AOL-email in 1992 except with the added provision that provide can look-at and alter your emails as part of the service. I'd envision this stabilizing to the present email system where webmail exists but has to work more or less the same of email to your personal computer.

codelust• 16 years ago

The problem with 'social' search is that it reduces the universe of documents to what is known to the members on the platform. There is already data that is not crawled by most of the search engines, which itself is a problem. This only makes it worse.

scorpion032• 16 years ago

> I find the notion that friends' recommendations will replace that as nothing short of bizarre.

Not for generic terms. But for _buying stuff_. If I am buying a camera, I'd be more interested in what my friends own, recommend.

ericsilver• 16 years ago

There are other options besides social or text-based algorithms. My company, Pikimal, is pursuing one - we're pulling together facts on items and are allowing people to weight those facts to dynamically create recommendations. We've added some "user rating" facets but are just barely past our minimal viable product.

I very much agree with you that social is limited. It's a filter which avoids spammers, but it also filters out experts and users. It also solves some of the problems introduced by the generic nature of search algorithms.

I'm betting my time on the idea that the solution which wins will combine an understanding of the product space, the value of new features, a current understanding of price, and which can be customized transparently to the needs of the user.

d4nt• 16 years ago

It's interesting that the way of "gaming" Google appears to be in having thousands of people generating SEO friendly content. I think Google's problem is that it's pushed SEO to the point where the definition of Spam depends either on a subjective view of what kind of site the user is looking for, or it's just mildly worse than something else that's out there (e.g. When I search for something coding related and get one of the stackoverflow scrapers).

Where do we go from here? Well, I don't think the answer is just a radically new way of indexing/ranking websites. That might work in the short term but the spammers will soon catch up. The answer probably lies in a combination of better language interpretation, context sensitivity using browsing history and location, and user profiling based on the social graph and search history. All of which google seems to be working on.

JusticeJones• 16 years ago

Tell me, how exactly is writing a sensationalized article that targets one of the Internet's oldest and largest communities to get fed by CPM advertising any different than what they decry? People have said this time and time again, but they never seem to debut let alone promise any sort of technology to address the issue. They just leave that end of the deal up in the air. As if to say that it's o.k. to spin topics as long as they strike a social nerve, but those who're less graceful at the craft are undeserving of the benefits which they themselves reap.

If the search giants had any balls they'd cut the "Internet Marketing" community off at the knees. Because the money making methods pushed by that community either don't work or are unsustainable, so they're entirely reliant on a steady stream of new recruits. If they want to promote gaming your system don't let them reap any benefits from it.

eitland• 16 years ago

What about a three strike approach like the one that was suggested a few days ago with AdSense?

Domains frequently being excluded by power searchers could be good signal.

(googling to find on that used to pester my search results, kods.net, it seems it has finally got banned. Hooray!)

kokon• 16 years ago

CMIIW, but is that the reason why Google acquired MetaWeb a few months ago? I'm expecting to see some improvement on that front.

tokenadult• 16 years ago

Let me see if I correctly understand the learned professor's article. In his view, the problem is that a user using a free search engine to find information will find a lot of information about people who want to sell products and services, gaining money by exerting their time and effort. What he hopes to obtain for free is email addresses of persons to whom he wants to send his survey, so that he can use their time and effort without compensating them to produce something of value to him. Exactly how is this a problem?

People who actively like to be contacted by random persons surfing the Internet make their contact information readily available (and answer questions sent through those publicly visible contact channels). But to many other persons, not being readily visible on the Internet is a feature rather than a bug. (Disclaimer: my contact information is readily visible on the Internet, so readily visible that it has been used by point-of-view pushers on Wikipedia to give me harassing telephone calls.)

tedunangst2• 16 years ago

I think he was actually trying to piece together the background stories for the people who didn't respond to his emails, not find their emails. I don't think the info he was searching for was particularly private either.

For instance, trying to find out the company a CEO worked at before their current one. The problem is that content copiers will produce so many copies of the PR announcement for their current job, it's impossible to find the announcement for their previous job. I've tried doing this exact search and it's very frustrating.

narrator• 16 years ago

I often find that things that really don't exist wind up with spam results. For instance, if you type in "free ipad" into google you will likely get thousands of search results, all spam because they don't actually exist.

Similarly, contact and personal information for CEOs of major corporations does not exist online either and any search will turn up spam.

One more example, to add to the many. If you get a genuine wrong number call from somebody who made a simple mistake and type their caller id into the internet, you'll just get a bunch of reverse phone lookup spam while if you search for a phone number of a know telemarketer or bill collector, you'll likely get a full dossier on that company.

jrussbowman2• 16 years ago

One of the new things I am working on with unscatter.com is getting quicker access to reviews and blog posts using the blekko api. The next release will be a major change as I've dumped most of the current search providers in favor of blekko and have moved realtime search to it's own page with analysis by providing lists of links in the realtime feed.

Nothing is released yet unfortunately. The site is officially a hobby for me write more but I hope to have the new stuff up in the next week or two. I may just hide the realtime stuff and get the blekko feeds up sooner rather than later.

Now that I am focusing building the site to fit my needs getting up to date info about products and technology, the bulk of my personal searches, is the top priority. Have to admit the blekko api has helped.

In the mean time I would suggest the slash tags /reviews and /blogs with /date on blekko would be very helpful if you are doing product searches. With unscatter I am really only providing shortcuts for the with additional ui tweaks.

Disclaimer: I am in no way associated with blekko other than having been given permission to use their api for a personal project.

jrussbowman2• 16 years ago

Sorry meant to say the site is a hobby for me right more. Wrote these comment with the swype keyboard on my phone so I apologize for any problems reading them

EGreg5• 16 years ago

Hey, so what you are basically saying is, "the best computer algorithms in the world" (you know, Google has like > 578690 Ph. D's) are not good enough to have effective search, so we should introduce the human element.

Fair enough. There is the Open Directory Project (which is pretty old) and of course there is Facebook, Twitter, and other, human-curated services. Starting a whole new company to do search and compete with Google (and Bing)? Seems like a waste of time as Google can just copy what you are doing and incorporate it into its already massive site (complete with traffic, audience, and lots of other goodies). Instead, why not get Google to add more social recommendation and feedback features?

ams61103• 16 years ago

Why would it be so difficult for Google to filter out spam sites? E.g. DuckDuckGo filters out eHow.com results, because they are low quality and tend to be spammy.

Oh of course, it's not in Google's interest to do this, because they make money from the spam sites. So I don't expect Google to really "solve" this problem.... their trick is to stay useful enough that users don't abandon them, but allow enough spam into the search results to provide revenue. A tricky balance...

gsmaverick• 16 years ago

If Google were to attempt to filter out sites, they would likely be met with lawsuits and anti-trust actions.

ams61103• 16 years ago

Well they already rank results, so filtering is not a big extra step. If you like, just push spammy pages down rather than eliminate them entirely. Then it's just enhanced ranking, not filtering.

moultano2• 16 years ago

None of Google's ranking decisions are concerned with revenue from AdSense. Period.

CamperBob• 16 years ago

And I should believe this because....?

SoftwareMaven2• 16 years ago

But that is different than saying the prioritization of this as a "problem" is or is not affected by AdSense revenue. What we, as a bunch of highly technical people, see as a problem, Google's ranking decisions sees as, at best neutral and at worst, a positive.

It may not affect ranking (and I do believe that), but I would certainly be willing to bet it affects them (and similar sites than aren't AdSense-based) not being de-ranked.

danilocampos4• 16 years ago

An obvious improvement to Google whose absence shocks the hell out of me would be this:

Personal domain blacklist.

There's a lot of spammy bullshit on the web and Google seems to have given up on keeping this away from me. Fine. But for my specific searches, there's usually a handful of offenders who, if I never, ever saw them again, it would improve my search experience by an order of magnitude.

So let me personalize search by blacklisting these clowns. Why can't I filter my search results so that when I search for a programming issue, I never see these assholes from "Efreedom" who scrape and republish Stack Overflow?

I don't, personally, need an algorithmic solution to spam. Just let me define spam for my personal searches and, for me, the problem is mostly solved.

(Also blacklisted: Yahoo Answers, Experts Exchange.)

steveklabnik12• 16 years ago

Just a small note, SO puts all of its content under a Creative Commons license[1][2], which they are (in my non-lawyerly opinion) following.

Now, this doesn't mean that filtering them wouldn't be useful to you, since at first glance it appears they're solely a duplicate. Just pointing out that they're not actually doing anything wrong, and they're (probably) not scraping.[3]

SO has specifically said that this is okay.[4]

1: http://wiki.creativecommons.org/Case_Studies/StackOverflow.c...

2: http://creativecommons.org/licenses/by-sa/2.5/

3: http://blog.stackoverflow.com/2009/06/stack-overflow-creativ...

4: http://blog.stackoverflow.com/2009/06/attribution-required/

danilocampos4• 16 years ago

> SO has specifically said that this is okay.

It doesn't look like Jeff is that okay with it, especially when it comes at the cost of Stack Overflow's own ranking:

Sorry, this is absolutely necessary, otherwise we get demolished by scrapers using our own content in Google ranking. – This is from a question about Stack Overflow's SEO strategy:

http://meta.stackoverflow.com/questions/71906/first-tag-in-t...

EGreg5• 16 years ago

You know, I am starting to think more and more that copyright should ALLOW peer-to-peer sharing but not republishing. That's where the line should be drawn. There should be a clear definition of publishing, e.g. serving content upon request to anyone immediately on demand. That way, if the authorities can download something copyrighted from a public source which is not the original author, they can go prosecute that source. Just a thought. Ideas?

kmfrk7• 16 years ago

Relieving your users' copyright of their own published content hosted on your own platform is very dubious territory legally.

It's fine that they have an expressed policy that says it's okay, but I'd keep it at that and not refer to terminology like CC licenses.

mcantor2• 16 years ago

Legal but unethical.

forkandwait• 16 years ago

If there was a blacklist option, then one could subscribe to a blacklist maintainer like one does for Firefox Adblock -- that would be really, really nice.

I am actually suprised there is no Labs application for this, unless there is a business case against it.

vyrotek• 16 years ago

Definitely an interesting idea.

The first thought that came to mind was what happens when I disagree with a couple of items on one of these 3rd party blacklists?

Then I thought, FORK IT and make the changes you want. You could even merge in lists from other people. Github for blacklists?

jhchen• 16 years ago

If this gets upvoted enough I will build a browser plugin that does this.

telemachos• 16 years ago

Slightly off-topic, but if you use Chrome, there is the 'stackoverflowerizer' extension[1]:

> Always redirect to stackoverflow from pages that just copy content, like efreedom, questionhub, answerspice.

[1] https://chrome.google.com/extensions/detail/gledhololmniapej...

Groxx2• 16 years ago

You can kind-of do this manually:

  -site:answers.yahoo.com

Someone could probably get a script going to add that to any queries you have. Unfortunately (for me), it breaks Instant.

shadowpwner• 16 years ago

32 words is the limit to a Google search, so it'd be difficult to block all sites you wanted to.

minalecs• 16 years ago

At one time google did provide this, if you were logged in , there was an [X] option to remove that result from your searches, as well as a voting up and down mechanism. I think it was just an experimental feature, but I wish they would of kept it, and expanded on it. If google is reading.. please bring it back

danilocampos4• 16 years ago

I remember that. The trouble was that it was result-specific, not site-specific. There wasn't a way to kill an entire site, at least that I was aware of.

gojomo2• 16 years ago

Similarly, I'd like a blacklist or advanced search operator that, for certain queries, would allow me to exclude all sites with AdSense on them. Might have to wait for Blekko or Bing to provide that, though.

dtby2• 16 years ago

Personal domain blacklist.

For whatever it's worth, blekko has this. It's one of the main reasons I switched to blekko over Duck Duck Go for the majority of my searching.

CWuestefeld2• 16 years ago

I find that DDG does a pretty good job of filtering spam sites on its own. That's the biggest reason that I use it.

epi0Bauqu2• 16 years ago

The only reason we don't have it is we don't have accounts. I'd be happy to add it as a cookie setting for now if people would use it.

Also, I'm also happy to take requests to ban these stupid sites for everyone.

evgeny0• 16 years ago

This is something a browser extension could do (though not as well as the engine itself, of course). There is Chrome extension called "Google Blacklist"[1] but it didn't seem to have any effect when I tried it out. Perhaps others will have better luck.

[1]https://chrome.google.com/extensions/detail/hbodbmhopadphblo...

jsrfded• 16 years ago

blekko has that feature. every result has a "spam" button underneath it. Click the link, and the host will be added to your personal /spam slashtag. Everything on your /spam list gets negated from all of your results by default.

Very handy. I put ehow.com on mine and never see results from them.

kmfrk7• 16 years ago

There must be someone at HN somewhere in the Google pyramid who can tell someone with decent clout about this proposal.

There must be some way Google's search engine could learn by looking at the blacklists people uses.

nostrademons• 16 years ago

Problem is that this was tried before, with SearchWiki. When it launched, it was widely derided as being useless and distracting. Its usage numbers didn't show widespread adoption. And then when it was removed, there was much rejoicing.

Now, it's possible that SearchWiki just needed a few more iterations, and with a few details changed, could be a big success. There have been a few other recent launches that were tried years ago, didn't work then, but had a few more iterations and now are big successes. I could at least raise the issue. But unless I can tell a convincing story about why people would use this when they didn't use SearchWiki, it may be an uphill battle to get resources devoted to this.

dejb• 16 years ago

I believe you could achieve this by creating a Google Custom Search Engine (CSE).

> I don't, personally, need an algorithmic solution to spam.

Not now you don't. But if everyone started using your approach then the spammers would adjust their behaviour and use many domains instead of one.

coderdude3• 16 years ago

Which would kill their chances at gaining significant ranking for each of their spam domains.

mmb• 16 years ago

I use the OptimizeGoogle (and previously CustomizeGoogle) Firefox plugin for this:

http://www.optimizegoogle.com/

atomical2• 16 years ago

I would blacklist ezinearticles. It's an SEO farm with no other purpose than to target keywords and redirect traffic somewhere else. The value you get is minimal per click compared with Wikipedia.

mattmcknight2• 16 years ago

One idea that comes to mind to deal with the wikipedia / stackoverflow problem is result clustering. With Google News, they have done a pretty good job of clustering articles on a single story. They are getting better at deriving the original source in many cases. The simple act of duplicate detection should enable them to identify sites that scrape content and show them as duplicate results.

In the interests of results diversity, you don't want the same content repeated ten times on the first page, although this has the side effect of pushing the original source onto the second page if you guess wrong.

taiyab• 16 years ago

Very good point. Scrapers really need to be dealt with and you should be able to define what you class as spam on a personal level in Google. It's astounding they haven't done it yet.

stcredzero4• 16 years ago

He couldn’t make head or tail of the results. Paul concluded that the “the entire web is spam when it comes to major appliance reviews”.

A simple solution to this: Consumer Reports. A subscription is well worth it! The likelihood that it will pay for itself in the next year is very high.

ams61103• 16 years ago

Consumer reports are probably about as objective a source as you can find, but I don't believe they are without their biases. They tend to give a lot of weight to things like value and reliability, and less on aesthetics, though the latter may be an important factor for some consumers. For tech products, the review them from the standpoint of an "average consumer" and probably won't evaluate factors that matter a lot to many readers here.

They are also politically left-leaning, if that matters to you.

stcredzero4• 16 years ago

For tech products, the review them from the standpoint of an "average consumer"

Yeah, don't use them to rate tech products. But as far as coventional appliances go, they are a very good resource.

didip2• 16 years ago

I just created a blekko account after reading this article (good job TC! It works this time.)

They seriously need to hire a capable UX person. The logged-in interface is full of problems:

* Twitter-like status update. I believe this has nothing to do with search.

* Form with 10+ fields on creating a slashtag. You cannot possibly expect me to enter all domain names I could think of into that tiny <textarea>?

* I finally created /python but I have no idea how to improve or update the slashtag. I cannot update that slashtag from search results page.

Overall, very frustrating experience.

Dramatize• 16 years ago

I'd like to have the option (like facebook has when you mouse over a post in your feed) to hide all results from X website.

If you tied that with the ability to follow other people and their search edits, the number of spammy results could be reduced.

kmfrk7• 16 years ago

How does yegg deal with this on DuckDuckGo? A lot of us use his search engine, and it's a great one at that, which is not worth forgetting.

epi0Bauqu2• 16 years ago

By way more aggressively removing useless sites.

DTrejo• 16 years ago

http://duckduckgo.com/ works very well for me.

  - less spam
  - programmer oriented results, when relevant
  - more legible search results

oliverdamian• 16 years ago

How about a P2P search/bookmarking platform where peers could publish search/bookmarking histories ranked by like/dislike/spam votes which other peers can subscribe to. Publishing peers can also be ranked according to number of subscribers. Actually P2P curation could be the next level up from raw centralised search. Is there anything like this out there already?

Why We Desperately Need a New (and Better) Google

Related Stories