This is probably why everyone still relies on Google

http://robotmay.com/post/23096164275/these-are-the-crawl-stats-for-the-last-month-for

by _ompc • 14 years ago

105 38 14 years ago

bmj1• 14 years ago

Reasons people still uses Google:

(Perception of) better results: until proven otherwise - why would you use an alternative/inferior SE - unless you value other things like privacy more than the content of the results - this doesn't apply to most people

Fast/it just works: give me a compelling reason to switch. "Privacy" really isn't enough for most people.

It's default: the default FF SE? The default homepage for a bunch of people? Default search in Chrome? Check. Check. Check

Network effect: Remember the expression "Nobody ever got fired for choosing IBM" (that might be too old for a lot of the readers here) - but the same thing applies - "nobody ever got in trouble for choosing Google"

robotmay8• 14 years ago

All equally valid points. For me I find that the quality of results is what still draws me back. I use DDG for most things these days (as it's especially good for quickly finding repos on GitHub etc), but if I have anything less certain (like a caching question) then I often find myself using Google.

sek• 14 years ago

I never had a problem with Google results. The only problem i always have is when there is not enough content about a topic. These are the times when i try another search engine, but the results are often worse then. I make around 20k searches a year with Google and i know how to find stuff with it, if there is something to find.

Google is free and Bing/DDG/Yandex make nothing better that would make me switch. There is not really a problem to solve when it comes to find available stuff. Quora and Stack Overflow try to solve a problem Google isn't good at. Getting answers to stuff when there is nothing to find out there.

robotmay8• 14 years ago

http://millionshort.com popped up a while back and I've actually found it incredibly useful, especially for any topics more likely to be found on little personal sites.

vibrunazo• 14 years ago

> Default search in Chrome?

It's not default in Chrome, it asks when you first use it what you wanna use as the default search. You have to opt in to google and bing is just one click away. Most still choose google anyway.

In contrast, to change bing from being default search, on the default browser, on the most used OS in the planet. You still have to go inside preferences and opt out, which takes several clicks. And then, Microsoft still tries to sneak bing to you if you install MSN (windows live or w/e it's called now), which will change your default search engine on FF and Chrome to Bing. Unless you opt out during instalation.

And still people choose google.

balac• 14 years ago

In Europe IE will also asks you to choose a default search engine on first start.

stefanve• 14 years ago

Yahoo is using Bing so it is not strange that is has zero indexed pages. Also Duckduck go is using Bing and some others like wolfram alpha. The only reason I still use Google it that they are still miles ahead of the competition. I have switched for a while to Bing, DuckDuckGo and others but always wend back to Google.

I do value my privacy, for instance, I switched from gmail to fastmail even if fastmail has less functionality. But for search there is not a real alternative for me.

robotmay8• 14 years ago

Aye, I think Cloudflare had Yahoo! in there as a bit of a legacy thing. I find myself appending !g to a lot of my queries on DDG; but I'm fine with that, as it offers me some interesting features that I don't get on Google.

yaix2• 14 years ago

Yep, I tried to use DDG for some months, but often find me repeating the same search in Goog. However, usually I open a incognito browser window in Chromiums when I search in Goog, so it doesn't get my normal user tracking cookie.

bgilroy26• 14 years ago

You might want to look into your specific useage situation, but generally incognito shares the usual information with the server, it only switches off logging to the client.

For instance,in most cases, your wife will not know what you've searched for, but Google will.

Tor is an example of a technology that does what it seems like you're looking for.

There also may be options in your Google settings to opt out of some of their info collection.

WildUtah• 14 years ago

Wait, DDG is using Bing? I thought it was an independent and written from scratch enterprise. Does it depend on Bing for everything?

crazedpsyc• 14 years ago

No, there are over 50 sources: http://help.duckduckgo.com/customer/portal/articles/216399-s... And they aren't just copied verbatim (with the same ranking, etc), but are re-ranked and mixed.

polyfractal• 14 years ago

Yeah, DDG uses Bing as one of it's sources. It's similar to the "meta search engines" of ye olden days.

DDG's big schtick (other than privacy) is to use lots of sources and then heavily filter the results/supplement them with additional sources (wiki, stack overflow, etc).

The founder of DDG has said he specifically wants to avoid writing a general-purpose spidering engine because you can sink millions of dollars into it and still be miles behind the competition.

koeselitz• 14 years ago

"DuckDuckGo gets its results from over 50 sources, including DuckDuckBot (our own crawler), crowd-sourced sites (in our own index), Yahoo! BOSS, embed.ly, WolframAlpha, EntireWeb, Bing & Blekko. For any given search, there is usually a vertical search engine out there that does a better job at answering it than a general search engine. Our long-term goal is to get you information from that best source, ideally in instant answer form."

source: http://help.duckduckgo.com/customer/portal/articles/216399-s...

nextparadigms• 14 years ago

Have you tried Blekko?

yaix2• 14 years ago

I tried DDG, but Goog's results are just better. And when I do tail -f log/access_log on my box, GBot is always there, hitting my sites every few seconds 24/7. Sometimes Bing is around, but less frequent and much slower.

robotmay8• 14 years ago

I always knew Google hit more (especially if you're watching the access logs) but I didn't realise quite how much more until I saw the graphs on Cloudflare.

jgrahamc• 14 years ago

Those stats look a little off, and I wonder if it's because the site in question is pretty new. For comparison here's a screenshot of the crawl information from my CloudFlare analytics panel (the same information as gathered by the OP): http://i.imgur.com/vrKlz.png

jgc.org has been around since 1997, daysoutnearme.com since April, 2012. I'm guessing that the other engines don't yet know about the site.

robotmay8• 14 years ago

Ah ha, that's really interesting. My site hasn't been around very long, but it has been submitted to both Google & Bing for the same amount of time. Nice to know that it balances out over time, but it looks like Google pays more attention to new sites than Bing does. Or Google has more resources with which to crawl sites, of which I have no doubt.

dwc• 14 years ago

My impression, based on eyeballing logs for my own low traffic, unimportant sites: Google collects and uses stats on how often pages change, how often they come up in searches, etc., and for static and/or low importance pages they scale crawling back. This frees up crawling resources to spend elsewhere, like on important, frequently updated pages, new sites, et al. I haven't looked closely in a couple of years, but the pattern used to be that new content would be crawled fairly agressively and after a while it would back off to an "appropriate" level.

Back then, other search engines took a good while to begin crawling my new stuff in earnest, and once they did their crawling pattern did not seem to change much over time between a (relatively) popular changing page vs. a lonely static page.

Domenic_S• 14 years ago

> I'm guessing that the other engines don't yet know about the site.

Didn't he say he submitted sitemaps?

binarymax2• 14 years ago

OT, does anybody actually know how to get Bing to index your site? I looked for maybe an hour with no luck.

robotmay8• 14 years ago

http://bing.com/webmaster. Good luck, it took about 2 weeks before it even hit it once :\

dangrossman• 14 years ago

> I looked for maybe an hour with no luck.

1. On the Bing homepage, click "Help"

2. Click "Find Help for site owners"

Arrive at the answer to your question.

binarymax2• 14 years ago

That's one of the paths I went down before, got to a place where I had to sign into windows live, signed in, and arrived at a dead end. Robotmay's link above looks good though!

PaulHoule• 14 years ago

I dunno... I have a site that would get creamed by Baidu's web crawler if I didn't block it, but never got noticeable traffic from it.

loverobots• 14 years ago

From what I can tell, Bing is very picky in indexing pages, whereas Google will index and ignore them if not up to their standards.

I use Bing and DDG and maybe once or twice a week I need to use Google to double check. Bing is more than fine for "most users," but Google has a powerful brand, they have a lot of goodwill and they open their checkbook to buy traffic (Mozilla, Opera, iOS, portals etc.)

Edit: All the above (even extra Googlebot servers to index new sites in seconds) are because Google can "monetize better," or has a lot of traffic which in turn attracts advertisers that bid up prices. Google is also shameless in adding dozens of ads in all places so they have lots of money to maintain their status quo. Put Bing or DDG results on Google.com, under the Google logo and very few "regular" people will complain.

asadotzler• 14 years ago

Google search is overwhelmed by new "hot" topics rather than solid valuable enduring sites. It's almost as if their news crawl has invaded their web crawl. I think the rush to have "fresh" content in the first 10 links is actually making Google significantly less useful.

gcb• 14 years ago

Ever tried searching a business phone number in Google?

No matter if said business has a page for 10yrs. The first 2 pages well be phone directory seo spam sites that cropped up last couple years.

imperialWicket• 14 years ago

Mildly off-topic, but even if I preferred google's results to duckduckgo's, I would still use duckduckgo's "g! my search phrase" over google - just for the better interface/styling that ddg offers.

robotmay8• 14 years ago

I must admit that a significant portion of my love for DDG is because of the duck. It just looks so enthusiastic.

alan_cx• 14 years ago

A random guess:

I reckon that people still use google because they have learned to "trust" google's results and they are afraid that if they use another search engine they will miss out on a useful result.

So, I wonder if new sites should show results comparisons to give new users confidence.

jast• 14 years ago

It's funny... just runned the query "days out near me" on google and bing. Bing shows the site at number one while google only shows it at number 3, 4 and 5 and never the top domain.

robotmay8• 14 years ago

Hah, that's quite entertaining. Doing a non-bubbled search on Google has it at 2/3/4/5 for me. The site at #1 is, I'm assuming, much older.

This is probably why everyone still relies on Google

Related Stories