Our new search index: Caffeine

http://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html

by Anon84 • 16 years ago

181 72 16 years ago

googleblog.blogspot.com

tptacek2• 16 years ago

Wow is that ever a terrible infographic.

rbranson• 16 years ago

One of the worst I've ever seen. It wasn't even worth it if it took 3 minutes to draw it up.

defen• 16 years ago

I couldn't tell if it was a joke... "magic internet stuff here"

tptacek2• 16 years ago

You think for a moment that at least maybe the different content types are allocated different colors, but, no, they're all dark blue. One of two dark blue orbits in Caffeine.

MikeCapone• 16 years ago

Considering they had that page with pigeons to explain how pagerank works, I would assume it's not supposed to be very serious.

jedsmith2• 16 years ago

I'd have to disagree. It shows us that Google has adopted the experimental "Bohr" model of database indexing.

resdirector5• 16 years ago

I'm not sure if I get this...is this just a pithy one-liner, or am I missing something deeper here?

bshep• 16 years ago

I agree. The depiction of old index looks better, that to me means failure to communicate on their part.

imagii• 16 years ago

Without reading the article (and ignoring the captions), you'd think the left side was better.

Ordered and color-coded, as opposed to flying around.

tjpick• 16 years ago

agreed, I immediately thought the old index is structured and layered, the new index is a big ball of mud...

orionlogic• 16 years ago

very much like itunes Genius bar logo with electrons changed with google sidebar icons.

Caffeine is not new and has been talked in SEO community from a quite sometime. I wonder why they make it public now.

To silence iPhone 4 news? No no,just coincidence...

dandelany• 16 years ago

Well, search is their primary service; their web indexing algorithms are sort of their Mother of All Secret Sauces. You wouldn't expect Coke to publish the exact recipe.

ErrantX• 16 years ago

Perhaps you have to "build search engines for a living like [them]" to understand it.

dskhatri• 16 years ago

Nice to see they reference an iPod instead of a Nexus One. Somehow I feel that if it was a Microsoft announcement, the company would reference Zunes.

batiudrami• 16 years ago

Nexus One total internal storage: 512MB 'the largest iPod': 160GB

The Nexus One is a phone, not a device with a large storage capacity. If Google made one of those, I'm sure it would be referenced.

edit: the Nexus One box also includes a 4GB micro SD card, but that'd be comparing flash memory with harddrives, and is hardly an intuitive explanation.

jodrellblank3• 16 years ago

That is an enormous amount of data.

I wonder how much is junk?

mkramlich• 16 years ago

You're new to the Internetz, right? :)

jodrellblank3• 16 years ago

I meant junk like search engine cheating forum and usenet posts and sites registered to repeat stock phrases and content of spammy aggregator sites, rather than making judgements against lolcats and breakfast time tweets.

gojomo2• 16 years ago

100 petabytes of information (100 million GiB) feeding their index is more than I would have expected.

moe2• 16 years ago

On the other hand it doesn't seem that much when you consider todays storage density.

You can fit around 0.5 PB into one rack nowadays. 200 racks then sounds a bit less impressive than 100 Petabytes.

However, that ofcourse doesn't account for redundancy, nor for doing anything useful with such a pile of data. Both of which impose some interesting challenges at that scale.

blasdel2• 16 years ago

You think Google's mainline storage of their index is on disk?

It is to laugh. I don't think that's been true since around the time they stopped building server shelves out of lego or cardboard.

They've been keeping the full text of the web in RAM (for the snippets), with indexes, several times over. With independent live siblings in multiple datacenters.

jgavris• 16 years ago

I too am skeptical of the hundreds of millions of gigabytes per day.

jodrellblank3• 16 years ago

Hundreds of thousands of gigabytes per day.

swah2• 16 years ago

Java or C++

coderdude• 16 years ago

If there could be an answer to what you said, it would probably be "both, and Python."

shadowsun7• 16 years ago

Google uses only four languages in their codebase: C++, Java, Python and Javascript. Yegge has covered this pretty extensively in the past. Go Google him.

jlouis• 16 years ago

And Go, presumably.

chintan• 16 years ago

huh? 40 miles of iPods.

+5 for an intuitive analogy. Priceless!

mkomo• 16 years ago

hopefully iPod-miles (iPm?) will become a new ISO standard for measuring storage capacity.

rimantas• 16 years ago

Would nicely complement a station wagon full of tapes for bandwidth.

Keyframe• 16 years ago

I wonder how much libraries of congress is that?

iworkforthem• 16 years ago

old index: problem: parts of the index were refreshed at a faster rate than others; the main layer would update every couple of weeks and to refresh a layer of the old index, Google would analyze the entire web first. impact: there was a significant delay between when Google found a page and made it available to its users.

new index: solution: caffeine, Google's new index will analyze the web in small portions and update its search index on a continuous basis, globally. As Google find new pages, or new information on existing pages, Google can add these straight to the index. improvement: user can find fresher information than ever before—no matter when or where it was published.

recent changes in Google search result page suggested that the searched results appears to be categorized further to the different structured data. This could be inline with the rolling out Caffeine.

Corrado• 16 years ago

Hmmm... come to think about it I have been getting quite a lot of results from CodeWeblog.com in my search results. Unfortunately, that site seems to be a link farm and its so useless that I reported it to the Google spam page. Maybe its appearance in the top sites was a result of the Caffeine changes.

biafra• 16 years ago

Ah! Its not only me who is annoyed by these codeweblog.com results.

Is there a way to remove domains form google results as a preference? I know I can always add "-site:codeweblog.com" But can I save this preference for future search queries?

joewein• 16 years ago

I reported them as well, but like your report, it doesn't seem to have been acted on.

See my blog post at: http://www.joewein.net/blog/2010/07/01/codeweblog-com-a-pile...

samaparicio• 16 years ago

Why is every measure of storage always "demystified" with a paper stack analogy? I find it funny that when the paper stack would not provide the right scale, the used a stack of storage devices...

jsiarto• 16 years ago

I'm waiting for Google to open up an API or dashboard on top of Caffeine to compete with some of the monitoring tools like Radian6 and Jive. They could own that space and it's growing like crazy right now. Plus, we don't need any more companies coming out with "social web" search engines.

spuz2• 16 years ago

What I dislike about Google's current algorithm is that when I search for a common term such as "javadoc simpledateformat", the API documentation for Java 1.4.2 (as opposed to 1.6) comes up as apparently, that is what most documents on the web link to. I hope this new index will allow more recent and more up-to-date documents to get to the top of the results list.

noss• 16 years ago

I tend to use "java6 simpledateformat" as my query for that.

Jermey128• 16 years ago

In Firefox I have the following quick-search to fix that (and make searching the API a little faster):

http://www.google.com/search?q=java+6+%s&ie=utf-8&oe...

This should also work in Chrome.

spuz2• 16 years ago

Yep, I've done exactly that for the Java 6 docs in Chrome and FF. The general point of how do you keep the latest version of a set of documents the most relevant still applies however. I hope Google can fix this :)

suraj• 16 years ago

Similarly searching for Apache directives (e.g. http://www.google.com/search?q=rewriterule ) returns result for apache 1.3 docs. You have to add 2.2 while searching for apache info.

cookiecaper• 16 years ago

I had the same annoyance searching for PostgreSQL stuff -- the latest docs are almost never accessible from the Google page, just docs 2 years old or more.

leej• 16 years ago

they could be using in-memory compression. they could be storing hot parts of index in memory. they could be using both.

dalore• 16 years ago

Aren't we all using Duck Duck Go anyway?

c00p3r• 16 years ago

Seems like the target group of this blog post are teenagers, not even students. Some street slang would look well near the size measurement in ipods. =)

keegangrayson• 16 years ago

code or it didn't happen...

Our new search index: Caffeine

Related Stories