Cuil Crawl Data: 310 terabytes of compressed data, snapshot from 2007-8

https://archive.org/details/cuilcrawl

by aw3c2 • 13 years ago

83 31 13 years ago

archive.org

jbellis• 13 years ago

Will this be integrated with the Wayback Machine?

vijayr• 13 years ago

how effective will it be? It is 5 years old

fhars2• 13 years ago

That is kind of the point of an archive, no? Keeping old stuff for further reference?

mjn• 13 years ago

The Wayback Machine archives old snapshots of pages as they appeared at various times. If the Cuil crawler has snapshots from 2007-08 of pages that aren't already in the Wayback Machine's own snapshots from that time (the Internet Archive crawler doesn't manage to get to everything), they could be used to supplement the archive.

sp3322• 13 years ago

According to a comment I saw on IRC, it already is (as of today).

gnu8• 13 years ago

I can't wait to check this out in my Flock web browser!

fhars2• 13 years ago

He gives you a hamburger, but it turns out you don't actually exist. He gives you terabytes of data. Something good comes out of it in the end.

alex_c• 13 years ago

For those wondering:

http://www.reddit.com/r/worldnews/comments/7da5i/police_raid...

daurnimator• 13 years ago

care to explain/give further context? Even looking at that whole reddit thread I don't understand.

jameskegel• 13 years ago

I can barely remember Cuil, but what I do remember is it being touted as a "Google Killer". Funny how that worked out; I wonder where they went wrong. Oh that's right, I searched for Fedora Linux and got pointed to a menswear retailer in the south of France.

treerex• 13 years ago

They touted themselves as a "Google Killer". They basically came out of the gate claiming to be the best search engine in the world, and then they went down in spectacular flames. Whoever developed their launch PR should have been put up against a wall and shot.

If they had rolled their technology with less fanfare they may have made a minor dent in the market, but even then its unlikely. But at least then they could have taken their IP and gone into the Enterprise Search space and perhaps have gotten bought out by someone evil.

sp3322• 13 years ago

The press picked them up before they were actually ready to launch. Their search model was to segregate results into different categories and show all the categories at once, in sections of the page. When the traffic spike, the most popular categories went offline, leaving only the long-tail sections active. Most people never saw how the search would have looked when the software was finished and the servers beefed up.

bane• 13 years ago

Even worse, they seemed to exist in a state of denial about the quality of the search, then proceeded to do nothing about it as they burned through investment dollars for a few years. Then put out an even worse automatic encyclopedia.

taylorbuley• 13 years ago

Is this a product of Cuil's infamous Twiceler bot?

bluedino• 13 years ago

I'd love to hear the story behind getting the data to the Internet Archive, from Cuil. 310TB is a massive amount of data, and was even more massive in 2007. The biggest hard drive you could buy were only 500-750GB. If they were SAS drives they were probably only 300GB at the biggest. Did they just throw 6-8 racks worth of a SAN onto a truck and ship it over? What did they (Archive.org) do with it once they got it?

adventured2• 13 years ago

Here are some photos of their hardware (business insider claimed their local setup held 7 petabytes in the article circa Nov 2009).

http://static.businessinsider.com/image/4af8564f0000000000e6...

http://static.businessinsider.com/image/4af84e320000000000b7...

http://static4.businessinsider.com/image/4af880fe00000000009...

http://www.businessinsider.com/cuil-office-tour-2009-11#cuil...

xhrpost• 13 years ago

Is the actual data available for download yet? I'm just seeing hundreds of meta-data XML files.

Mithrandir• 13 years ago

I don't think so. If you go to one of the items (e.g. https://ia601408.us.archive.org/18/items/cuil-domainshard-co...) and try downloading one of the .arc.gz files, you'll get a message saying "The item is not available due to issues with the item's content." I've seen this happen before with other crawls, like for Youtube (https://ia600301.us.archive.org/26/items/IA-YOUTUBE-000-2007...)

It could be a copyright issue or to keep bandwidth low, but I don't know for sure.

hosay123• 13 years ago

So for about $12,000 in bandwidth fees, $17,000 in 4TB SATA drives, and say another $15,000 on 10 2U hosts, you could have a slightly stinky copy of the Internet that would fit in a half-height rack.

That's freaking awesome.

deletes• 13 years ago

Even more interesting is, will in the future that space that represents the entire internet ( the rack ) grow in size or will disk storage improve faster than the growth of internet when finally you will be able to fit internet on a small hand held disk.

adventured2• 13 years ago

Media alone will vastly outrun micro storage capabilities over any relevant length of time we can project (5, 10, 20 years). It might be an interesting thought experiment regarding text (and probably only a fraction of a fraction of that due to social media).

YouTube alone will guarantee you could never store the Internet on a small hand held disk. They're adding 72+ hours of video to the service every minute. And ten years from now, it'll all be HD+ content, and they'll probably be adding 500 hours per minute or something similarly crazy.

I think Wikipedia is around 42gb right now, uncompressed (just the content pages). I don't think that includes the images. So right now we're just to the point where you can store a text Wikipedia on your smart phone with a $30 or $40 sd card. I'd guess in eight to ten years we'll have 500gb to 1tb smart phone equivalents depending on how that all evolves. You might be able to store a dozen plus dual layer full blu ray discs on your smart phone in a decade.

In other words, it's easy to say that we're not going to get anywhere near storing even a sliver of the Web / Internet on a small device in our lifetimes.

6d0debc071• 13 years ago

I believe humanity's data was increasing by around 23% a year in 2007 when we were estimated to have around .3zb. Assuming that trend's holding steady with respect to the internet I'd expect us to massively outrun increases in storage.

bediger4000• 13 years ago

Has the NSA downloaded it yet? That would be a good start to the Utah Data Center's contents.

wmf• 13 years ago

The way they're spliced in means they don't have to crawl anything. When you surf the Web you're crawling for them.

NelsonMinar• 13 years ago

Anna Patterson, one of the Cuil founders, previously worked at archive.org and built an early search engine over their archive (named Recall, IIRC). According to her LinkedIn profile she currently works at Google.

ashrust2• 13 years ago

It wasn't just Anna, there were many former archive staffers at cuil. I suspect that's how the data ended up being transferred.

Cuil Crawl Data: 310 terabytes of compressed data, snapshot from 2007-8

Related Stories