How we got 100,000 visitors without noticing

http://blog.historio.us/how-we-got-100000-visitors-without-noticing-4

by StavrosK • 16 years ago

80 51 16 years ago

blog.historio.us

jacquesm6• 16 years ago

I predict a lot of sites featuring the words 'free sms' will be hammered out within hours of this articles posting.

StavrosK18• 16 years ago

We didn't know one of our users publicised it, though, and we didn't know Google would even see it...

chopsueyar2• 16 years ago

in portugese?

chrismiller2• 16 years ago

This might be slightly off topic but if you don't mind could you share the specs of your Varnish server? Everything I have read about Varnish says that it will run extremely well on pretty much any decent hardware. But I am still unsure how much power I should be providing it.

jacquesm6• 16 years ago

How many hits per day are you expecting?

We serve up about 2000 requests per instance per second, 8 instances on an 8 core machine with 32 G of RAM.

chrismiller2• 16 years ago

Thanks for the numbers they are very helpful.

Probably <500 cache hits per second (a large portion of the content isn't cacheable).

So I should be able to serve that from a fairly low spec machine.

pvg• 16 years ago

Why run separate instances on the same box?

chopsueyar2• 16 years ago

This for ww?

StavrosK18• 16 years ago

As jaquesm said, with your expected visits, you will probably see no load at all, so don't worry about it. You definitely don't need to give it more power, unless you're running it on a cellphone...

RBr• 16 years ago

The short form if don't want to read the entire post is summarized with this snip:

"By some algorithmic oddity, the historious cache actually ranked higher than the site (which turned out to be the most popular free sms service in Brazil), so anyone who searched for "free sms" in Brazil ended up on the historious page!"

In short, Google ranked an archive page (historious) higher then the content. Google corrected this problem, presumably automatically, 3 days after it happened.

templaedhel• 16 years ago

This makes me want to go and try out varnish. I have heard great things, but never gotten around to setting it up.

csomar• 16 years ago

nice discovery. I read somewhere in Blackhat forums about Google Search Engines traffic exploits. I didn't understand what they are talking about until I read your post today.

emehrkay• 16 years ago

I just hit the "historify!" bookmarklet on the varnish page. :)

StavrosK18• 16 years ago

Your actions have placed all of us in great danger.

mmaunder• 16 years ago

Eek, not to rain the parade, but this is not good historious team. Google's duplicate content filter is generally accurate but in this case it penalized the site that originated the content and ranked yours higher because it assumed you were the owner of the content i.e. the page about sending free SMS's.

Google will quickly correct this error and you'll find the traffic on that page drops to nothing. You may also find other duplicate pages on your site penalized in the same way. You may also find your site penalized by Google for essentially screen scraping sites and copying their content verbatim and in it's entirety.

I'd recommend you set up a robots.txt to block Google from indexing identical pages on your site. Copying content verbatim and republishing in it's entirety is not a good SEO strategy unless you have thousands of throw-away domains and wear a black hat.

Sorry about the negative message, I'm sure this was very exciting for you guys but I don't think it's sustainable.

Perhaps an alternative is to have your users highlight the paragraph or snippet on each page they found interesting and archive that instead. Then publish those snippets on your site. Generally I've found republishing paragraph size chunks of text is OK with Google and will net you decent SEO traffic. It worked for my job search engine when we republished job descriptions and limited them to the first 400 chars. You can also mash the paragraphs up into pages with multiple chunks of text that e.g. show all chunks of text a particular user found interesting or a set of users or by date or location. That will give you the best of both worlds - lots of content to SEO and no dup content penalty.

Best of luck!!

JeremyChase• 16 years ago

Or set the canonical address of your cached pages to the real page.

StavrosK18• 16 years ago

Do you mean the "base" tag? We already do set that (admittedly, after this happened and because of it).

JonM• 16 years ago

I honestly didn't know this existed.... thanks!

StavrosK18• 16 years ago

Ouch, we didn't know that :/ Thank you, I'll block robots from the cache right away!

StavrosK18• 16 years ago

For the record, something I forgot to mention. All the ads load for the site of the cached page, and all the links lead to it. So, the only way the original site is impacted is that they don't have to bear the load of 50k visits on their site. Otherwise, they get all the ad revenue, signups, links, etc.

Essentially, we're the only ones who don't gain anything from this.

bananaandapple4• 16 years ago

They could sue you for copyright infringment if they wanted.

And allowing google to index your "stolen" content pages is just outrageous. You don't own that content.

StavrosK18• 16 years ago

Then we could sue Google for copyright infringement for caching our pages, I guess... Why would we not allow Google to cache it? Each cached page has a great big box on top saying that this is the historious version of the cache and linking to the original site...

Example: http://cache.historious.net/cached/515865/

sounddust2• 16 years ago

There's a huge difference between what Google is doing and what you're doing.

1) Google is caching pages for a specific purpose and ensuring that they aren't cached/scraped by others:

http://webcache.googleusercontent.com/robots.txt

By not excluding robots, you're opening yourself to all kinds of situations where you are responsible for draining revenue from the owner of the content, which leaves you liable to lawsuits. By contrast, the way that Google caches content and their rules surrounding it do not generally harm the copyright owner.

2) Google honors all robots.txt, no-archive meta-tags, and other indications that the author doesn't want the page to be cached. Is historious doing the same?

bananaandapple4• 16 years ago

Indeed, what google is doing (caching a page and showing it to the user) is copyright infrigment in some countries. (e.g Belgium, ...).

There hasn't been any case against them but theoriticaly someone could sue them. Who will win is a different story.

jasonkester• 16 years ago

This is exactly what should happen when your low-traffic site suddenly gets 50k visits in a day: Absolutely Nothing. Kudos to the historio.us guys for building something that can actually handle a little spike in traffic without falling over.

We see so many sites come through here that are showing 500 errors by the time they get halfway up the front page that you'd think nobody knew how to do this stuff anymore.

Great job of ticking off the basics, guys. Building on top of that foundation, you shouldn't have any trouble scaling out when this sort of traffic starts becoming a daily occurrence.

StavrosK18• 16 years ago

Thank you! To give credit where it's due, buro9 from HN urged us to implement the Varnish caching feature for cached pages, so we had it written but disabled because we didn't think it was going to get that much traffic. We hadn't turned it on for the first 30k visitors and the server was still doing great, but when we saw it we just switched it on and load dropped to nothing.

Serious amounts of love for Varnish here.

How we got 100,000 visitors without noticing

Related Stories