Stack Overflow Architecture Update - Now At 95 Million Page Views A Month

http://highscalability.com/blog/2011/3/3/stack-overflow-architecture-update-now-at-95-million-page-vi.html

by ZeroMinx • 15 years ago

128 61 15 years ago

nopal3• 15 years ago

How did the code have to change to read from/write to Redis?

gecko• 15 years ago

I doubt very much. They're basically using Redis as a much more powerful memcached. SQL Server still is the ultimate owner of all StackOverflow data, last time I talked to the team.

nopal3• 15 years ago

I guess I just haven't seen memcached integration in .NET. All of the caching examples I've seen use the framework caching components.

I was also wondering if they'd developed any generic custom code to fetch from Redis or fetch from SQL/insert into Redis, and how it fits into their application logic. (e.g. does it happen before they make a LINQ to SQL query, or does it intercept LINQ to SQL queries and return results from Redis).

citricsquid4• 15 years ago

I use stackoverflow as the benchmark for success on my own* sites, we're still winning. We did 145m for February!

spoiledtechie• 15 years ago

whats your website?

ramidarigaz• 15 years ago

I believe criticsquid runs the Minecraft wiki...

citricsquid4• 15 years ago

It's not even related, I feel bad now because it's irrelevant to the post, it just happens that the first time Stackoverflow announced their traffic I mentioned ours and Joel (I think?) said something about it, so now I always use them as a benchmark. Anyway, minecraftforum.net and minecraftwiki.net, I don't own them any more (reason for the * -- curse.com acquired them). http://www.quantcast.com/minecraftwiki.net & http://www.quantcast.com/minecraftforum.net combined 145m for feb

rabc• 15 years ago

Really cool to see how they moved from Windows-only to other technologies like Redis.

They dind't jump into the hype at first version, they started using other things just now when they really needed that.

btipling• 15 years ago

Interesting on the switch from the Windows/MS stack. Is there not a decent No SQL option for a windows stack?

michaelbuckbee• 15 years ago

I'm not sure what switch you are referring to, they've been using HAProxy and Lucene for quite some time.

Also, what would a No SQL data store give them (for central data storage - not caching) that they aren't already handling with SQL? Keep in mind that they also have relatively easily partionable data (if necessary they can break off any StackExchange site and database to their own dedicated hardware).

mattbriggs3• 15 years ago

redis is nosql, and they are using it on linux

Athtar• 15 years ago

Raven DB (http://ravendb.net/) is a pretty good NoSQL option for the .NET/Windows platform.

vyrotek• 15 years ago

Wow, I had no idea this project existed. Thanks for the link!

I was looking for something like this a while ago for my startup but we ended up going with Azure and using their Table Storage.

Jamiecon3• 15 years ago

AppFabric Server is closer to Memcached than Redis. However since their persistence layer is SQL Server, they could have used AppFabric in place of Redis.

I'm guessing they didn't want to use a technology that is so recently out of beta perhaps?

grandalf3• 15 years ago

It's fascinating that a MS loyalist like Joel would end up using so much linux/oss software.

I wonder if Joel regrets not going 100% open source. I guess the cost of Microsoft licensing for all that is still very cheap if viewed as the price of being featured in Bing searches.

spolsky• 15 years ago

This might possibly be the most ridiculous comment I've ever seen on the internet, let alone Hacker News.

* I am not a Microsoft Loyalist

* I don't make the technical decisions. Why would I? I'm the person who knows the least

* It is not necessary to use Microsoft licensing to be featured in Bing searches... that is possibly the most preposterous thing I have ever heard in my life

* Microsoft licensing is a rounding error, with the possible exception of SQL Server licenses

* Your bizarre personification of Stack Overflow as "Joel" is charming and quaint and completely obnoxious to the 20-odd people at Stack Overflow that actually do the work instead of sitting around browsing hacker news like I do

MatiasNino• 15 years ago

Welcome to the modern Internet - where the digital equivalent of the crazy homeless zealot also happens to know your name and career details.

jswinghammer• 15 years ago

There is a group of a people who will get worked up if they think you would say anything neutral or positive about Microsoft products. I'm not sure why but a sizeable number of people believe that you must be getting bribes from Microsoft to use their software.

keithwarren• 15 years ago

That you would reply to this - makes me respect you even more

grandalf3• 15 years ago

1) I think the word "loyalist" was poorly chosen on my part. I meant it only to describe how Fog Creek has traditionally been a fairly Microsoft oriented shop... even after (in my opinion) open source languages/platforms began to surpass MS in usability, documentation, and community support.

2) While it may be the case that it is not necessary to use MS technology to be featured in Bing searches, I think you (Joel) are among the more influential developers in the world and that your firm leverages Microsoft's latest technology was likely a factor in the decision to "feature" the results instead of simply displaying them first when they're relevant. Maybe I'm cynical but that seems like a smart marketing move on Microsoft's part.

3) My mention of license pricing was intended to include all licensing costs, including SQL server. I'd guess the cost of your setup as described would be about $150K in licenses... Perhaps rounding error but still a nice developer salary.

4) No disrespect or insult was meant by my comment. I started out using 100% Microsoft tools and technologies, and I've been intrigued to see open source projects start to offer compelling value and features. Mostly my comment was intended to express surprise that it's not simply a matter of "rounding error" costs to do a 100% Microsoft solution for a major site... and that things like Redis and Ubuntu are part of the reason that StackOverflow is as performant as it is.

selectnull• 15 years ago

"...instead of sitting around browsing hacker news like I do."

I think I'll redefine my definition of success to include this. :)

kprobst2• 15 years ago

I'm going to guess it's just a result of just using the best tools for the job. His developers know the MS stack, so it makes sense to build the application on top of that. Things like Redis and HAProxy are traditionally better left to run on a *nix server. I use MongoDB on Server 2008 for testing and whatnot, but once that goes to production it's going into a Linux box.

That the opposite isn't generally true (a mostly FOSS project using Microsoft bits) is a testament to what's available on the FOSS side, of course.

Your use of the word "loyalist" implies some kind of blind joined-at-the-hip fanboysm is at play on Spolsky's brain. I think that's probably very far from the truth. Those of us who use and take advantage of Microsoft technologies are also quite capable of recognizing that some things are better done on other platforms. I.e., best tool for the job.

kenjackson3• 15 years ago

I think this is one of the things that a lot of people in the FOSS don't get. Most ppl I know on the MS stack use it because its the best tool for their job. But will also use FOSS where it makes sense. They're just not as ideological, and are much more technology driven.

I suspect the staff at Stack Overflow never blinked at using FOSS software. I suspect they said, "we need capability X, lets find the best tool to handle it".

mythz2• 15 years ago

I really wish it were the case that most devs on the MS-stack don't drink the kool-aid and use all-things-Microsoft. There are still devs using DataSet's in VB.NET, pushing the SOAP envelope, using the MS Patterns & Practices stack and continue to develop ASP.NET web apps like state-heavy windows apps since that's what they were told to do.

I don't blame anyone for forming a stereo type on .NET devs, considering pre-ASP.NET MVC (years late to the game), there was no competing web framework to the anti-web state-heavy ASP.NET, simply because IMHO there was none sanctioned for use by MS. Sadly there is an inherent Microsoft culture that will only believe and advocate anything as long as they've read it on microsoft.com.

Despite their failed DAL's and poor application frameworks, Truth is IMHO Microsoft have created a superior development platform with C#/.NET and with VS.NET/R# by the C# devs side, it's one of the most productive ones to use.

Fortunately there are free-thinking pro developers, not constrained to think inside MS toolbox, who just want to use the best software they can use, regardless of the culture. Interestingly it seems a lot of the time C#/FOSS devs will independently come to the same conclusions on best-of-class FOSS software to use: e.g. nginx / redis / linux / naigos -- i.e. the best tool for the job.

StackOverflow is clearly one such 'pro-dev company' not shackled to the MS Platform nor will purposely/religiously maintain an anti-MS stance since they'll continue to use MS tools and products where they think to be highest quality (i.e. C# / IIS7 / SQL Server).

It's quite clear StackOverflow has a job to do, and they'll end up choosing the best tools they think to do it. I believe this pro-dev, high-quality culture is why they've become so popular, providing some of the most enjoyable website experiences on the Internet - all without skipping a beat, or with a hint of growing pains.

mattbriggs3• 15 years ago

I find the opposite. Back when I was working in .net, I remember one argument I got into with my boss with regards to software evaluation. MS has a testing framework and a build tool that are basically a generic xunit and an ant clone. Now, there are a great many better xunit implementations on .net, and I think we can all agree at this point that xml isn't the greatest programming language for build scripts. I was pointing out various choices that beat out the MS ones in terms of community, maturity, and features. His argument was basically that if MS doesn't have the best product, eventually they will make it better and it will become the best.

Every community has strengths and weaknesses. IMO the big weakness of the .net community is also the worst thing about MS itself from a tech point of view - a real unwillingness to look at what everyone else in the world is doing. Because of that, you see this strange effect where enterprise trends seem to happen about 5 years later in .net then they do in other communities. For example, it is only extremely recently (like in the last two years) that it started to become normal for .net shops to use ORMs, and the hot debates over whether or not to unit test are just starting to die down. And it is really in the last year that the most bleeding edge shops have started to look at DCVS (the cutting edge ones are on svn, and the majority on a cvs type system).

And for the record, I don't think .net is a bad platform or anything like that. What I do think is that the single thing that could improve most .net shops is being more open to non ms technology.

mymex1• 15 years ago

The site states that they are using a WISC stack via BizSpark, which gives companies free use of a TON of tools and software for something like 4 years or once they make over $1 million in revenue. Sure, you could also do this for free using open source tools, but I would GLADLY pay a couple grand in licensing fees if my company was making over a $million in revenue using the Microsoft stack.I think the fact that StackOverflow uses a WISC stack is a testament to its reliability and robustness. Hey, whatever gets the job done.

ig1• 15 years ago

I'd presume that they've broken the million dollar bizspark limit or they're doing something very wrong...

matthewcarriere• 15 years ago

While the comment might have come off as "ridiculous" I think the underlying theme is quite pertinent.

First, in response to Joel's comment, obviously Stack Overflow is personified as "Joel", the same way Apple is "Steve" and Windows is "Bill". What I find charming is your personification of it as collection of individuals who are are equally invested... it must be ponies and rainbows over there.

What's pertinent about the comment and probably at the heart of his "fascination" (mine too) is how this product is held up as a poster child for ASP.NET MVC, without giving the "whole story". The is very common of the Microsoft evangelists. Let's be clear, I am not accusing Joel of being a "loyalist" I think he is probably only pinned as one because of these evangelists running around trumpeting the use of ASP.NET MVC for this successful site. The true MS loyalists will never read this article or decide that Stack Overflow just "doesn't know about" the correct MS technology stack that could eliminate his need for any Open source.

So that's what's "fascinating". That an ASP.NET MVC application did not stay inside the reservation.

As for the right tool for the job nonsense, let's cut the bullshit. We all make religious decisions when we "start" a software project. You build out an idea your excited about in a language/platform your excited about. What's interesting here is that the real world of success has demanded an open source stack... and it could very well be cost and not capability. You are in FANTASY LAND if you think you would be paying a "couple grand" in licensing to scale out to what Stack Overflow must need to run its service. Add about 3 more zeros to that guess-timate.

End of the day, I don't think anyone regrets not going 100% open source you build it in what your excited about. The story I would be most interested in is what performance/cost analysis was done that steered them in the technology directions they took. I think we could all learn from that.

citricsquid4• 15 years ago

Ah I meant to upvote your comment and hit the down arrow, sorry.

kenjackson3• 15 years ago

As for the right tool for the job nonsense, let's cut the bullshit. We all make religious decisions when we "start" a software project.

I think you speak for yourself here. The fact that you can't even imagine that people don't make religious decisions at the start of a project is telling.

mistermann• 15 years ago

>You are in FANTASY LAND if you think you would be paying a "couple grand" in licensing to scale out to what Stack Overflow must need to run its service. Add about 3 more zeros to that guess-timate.

Couple grand = $2000 Add 3 more zeros: $2,000,000

Stackoverflow is paying $2 million in software licensing fees! Wow, using the Microsoft stack is expensive!!

KyleBrandt• 15 years ago

People rarely pay sticker price for this stuff. Right now SO is still using bizspark, but lets pretend we are paying sticker for our NY Datacenter:

* 10x Windows Server Standard R2 for the Web Tier (~1k per)

* 4x Enterprise OS (4k) for the DBs (were are going to give SO its own DB soon)

* 8 Sockets of SQL Server Enterprise R2 (27k) .

So 10x1000 +4x4000 + 8x27000 ~= 242k sticker price. Again people don't tend to pay sticker price for this stuff -- but this guesstimate is not $2 Million by any means.

References: http://www.microsoft.com/sqlserver/2008/en/us/pricing.aspx http://www.microsoft.com/windowsserver2008/en/us/pricing.asp...

kprobst2• 15 years ago

There are Java projects that use Oracle as the back end. Since Java is Free Software (technically at least) and Oracle is not, would you consider that to be a failure for Free Software?

Do you use a binary blob NVidia driver on your Linux computer? Would that be a case of not staying in the reservation?

You're a Rails developer, aren't you? twitter was hailed for a while as validation for Rails and Ruby in general. That is, until they realized it wasn't scaling and had to use something else. Failure for RoR? I guess they weren't religious enough (as you say) to stick with what they had and gosh darn it, make it work at all costs. After all, you can run your blog on RoR, just like you can on MVC. If you imply that using Redis on Linux to scale invalidates MVC as a viable platform, how is using TokyoCabinet (or whatever) to scale RoR any different? Unless of course your argument is purely ideological.

We make decisions on what technology to use based on what we know. On our areas of expertise and knowledge. If we pay X for something, it's because we believe that X is a good price for what we're getting back. If we decide to use Y because it's free, it's because it works the same as Z, which is expensive or limited.

If there are "true" MS loyalists out there who make decisions based on their adoration of the company, then that's their problem. More power to them and all that.

And by the way, if you feel that this represents a form of "capitulation" for the MS stack, then you need to go talk to MS about it, not StackOverflow. Because they made their technical decisions based on practicality, not religion (as you say). And besides, if it's supposed to be so embarrassing, why mention it at all? No one is forcing them to disclose what platform they happen to be running their key value store on.

razvan346• 15 years ago

T9space.com up to 42 Million pageviews a month: http://i.imgur.com/Vggvl.png 10 App servers dynamically converting websites to mobile versions + 1 MySQL DB + Haproxy LB + 1 Lucene Search

carson• 15 years ago

For those who aren't reading it the ServerFault blog http://blog.serverfault.com/ is a treasure trove of information. The work they are doing and blogging about on the systems side is something a lot of people take for granted.

moe6• 15 years ago

Hm. I love Stack Overflow but architecture only starts to become interesting at 4-5 times their traffic.

95M/mo translates to a mere 36/sec avg. A single web- and db-server will handle that without breaking a sweat, although you of course want some more machines for peaks, redundancy and comfort.

citricsquid4• 15 years ago

36/sec but then they're doing 800 http requests per second, not sure what could cause such a big difference (lots of ajax stuff?) but I think 800/sec is an interesting thing to see them deal with, that's something in the region of 1bn a month, which fits your requirement for interesting-ness :p

icey• 15 years ago

Perhaps the 800 req/second number is a peak figure? 36 requests a second average isn't meaningful when your traffic is spiky.

jonknee• 15 years ago

Probably a combination of other page resources (images, JS, stylesheets, etc), bots and peak access times.

moe6• 15 years ago

which fits your requirement for interesting-ness

Sorry, the threshold of interestingness for serving static assets is even higher. ;)

A single nginx on a moderate host will barely warm up before ~10k reqs/sec. The network tends to be the bottleneck here. Anyways, ~800/s should be doable from a $10 VPS.

PostOnce• 15 years ago

I wish I had to figure out how to deal with a "mere 36/sec avg" number of visitors. I'd sell my company and retire.

siavash• 15 years ago

We (blocket.se) are hitting almost 1b/mo and I can assure you that it's irrelevant wether the average p/v per second can be handled by some basic hardware or not. During peak hours it's an order of magnitude difference. Worth mentioning though that we are tightly focused in only one country, and probably have larger variations during peak hours than stack overflow does. I'm not sure what our traffic looks like combined with our other local sites.

moe6• 15 years ago

If your peak hours exceed average by an order of magnitude then you have oddly shaped traffic. Most sites follow a bell curve where regular peak hours range around factor 2-3 above average.

Either way I didn't mean to discount their efforts. Was just trying to point out that their architecture is not very interesting from a performance point of view, yet.

IDisposableHero2• 15 years ago

architecture only starts to become interesting at 4-5 times their traffic.

Which suggests that even sites with a medium-to-large internet presence, which dominatre thier niche, can do so with "uninteresting" architecture.

moe6• 15 years ago

Which suggests that even sites with a medium-to-large internet presence [...] can do so with "uninteresting" architecture

Absolutely. Hardware is evolving so fast (Moore's Law & friends) that we humans have a hard time to keep up mentally.

Go back six years in time and the traffic they're dealing would have required roughly 24 servers, instead of the 3 that they have today.

This is of course a rough extrapolation and six years seems like an eternity on the internet-calendar.

However we're quickly approaching a point where there's only two scales left to worry about: "normal" and "web-scale", with only a few hundred sites falling into the latter category.

bluesnowmonkey• 15 years ago

> All pages accessed by (and subsequently served to) annonymous users are cached via Output Caching.

I'm surprised they don't cache output on logged-in pages as well, using ESI or some kind of client-side replacement. There's not that much personalization going on. Most page content is the same between users.

Jamiecon3• 15 years ago

I would imagine that they calculated it would be cheaper to scale up their application servers and backend caching than pay for the development resource to facilitate this change.

keyle• 15 years ago

I question the truth in that article and whether they really use Razor already. It's barely out, so is MVC3 and I haven't noticed any changes that would suggest that they would have converted it all from MVC2 to MVC3. It's a major task.

They might have some Razor in there because it is backwards compatible but to have it all razor is a stretch.

Jamiecon3• 15 years ago

We're using ASP.NET 4 and MVC 3 in production. New features in both made it very worthwhile to invest the time in conversion.

We are not using Razor yet, but it's feasible they are. Take a look at this: https://github.com/telerik/razor-converter

ralphc• 15 years ago

I understand the tradeoffs between Windows and Linux and choosing one or the other for a particular task, buy why run Ubuntu Server on one machine and CentOS on others? Are the tools that Linux specific that it's worth keeping track of two versions, two package managers, etc?

Stack Overflow Architecture Update - Now At 95 Million Page Views A Month

Related Stories