StackOverflow Architecture (2016)

https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/

by enitihas • 9 years ago

122 37 9 years ago

number6• 9 years ago

Why Microsoft? Why IIS? Why asp.Net?

Was ist because there was little other alternatives or is there a rational behind this?

Coming from a Linux World in all Server I am ignorant to the benefits it might have. Or why to pay let alone the license for services and software that is enterprise ready and for free in Free Software Land

enitihas• 9 years ago

I too initially thought the same. We rarely see any post on someone using IIS on HN. However, if you take a look at the amount of resources they use(they say they need only one web server) , it is clear that their stack is working very well for them.

timrichard• 9 years ago

To truly consider this as "a benchmark of the MS stack at scale", it would only be fair to factor out HAProxy, Redis, and Elasticsearch. And I suspect it wouldn't fare too well.

nine_k• 9 years ago

They run very few servers, but the machines they use are huge dedicated servers, with as much RAM as the board would support. They published descriptions of their hardware stack several times.

This is very different from the typical modern architecture, with many smaller nodes scaled on demand in a cloud.

justboxing3• 9 years ago

Why Microsoft? Why IIS? Why asp.Net?

I believe it's because Joel Spolsky started his career on the Microsoft stack, and even worked at Microsoft [1] very early in his career. After the .COM crash, he started Fog Creek Software as a Consulting Company and at that point (he mentions it somewhere in his old blog posts) he was doing consulting for clients mostly on the Microsoft Stack.

Same holds true for Jeff Atwood. You can see the archives of his codinghorror.com blog to see that his background at that time was also heavily on the Microsoft stack.

Build stuff (esp. Side Projects) with the tools and technologies that you know.

Many wise-men here on HN have said something to this effect.

That's probably why.

That said, there's nothing wrong with their architecture.

not many developers realize that this site serves them with incredible efficiency. For example, a question and answer page is typically served in 10ms.

Source: Marco Cecconi: High Performance Architecture Of Stack Overflow - JSConf.Asia 2016 => https://www.youtube.com/watch?v=uNVlQ1yPsto

[1] Source: 2001 - Working on CityDesk, Part Three https://www.joelonsoftware.com/2001/10/17/working-on-citydes...

kemiller2002• 9 years ago

Spolsky had very little to do with the actual coding of it. If memory serves, it was his idea and he engaged Atwood to build it (hence the partnership). It was Atwood and two other developers, which I sadly cannot remember their names at the moment who did the actual programming. You are correct though, the reason Atwood picked the MS stack was that it's what he knew, and they had ties into MS itself. It was built on the original MVC framework, before the MS framework was released to the general public. If you go back to the original podcasts it talks about why they built SO the way they did at the time.

ge963• 9 years ago

Damn his blog is impressive 284 pages haha, going since 2004 wow

scrame• 9 years ago

In addition to what others mentioned, they were heavily promoting the bizspark program, which was free licenses for the MS stack for the first two years or something. They heavily promoted it in interviews and their other marketing channels, so I'm sure there was some extra kickback from that.

neverminder• 9 years ago

Couple of years ago I was at this event where Stackoverflow had their stand, so I asked them the ultimate question - what stack does Stackoverflow run on? When they answered ".NET" I couldn't believe it, thought it was a joke. Stackoverflow, one of the most prominent sites in the world of free and open software is actually running on a stack that is neither free nor very open.

johannes12343212• 9 years ago

In what sense is SO associated with free and open source? They run a commercial project on proprietary software and I assume most questions are by students and people working on non-open, commercial projects. True, some questions build around open source, but that's since many development stacks use openSource these days.

darklajid• 9 years ago

While that might've been true (if we discount projects like Mono), today asp.net is open and actually _developed_ in the open (aka 'better than Android') and dotnet core just reached version 2.0.

So .. I don't think what you're saying applies in today's world. The biggest (proprietary) dependency they have in this stack as far as I can tell? I'd guess it's Microsoft SQL Server.

romanovcode2• 9 years ago

neither free nor very open.

.NET is released under MIT license[0]. What the hell are you talking about.

[0] https://github.com/dotnet/coreclr/blob/master/LICENSE.TXT

user59944612• 9 years ago

>> Or why to pay let alone the license for services and software that is enterprise ready and for free in Free Software Land

Roughly speaking, all software are free.

The bulk of the money goes to pay the developers and the sysadmins around the software, not the software itself.

nafey• 9 years ago

I have had the impression that this statement is not true for enterprise software.

highace• 9 years ago

The MS stack is better than a lot of people give it credit for. Back in the old days of ASP.NET webforms I'd agree that it sucked, but things have changed a lot since then.

There's a web framework similar to RoR. C# is fast out of the box, strongly typed, and easy to work with. Also it's similar to JS so working with the two simultaneously isn't jarring. Visual Studio is arguably the best in class IDE. IIS is fast and can do everything you'd expect. Most tools have a GUI default, which really helped when I was learning.

Windows licensing costs are minimal compared to the total cost of a server (like an extra $20/month). I ran a db-driven site with 1,000-2,000 concurrent users, 30-50 reqs/s, and 40mil pageviews/month with a single $100/month Rackspace VM. SQL Server can get expensive when you start scaling up, but it's easy to switch over to Postgres when that happens.

Also I kinda like how steady it is. I got into development to build products, not to be constantly on the edge learning about the latest libraries or paradigm shifts. That's what frontend JS is for.

drharby• 9 years ago

thats what frontend js is for.

My sides have gone in orbit

mattmanser• 9 years ago

You can get it straight from the horse's mouth, they made when podcasts while building it. They're all still available and transcribed:

https://stackoverflow.fogbugz.com/default.asp?W4

Jeff + Joel decided they wanted to make a site together, mainly because experts-exchange.com, which was the biggest resource for us devs back then, were putting up paywalls while using dodgy SEO tactics. They refer to experts exchange as the site with the dash, some explanations in the first episode:

https://stackoverflow.fogbugz.com/default.asp?W6

A few episodes later, here's the explanation of why those chose .Net:

https://stackoverflow.fogbugz.com/default.asp?W56

The explanation's about a third of the way down (this is Jeff Atwood speaking):

We are not personally going to be language agnostic, because we need to actually build the site. And in terms of people actually working on it, Joel's in an advisory role, I'm gonna be writing code, and then a friend of mine, Jarrod, I'll be working very closely with. So it's sort of like 1.5 developers, so I need to actually get things done. In order to do that, I'm gonna fall back on what I know, and what I know is essentially ASP.NET. So ASP.NET is gonna be the platform. And I actually really like ASP.NET. There's things I don't like about the stack, like the webforms model, and so on and so forth, but as a core programming environment, I feel like it's more than up to the task, quite a bit of tooling around it that's gonna help us out. So I'm looking forward to that. I just want to get that out of the way early on as far as like technology choices. And also to segregate and say again, what we build the site in does not equate to what the ultimate StackOverflow site will be about, in the end. Technology choice doesn't matter. And Joel, you know that more than anybody else, with the whole Wasabi thing, right? Like--which I've come full circle on, I used to be very critical of that--and now I've decided, it doesn't matter. Because unless you're open-source, nobody's gonna actually see your--

When they first started working on the site ASP.Net MVC was just released, so they used it to avoid webforms. I'm sure they talk about it in later episodes.

Incidentally, when they were making stack overflow was when RoR was really just becoming the super-hip framework to use, but it was pretty slow back then. At the same time Django was also on the rise. If they'd have used RoR or Django I think they'd have had a lot more problems. PHP was still pretty icky and didn't have a decent MVC framework at the time.

At the time (and this probably still holds true) C# was much more powerful than Ruby, SQL Server was much more powerful that MySQL, so SO held together well even as it grew very fast, and the impression was that they got support from Microsoft because SO was a good advert on how powerful the .Net stack actually is.

I remember one problem they had with performance, but I thin it was caused because they'd put tons of logging in and it caused some sort of feedback loop.

Realistically, back then the choices were PHP without MVC, RoR, Django, ASP.Net WebForms, ASP.Net MVC or Java (Spring? Was never really in the loop with Java web development). Most people still opted for PHP.

romanovcode2• 9 years ago

Why not?

It seems it worked out pretty great in the end. Their infrastructure is better then 99% of Linux projects that are out there.

ramshanker• 9 years ago

With all the good and smart people over there, and considering that they have like <100 servers self-hosted, I really hoped they would be ready for IPv6 and set standards for others.

Nick: If you are reading this, any idea, when you guys would be ready. Wold be interesting to read the execution blog post.

http://ready.chair6.net/?url=stackoverflow.com

skrebbel• 9 years ago

I'm a networking noob. Why would they invest time in this? What's the benefit for them to run on IPv6? What's the benefit for you, a user?

y4mi• 9 years ago

ipv6 clients cant talk with ipv4 servers. so they need a packet translation, which increases the latency.

there shouldn't be any benefits in this particular case though, because endusers would never actually talk with the application servers.

the loadbalancers are probably the only public facing servers, which probably (didn't check) support both.

so communication is like this

    _client_ <== ipv6 ==> _loadbalancer_ <== ipv4 ==> _application server_

vs.

    _client_ <== ipv6 ==> _loadbalancer_ <== ipv6 ==> _application server_

the situation the GP was talking about is this:

    _client_ <== ipv6 ==> _ipv6 to ipv4 translation proxy_ <== ipv4 ==>  _loadbalancer_ <== ipv4 ==> _application server_

you could theoretically reduce minor latency by just addressing application servers directly

    _client_ <== ipv6 ==> _application server_

but thats a terrible idea from both reliability and security perspectives

callmekatootie• 9 years ago

Does anybody know if there is a similar blog for their Software Architecture? Particularly, how they have modelled the points system and the badges? Does it all exist in their models?

zython• 9 years ago

I enjoyed this very much. Any more of this ? I like highscalability.com but this is more hands on. Anyone have something like this ?

galeforcewinds• 9 years ago

Similar detail, itemized, and with hardware specs appears to be at https://meta.stackexchange.com/questions/10369/which-tools-a...

jlgaddis2• 9 years ago

Or, for damn near all the detail, just click on the third link in the article, "Stack Overflow: The Hardware - 2016 Edition" [0] -- and/or check out the rest of this series (conveniently mentioned and linked to at the very top of this article).

[0] https://nickcraver.com/blog/2016/03/29/stack-overflow-the-ha...

Decabytes• 9 years ago

The most interesting statistic here for me is the time spent processing SQL queries. It really only takes a few tens of ms in most cases to processes. But due to the sheer volume of requests over the course of a day,it ends up being days worth of time.

StackOverflow Architecture (2016)

Related Stories