The architecture of Stack Overflow [video]

http://www.dev-metal.com/architecture-stackoverflow/

by schmylan • 13 years ago

98 48 13 years ago

schmylan2• 13 years ago

Before the title was moderated there was an important tidbit. StackOverflow doesn't unit-test. Fascinating.

schmylan2• 13 years ago

tldw; He says he doesn't advocate it but they get away with it by having the community test it out for them in their meta site. Then the community writes up the bugs.

merak1363• 13 years ago

He actually says " I'm not advocating that you shouldn't put in tests. [ The reason we can get away with this ] is that we have a great community. "

I take this to mean that he feels that StackOverflow doesn't need tests. Not that tests are useless.

carsongross3• 13 years ago

The most important thing, technically, is having great developers who ship.

For piths sake, I want to say "Everything else is noise" but that isn't true. Everything else can help or hurt, depending on the application and how doctrinaire the application of a given approach/methodology is, the organizational knock on effects (e.g. "Mr Tough Guy Testalot" holds up the release train or nukes your architecture to make it 'testable'), etc. but, seriously, "great developers who ship" is really what moves the needle.

WestCoastJustin• 13 years ago

Having a great Ops staff also helps ;) Of note is Thomas Limoncelli who wrote "The Practice of System and Network Administration" [1] and "Time Management for System Administrators" [2] works for Stack Exchange (formerly at Google). The Practice of System and Network Administration is basically the bible for most sysadmins, myself included.

ps. I only singled Thomas Limoncelli out as an example just to highlight the caliber of their Ops staff.

[1] http://www.amazon.com/Practice-System-Network-Administration...

[2] http://www.amazon.com/Management-System-Administrators-Thoma...

carsongross3• 13 years ago

Violently agree.

UK-AL• 13 years ago

Why not combine both?

Every feature/user story has to go through a workflow of selected for development -> UX design(If required) -> Development -> Unit Tests(Or the otherway around) -> Staging -> Load Test -> Acceptance Test -> Production -> Analytics(To see if people actually use it) -> Learn from analytics -> back to start if required.

The goal is to get as many issues through the workflow as fast and rigoursly(no shortcuts) as possible at a sustainable pace. Have a continuous flow of features rolling out through this process. Ideally with continuous delivery to automate the majority of it.

DavidWoof• 13 years ago

Everything else is noise. If you have great developers who ship, then by definition you don't have doctrinaire methodology or "Mr Tough Guy Testalot" (I generally find "Mr No Test" to be a much bigger problem anyway). You might the situation where you have great devs but bad management, but that's next to impossible in the real world.

There's really only two steps to great software development.

1. Hire good developers.

2. Don't hire bad developers

esw• 13 years ago

Here are the slides for anyone who's interested: https://speakerdeck.com/sklivvz/the-architecture-of-stackove...

merak1363• 13 years ago

Some points that I find interesting:

[1] StackOverflow has VERY FEW tests. He says that StackOverflow doesn't use many unit tests because of their active community and heavy usage of static code.

[2] Most StackOverflow employees work remotely. This is very different than a lot of companies that are now trying to force employees back into an office.

[3] Heavy usage of Static classes and methods. His main argument is that this gives them better performance than a more standard OO approach.

[4] Caching even simple pages in order to avoid performance issues caused by garbage collection.

[5] They don't worry about making a "Square Wheel". If their developers can write something more lightweight than an already developed alternative, they do! This is very different from the normal mindset of " don't reinvent the wheel ".

[6] Always using multiple monitors. I love this. I feel like my productivity is nearly halved when I am working on one tiny screen.

Overall, I was surprised at how few of the "norms" that they follow. Either way, seems like it could be a pretty cool place to work.

tegeek• 13 years ago

[7]. Millions of page views and just 25 servers for whole infrastructure, which includes everything including load balancing, cache, dbs' etc. etc.

Seems very very optimized and cost effective. Its brilliant.

thedufer2• 13 years ago

StackOverflow employees work from home.

Many do, but they have a fairly large office in NYC and a smaller one in London.

merak1363• 13 years ago

You are correct. I edited my post and actually found a good blog post on the subject.

http://blog.stackoverflow.com/2013/02/why-we-still-believe-i...

kmontrose5• 13 years ago

The Stack Overflow Q&A dev team has 2 people in New York, out of a team of 10 team. The Careers dev team is more New York heavy, 3 remote and 5 in New York. The sysadmin team is also quite remote, though I don't know the breakdown offhand.

I believe at this point most new technical hires are remote.

Our offices are mostly sales, Denver and London exclusively so.

vladimirralev2• 13 years ago

I see more and more static methods and classes last 2 years maybe. It's probably more about the stateless design and less side effects, but it definitely helps garbage collection if you avoid classes at session scope or smaller. In OOP there is another pattern that helps - object pools, but it's a lot of work to get it to work correctly and it's not as efficient.

edwinnathaniel2• 13 years ago

If most of your classes are small, I don't see why people have to resort to static methods.

If your methods are static, there are tendency/lust to use static member variables (hence stateful) which will cause side effects.

Don't forget the following points too:

1) You still have pooled objects somewhere (stateless business logic classes like XYZServices, repository classes that may be backed by pooled DB connections and Transaction Managers) provided/managed by your Application Server or by 3rd-party framework (Spring does this).

2) Your Application Server tend to have beefy hardware, good enough not to care of GC hiccups.

There are other reasons to use static methods but I don't think they're strong enough in this case.

JasonPunyon2• 13 years ago

It's not that no one follows the norms or tests. On the Careers team we do much more automated testing because there's money and literally people's jobs at stake. We have unit tests, integration tests and UI tests that all run on every push. All the tests must succeed before a production build run is even possible.

y0ghur7_xxx• 13 years ago

I would love to know more about the Databases:

- Are they used for different things on the sites?

- Is data partitioned across tables?

- Are they all SQL Server instances?

zero1zero• 13 years ago

I would like to know more about this as well.

It sounds like they are all SQL Server instances. However, he made it seem like they are reproducing the schema once per site? I.e., a separate database per site rather than sharding the shared data to multiple hosts per site. Did I hear this right in the question/answer portion?

kmontrose5• 13 years ago

Stack Exchange has one database per-site, so Stack Overflow gets on, Super User gets one, Server Fault gets one, and so on. The schema for these is the same.

There are a few wrinkles. There is one "network wide" database which has things like login credentials, and aggregated data (mostly exposed through stackexchange.com user profiles, or APIs). Careers Stack Overflow, stackexchange.com, and Area 51 all have their own unique database schema.

All databases are MS SQL Server.

notastartup• 13 years ago

is there an open source, self-hosted version of stack overflow that you can deploy on your own domain?

robzienert• 13 years ago

Yes. http://meta.stackoverflow.com/questions/2267/stack-overflow-...

m_myers• 13 years ago

To be clear: there is no version of the actual Stack Overflow code that is publicly available. There are, however, numerous open-source reimplementations of portions of the site code.

Also (as the video perhaps mentioned), the Stack Overflow developers have often been able to spin off pieces of the code as open-source libraries. See http://blog.stackoverflow.com/2012/02/stack-exchange-open-so...

skittles• 13 years ago

He mentioned that they use the servicestack.text library. I've looked into servicestack recently (using the nuget packages), but then found the library to be pay-to-play. There's an older version (v3) that is BSD licensed that is being maintained. Do any of you have experience with it? I have grown tired of Microsoft pushing new solutions to the same problem (REST service with WCF and then Asp.net web api).

sklivvz19713• 13 years ago

We used it at the time I gave that talk, we don't anymore. We only used JSON serialization and we have rolled out our own free solution, Jil.

https://github.com/kevin-montrose/Jil

kmontrose5• 13 years ago

Technically we use Newtonsoft and Jil, Jil replacing Newtonsoft as we become increasingly confident in it.

I wouldn't suggest anyone use Jil in a production role unless you're at Stack Overflow. It's too untested at the moment, and the typical person can't get me on the horn to fix whatever just broke.

dan_b• 13 years ago

ServiceStack is just plain awesome when it comes to developing web services, though it's gone commercial for v4 onward. Nancy is another popular alternative - it's basically Sinatra for .Net. Every time I go back to WCF I want to stab myself in the face.

dlazerka• 13 years ago

I wouldn't trust Joel Spolsky's code expertise -- just look at Excel internals! Nevertheless, Stack Overflow is super cool. But that tells nothing about its architectural quality.

alexgartrell• 13 years ago

Dear any Stack Overflow Developers,

Can you describe the network infrastructure in finer detail? Specifically what type of load balancer are you running?

And what's peak RPS? Where are your network peaks? (I'm guessing major peak US Pacific and minor US Atlantic?)

TacticalCoder• 13 years ago

IIRC at first they had an entire Microsoft stack (I may be mistaken on that).

But nowadays, from what I've read here on HN by SE devs in other threads, they're using lots and lots and lots of Linux: HAProxy, Redis, Nagios, etc.

I just double-checked the slide and although I didn't notice it at first, you can see that 'HA Proxy' and 'Redis' are mentioned.

The core Q&A is in C#/MS-SQL so that's probably not going to move to Linux anytime soon.

dimension64• 13 years ago

This might be a stackoverflow question, so what is a static code?

The architecture of Stack Overflow [video]

Related Stories