Building Products at SoundCloud – Part I: Dealing with the Monolith

http://developers.soundcloud.com/blog/building-products-at-soundcloud-part-1-dealing-with-the-monolith

by ABS • 12 years ago

66 19 12 years ago

developers.soundcloud.com

jbwyme2• 12 years ago

Loose coupling is awesome and dogfooding your public API is smart as well. However, it seems like there is a lack of strong technical leadership and a lot of big decisions are being made without thinking them through. For instance, "We were constrained by the same rules we enforced on third-party applications... It was not possible, for example, for a microservice to notify users about activity on private tracks". This type of constraint should be immediately obvious to the person/people making large architectural decisions. Throughout the article there are a few examples of "we tried this but shortly found out it wouldn't work". I may be wrong though and this whole post was actually the result of an hour meeting to discuss new architecture but phrased to sound like you actually tried all of those approaches.

Xorlev• 12 years ago

Having been through a similar decoupling, sometimes that few hour meeting would end on a positive note, "this looks good guys, lets get to work." Only for a few hours later, someone gets into a new mindset only to realize a few aspects need reevaluated.

rock_hard3• 12 years ago

I agree with you. Looks like poor engineering leadership.

Guess its one of the disadvantages when you are not based in the Silicon Valley...you simply don't have access to 1st class talent that has done the same task at 2 other high growth startups before.

alexcason• 12 years ago

This is what arrogance looks like folks.

phillmv2• 12 years ago

There are network effects regarding the portion of outliers that SV captures, but if we're talking about experience alone then I'm skeptical of your claims given how much of SV is driven by freshly minted CS grads.

edwinnathaniel3• 12 years ago

The disadvantage here is money and VC.

Where else you can burn money and not making profits in the name of engineering efforts?

phillmv2• 12 years ago

You're reading a lot into a single sentence fragment, and an awful lot of these decisions are not obvious a priori. In this case, the impression I got was "separating a large monolithic app is a risky, expensive move" and that concern overrode other practical concerns.

I doubt these constraints came about as surprises once their hand was forced.

Smudge• 12 years ago

it seems like there is a lack of strong technical leadership and a lot of big decisions are being made without thinking them through

I didn't get that impression at all. The ability to do it right (tm) the first time is quite often hindered by the need to iterate quickly and focus on growth and/or revenue. You'll eventually end up with an increasingly-more-painful monolithic core system, but that's an okay trade off if it got you to a place where your business really takes off.

At that point you have a few options. You can do a major rewrite, re-architecting everything from the ground up. Or, you can take Soundcloud's approach, and slowly migrate the old system to a more service-oriented architecture, reducing the amount of dev resources you need to throw around.

The trick is that the slow migration will be different for every business, and it depends on the needs of the platform and the way things were constructed initially. Just like you can't blame the technical leadership the trade-offs made early on, and you can't blame them for experimenting and iterating before finding solutions that worked for them. (After all, hindsight is 20/20.)

jbwyme2• 12 years ago

I think iterating quickly is great for product decisions where you don't know how people will react. Technical decisions, however, can be objectively evaluated up front given the right experience. I don't think a migration from a monolithic system is particularly exceptional.

easymovet• 12 years ago

Great product with some impressive traffic numbers! Looks like a good candidate for flow based programming. side-note: I'd happily pay for streaming that had less compression applied.

dmur2• 12 years ago

As an outsider to scaling issues, what strikes me about SoundCloud's struggles with shedding their Rails past is how similar their problems sound to Twitter's problems. Granted, a microblogging service has a different set of concerns than an audio delivery service, but the pain is the same.

Going out on a limb, perhaps this similarity contributes to the reasoning for the Twitter/SoundCloud acquisition rumors: more exciting problems for Twitter engineers to solve now that they've tamed their own beast? Or to put the theory in a business light, Twitter engineering leadership may have confidence they can architect SoundCloud better than the SoundCloud team has done.

joshdance• 12 years ago

My company currently is facing the task of breaking up the monolith. Not going to be fun. Glad that others had this problem as well and solved it in manageable ways. Thanks for sharing.

Building Products at SoundCloud – Part I: Dealing with the Monolith

Related Stories