Architecting Back end for a Social Product

http://highscalability.com/blog/2015/7/22/architecting-backend-for-a-social-product.html?utm_source=feedburner&utm_medium=twitter&utm_campaign=Feed%3A+HighScalability+%28High+Scalability%29

by kilimchoi • 11 years ago

4 5 11 years ago

highscalability.com

cihangirsavas2• 11 years ago

having worked on & maintained & architected 5 different social backends and failed at 4 of them, here are my key take aways;

data storage: * keep data structure as simple as possible, * do not go with the hypes, stick to old?, proven technologies * have your db constraints as strict as possible in early stages, later you can remove them as for performance improvements * test the key features of your database choices(does transparent sharding really works?), you will see them failing... * you dont need a graph db, you need a graph-like access layer to your data * your >1 month old data wont be accessed/modified at all (mostly) chose your shard key accordingly * keep duplication of your data as small as possible in early stages

indexing: * >1 month old thing might not apply here * ACL can be problematic/hard to manage, try to keep it simple.

queuing: * to me this is one of the most important elements, if you want a simple way to keep every component in sync with others, use event publishing, really. * have your re-try mechanism, dead letter queue, delayed processing in place.

We can use ActiveMQ, which is the most reliable queuing software. a very bold statement :)

caching: * think your cache invalidation strategy from the beginning * keep your immutable and dynamic data cache separately in your code (at least visually) * try not to mix your business logic and cache code

We should be ready for anything in the order of billions of queries per seconds. if only you are next/already facebook...

manig2• 11 years ago

Thank you for reading this and sharing your thoughts.

mianos• 11 years ago

Pretty good and much the same sensible architectural choices as many other already successful companies aside from the neo4j choice. Just use a normal db like postgres like everyone else. There are good reasons why the biggest sites in the world use or have moved to using an RDBMS. They may not be cool but they work. I'll not expand on all the reasons but a trivial search will enumerate them extensively. You already have some cool stuff like mongo, a store I love, and hate. ActiveMQ? I can't think of anyone else using it. Is there a good reason why everyone seems to be using rabbitmq?

cihangirsavas2• 11 years ago

Is there a good reason why everyone seems to be using rabbitmq?

it simply does work, has extensive language support (client packages), plenty of good documentation

manig2• 11 years ago

Thank you for reading this and sharing your feedback. Well this is more of a template to get started. It is not really "one size fits all" solution. I agree, instead of getting carried away by all these no-sql solutions, one may get started with RDBMS and then as they grow, may take a decision to move towards a no-sql solution or what ever solves the problem. But from my experience I can tell you that, if things go "as-planned" in terms of data and traffic, soon one will have to move a part of their data to a no-sql store.

> ActiveMQ? I can't think of anyone else using it

I have used ActiveMQ multiple times, and found it to be a stable MOM. I love its scheduler support and some of its cool features like message grouping, virtual destination, composite destinations and wildcards. Of course, like you said there are many other alternatives too.

Architecting Back end for a Social Product

Related Stories