Systems Design for Advanced Beginners

https://robertheaton.com/2020/04/06/systems-design-for-advanced-beginners/

by _ttg • 6 years ago

945 110 6 years ago

Nice tutorial! One small nitpick:

Calculating a hash value from an input is computationally very easy, but reversing the transformation and recovering the original input from its hash value takes so much time and computing power that it is, practically-speaking, impossible.

The above is true for encryption but not for hash codes. Recovering the original input from a hash code is not just practically impossible; it's provably impossible -- even with an infinite amount of computing power -- because in general a hash code contains less information than the original text.

lamida• 6 years ago

I think the author means recovering the hash by using dictionary attacks

exdsq2• 6 years ago

You’d still have hash collisions

sz4kerto• 6 years ago

Actually, it might be occasionally possible. Not perfectly, of course. If the input space is limited and you have other constraints on it, then you might be able to find the original text. For example, let's say your input is a 280 character long tweet, the hash is only 16 bytes long. If you find that "I AM A STABLE GENIUS" is one of the inputs that hashes to the given hash value, then you can be reasonably sure that this was the original text, given that most of the other inputs with the same hash won't be meaningful English sentences.

talaketu• 6 years ago

I'd say the count of believable English tweets would be many orders of magnitude greater the count of 16 byte hashes, so in general we can't just find a believable tweet and declare that we found the plaintext. There would be many possible collision.

The phrase you mentioned is notable (with a slight edit) - so I suppose you mean you could index phrases by notability, then it may be tractable find tweets of notable phrases, since the number of notable phrases is relatively low.

KMag• 6 years ago

I think the author is just using imprecise language for a preimage attack.[0]

[0] https://en.wikipedia.org/wiki/Preimage_attack

oconnor663• 6 years ago

Information is sometimes destroyed, but we can't count on it. An algorithm like HMAC might be used to authenticate a short message. Or even if the message is long, it might also be known, so perhaps the only unknown input (the key) is short. HMAC still provides security in this case, but it's not provable-because-information-is-destroyed in the sense you're describing.

netsectoday2• 6 years ago

215D70DA68DBDC2217D13916E40F89B87DDC3DAACA99892C943E0DCC76C8C1AF68097C86CC4CB53F15BD6C3870418A1E3577A076611A5104C6D4FC33939A21FB

dreamcompiler2• 6 years ago

I said "in general".

imvetri• 6 years ago

Title corrected - Web application system design for advanced beginners

fouc• 6 years ago

Yeah, it's not true "Systems design" since it's specific to programming

pc9• 6 years ago

How can I gain practical experience in these things? The jobs I've had mainly revolve around adding new features, not doing any of the things described in the article. Does working at bigger companies actually give you experience with this?

qppo• 6 years ago

Work at a startup as the systems person, because they don't already have one

taigi1002• 6 years ago

Yes and no, depends. It gives such experience to a software architect, otherwise not really. Tho, software architects do a lot more than just systems design (tho, that is pretty much their main task). But yes, you get a job as a software architect or something like an "in-training" one, helping one, etc. at a big corporation to get such experience (I'd argue that small companies only need some code design, you want a lot of systems/software design you should go for a big corporation)

mav3rick2• 6 years ago

You evolve into the architect role as an IC. No new grad is hired like this. You design things and keep going up in scale.

PeterStuer• 6 years ago

Enterprise architects usually do not work at large enterprises, but consult to them. In consulting you typically would have risen through the dev stages (junior 1st year, medior 2nd-3rd, senior from 2nd or 3rd onwards (ikr), then onto solution architect (tech lead on small projects ~300k), a few technical analyst team roles on large projects (several M$), experience in a key role in such a project such as Lead Integration Architect, en then onto Enterprise Architect jobs. Note that for anything beyond the solution architect stage your political and managerial skills will become as or even more important than your technical progression.

Alternatively you could be the tech person in a 2 man startup that lucks onto success and grab onto your seat in a wild ride and just never let go.

zachrose• 6 years ago

Does working at bigger companies actually give you experience with this?

If you can find someone who needs something for their business or organization, you'll learn a lot of this by working backwards from what they need and building it. So I would suggest the opposite: the smaller the organization the more you can jump around and set up the basics of all these things.

alienspaces• 6 years ago

Build something that exercises the gaps you want to fill and you'll learn from it. Make your code open source and there's your proof of knowledge. Unless the thing you build is in production with reasonable traffic you won't get to truly experience some of the good or not-so-good decisions you made but at least you made decisions and have some experience.

Davertron2• 6 years ago

Where I work, we have Architects but they often play a supporting/consulting role and the teams are expected to learn and implement most of the things listed in this article. They will push you to make good decisions, and will want to vet your designs, but they aren't dictating this stuff from on high (unless of course you're making really poor choices). It's been great because no matter who you are or what role you play on the team (I'm primarily a FE dev) you can learn as much of this stuff as you want. I have to imagine its the same at other companies, so anyway, what I'm saying is you can get a ton of experience in all of this kind of stuff without being an actual Architect at some places (we're a large company). The flip side of that coin here is that it's VERY difficult to get the actual title of Architect; it's a very limited role and it takes a lot more than just systems knowledge to get there (i.e. politics). A lot of the people I've worked with who are Senior level could probably easily transition into a systems architect position at other companies. I'm not sure if this is the best way to get there if that's what you're interested in though, just giving you some food for thought. I don't know how you identify companies where you can get this kind of experience, but if you're interviewing you should ask about "you build it you run it" or stuff like that to see how they manage their infrastructure, that might give you a hint.

Hyperborian• 6 years ago

How do they store their data?

SQL.

How do their different applications talk to each other?

Proprietary APIs.

How do they scale their systems to work for millions of users?

T H E C L O U D

How do they keep them secure?

They just... don't.

How do they make sure nothing goes wrong?

They just... don't.

What are APIs, webhooks and client libraries, when you really get down to it?

Easily outsourced to India.

mberning• 6 years ago

That last point is so painful and true. I work with a product that has dozens if not hundreds of “connectors” to interface with other systems and you can tell that most of them were farmed out to the lowest bidder.

amdelamar• 6 years ago

This basically outlines my current job. Huge system with many, constantly moving/upgrading parts and services across teams, all while utilizing dozens of internal services, tools, and navigating corporate policies. I’m thriving in it, but totally recognize it’s a steep learning curve and takes longer to onboard newcomers.

Eventually you get to enjoy deprecating old services as much as building new ones, simply because you never have to teach others about them again.

dzink3• 6 years ago

As a solo tech founder of several sites, I’ve had the pleasure of digging into each of these problems and more over the past few years. Some topics worth adding to the list for consumer sites:

1. Prototype and benchmark each of your stack pieces before you pick a stack. It is far easier to fix architecture mistakes when you don’t have 10000 users expecting overnight customer service. If you are using new technology, find good open source products to see how they’ve structured their projects. Your architecture, designed for speed and experience, will be your key differentiator. Inherent speed at core task, design to user needs, and name choice are the 3 musketeers of a solid growth.

2. Prepare for abusers to attack your system from every direction, especially if you enable users to publish content under your domain. You will see bots looking for Wordpress installations, users trying to fill content with SEO links, users trying every hacking vector known to market. Collect known vectors and test for them and never refuse a legitimate bug bounty request.

3. There is an eternal debate about the trade off between building a quality product at start or opening early to get user feedback before you get too deep into features. There is merit to both choices as a solo founder. The moment you open the gates to users, your ability to make changes comes with very high friction. With time, trying new features becomes a tremendous luxury hidden under bug requests, roadmap, customer service replies, etc. Build your biggest riskiest assumptions first.

4. Testing is hard as a solo founder. Selenium is your friend. If you don’t spend time with them, your users will take that time in multiples after a mistake.

The best way to learn is to come up with a product you really want and build it in your own time. You can test launch in a weekend.

When I started launching consumer sites solo as an engineer, I went from a tight specialization to being unafraid to try any tech if it gives me an advantage in solving a problem. Once you’ve simulated enough problems and dealt with the consequences of your choices personally, you can play that 10 level chess game with the architecture of each new feature much faster.

cactus2093• 6 years ago

This is interesting advice because every company I've come across that has been successful enough to get past the early startup phase has by and large ignored all of this.

The companies I've seen succeed were 100% focused on shipping their product to customers. Not 90% focused on customers and 10% focused on code quality, but 100% focused. They'd rather have to spend 30 engineer-days a few years from now fixing an issue if they get to that point than spend 3 hours getting it right upfront.

As an engineer that goes against every instinct I have, it really seems like spending a couple hours upfront must be a better use of time. It seems like it should be possible to spend 10% of your time setting yourself up well for the future, that's still just a rounding error of your time. And then if you do survive another few years, you'll have a huge leg up on other series B or C stage competitors if you're not hindered by a lot of tech debt at that point.

But from a capitalist perspective, it's probably not so crazy. If you are working with a $200,000 seed round in the beginning, and an engineer costs $80,000 a year, 3 hours of their time costs $115 which is 0.06% of your funds. And more importantly, that $200k is maybe enough for a year of runway, so 3 hours is 0.14% of the time you have to live given a 40-hour a week year (or 0.07% of an 80 hour a week year). Every bit of that starts to add up. Whereas by the time you're a later-stage company and you've raised, say, $40 million dollars and are paying engineers $150k, 30 engineer-days of work is $17,300 but that's only 0.04% of the money you've raised plus your runway is now approaching infinity if you're close to profitable.

I'm still kind of playing devil's advocate here, my instinct really wants to believe that a better balance than what I've seen is possible. One huge missing factor is that people have a strong tendency to ignore spread out costs like the time wasted fixing bugs that pop up later whereas the upfront cost of writing a bunch of tests is more visible. But it has been interesting for me to consider that maybe most founders really are acting pretty reasonably, even though it originally seems pretty careless and lazy to let your startup build up a ton of tech debt early on.

dzink3• 6 years ago

Engineering is a multiplier in this case - so early investments, not necessarily in code beauty, but in picking the right architecture for the job, pays back compounding dividends and reduces funding needed. Whatsapp had a handful of engineers able to handle millions of users. Funding is not promised, its debt. Use every lever you've got and you'll need less of it.

jacobr1• 6 years ago

There are compounding gains from making "the right choice all else equal." This is one of those semi-mythical 10x engineer super-powers. Or it might just be experience. But technical choices need to be made and making slightly better ones will pay off even in the short term. The technical changes should just optimize for rapid change rather than stability or scale.

abhishekjha• 6 years ago

How do you prototype? What do you look for while prototyping lets say, nginx or haproxy? You certainly can't simulate real world traffic on a toy application. So how do you decide which one to pick? I am beginner in this field.

dzink3• 6 years ago

You can simulate traffic with Blitz or Bees With MachineGuns or many other tools. You can use your own prototypes for a few days. You can create fake words and test how your approaches rank against each other on google (and keep tracking that over time in case algorithms change). There is a lot of inherent reward in finding creative ways to break your own stuff and the learning is unparalleled.

semicolonandson2• 6 years ago

Thought I'd post this here since I imagine there's a strong cross-over of interest:

If you'd like a guided video code-tour of how all the pieces of a production web-app fit together, I publish detailed weekly screencasts showcasing the code, systems, and architecture behind Oxbridge Notes, the business that's supported me for the past decade.

So far I've covered:

— software dependency vetting

— data integrity systems (constraints, foreign keys, transactions etc.)

— integration testing systems

— trade-offs in software quality between customer-facing and admin areas

— softer stuff, like designing for SEO (marketing ease is a critical part of any system I design, as an indie-hacker)

https://www.semicolonandsons.com/series/Inside-The-Muse

bradleykingz• 6 years ago

Interesting. Thanks.

sciencewolf• 6 years ago

Hey Jack,

Just a heads up - I tried to sign up for your mailing list but kept seeing "ERROR: Did you forget to type your email?" even after typing the email. I've tried a few different emails and also tried viewing the page in incognito mode. Hope you get this fixed soon, as I'd really love to get more of your content!

igneo676• 6 years ago

I get the same error :/

Also, it'd be great to be able to turn off the mailing list sign up on the videos. I'm already technically subscribed via RSS and it'd be great if the videos play through without intervention

semicolonandson2• 6 years ago

Thanks for the head's up! I checked the logs and it was just a case of an incorrect error message. Fixed now.

iudqnolq• 6 years ago

Your site appears blocked on Vodafone UK's mobile network. I get a connection interrupted error, the same error I get if I try to access a site I know is blocked (such as archive.org). If I use a VPN I can load the page, even if the VPN connection terminates in the UK.

bibabaloo• 6 years ago

Steveslist currently has a very simple and slightly fragile cron setup. We have a single “scheduled jobs server”. We use crontab on this server ... This setup isn’t scaling very well ... We’re considering setting up a new system using a modern tool like Kubernetes.

Do people use Kubernetes for running scheduled jobs like this? It seems like it'd be overkill but in saying that I'm not sure if I know of anything that can be used for running scheduled tasks in a reliable and observable way that's scalable. Maybe Jenkins?

chucky_z• 6 years ago

this is a pretty common use-case for ci/cd tools.

azure devops, jenkins, concourse, drone, gitlab, ...

also a lot of etl tooling has this stuff (airflow comes to mind immediately)

there's also nomad which is easy to run and can schedule a lot of jobs pretty quickly, with decent observability.

there's... a surprising amount of tooling in this exact space!

harpratap• 6 years ago

The biggest Kubernetes cluster (arguably) is basically one huge batch job - https://cloud.google.com/blog/products/containers-kubernetes...

cyberdrunk• 6 years ago

Do people use Kubernetes for running scheduled jobs like this?

We do, mostly because we already have Kubernetes and automation to deploy to it, so why not? The reliablity and monitoring are better than something we'd cook up on our own.

sciurus• 6 years ago

For better or worse, sure. Here's one case study- https://stripe.com/blog/operating-kubernetes

seneca• 6 years ago

Kubernetes include cronjobs as an object type. Meaning, this functionality is built in. If you're already running Kubernetes, it's a very simple way to do schedule jobs without needing new management tech.

boredatworkme• 6 years ago

off topic: I like the simplistic design of this website. Can someone please help me understand if this is hosted on WordPress or something like that?

cricalix• 6 years ago

Looking at the page source, it doesn't appear to be Wordpress; none of the usual cruft that WP scatters through the source (though it could have been processed via a filter).

Looking at the request/response headers in Firefox's web developer tools, it's gone through Cloudflare, but there's also a x-github header, so maybe some custom tooling that's pushed content to github?

tonyedgecombe• 6 years ago

It might be Jekyll according to https://robertheaton.com/2014/07/26/lessons-from-a-surprisin...

iamAy0• 6 years ago

Nowadays, many sites are built using a static site generator. Have a look at Hugo, Jekyll, Gatsby, etc.

Usually you have plenty of themes to choose from. I use Hugo and can tell you there are many themes similar to the one in this website.

swordbeta• 6 years ago

It's jekyll: https://github.com/robert/robert.github.com

moonchild• 6 years ago

They have this diagram:

  +-----------+   +--------------+   +-----------------+
  |Web Browser|   |Smartphone App|   |Client Libraries/|
  +-----+-----+   +------+-------+   |Other API code   |
        |                |           +-------+---------+
        |                v                   |
        |          +-----+------+            |
        +--------->+ Steveslist +<-----------+
                   |  Servers   |
                   +------------+

Why not this?

  +-----------+   +-----------------+   +--------------+
  |Web Browser|-->|Client Libraries/|<--|Smartphone App|
  +-----------+   |Other API code   |   +--------------+
                  +-------+---------+
                          |
                          v
                    +-----+------+
                    | Steveslist |
                    |  Servers   |
                    +------------+

monocasa• 6 years ago

Different ergonomics of each of those environments, at least in the ideal case for each combined with the server API being a clean enough API boundary if I had to guess.

filipn• 6 years ago

Yes, I think the second diagram that you showed is the preferred way of doing things, since using or "dogfooding" the client libraries will ensure they are always tested and correct, plus you'll get immediate feedback from the other developers who work on the apps as well.

amarant• 6 years ago

In this day and age, your client libraries should be generated automatically. And of course used in your apps/webpage to save time.

See openapi/swagger for good examples how client generation works

TuringNYC• 6 years ago

Another benefit would be easier testing - you can easily auto-test the logic and separate the testing of the UI.

bananaface3• 6 years ago

Nicely wrapped-up public APIs with a pleasant presentation & small learning curve require tradeoffs. Why force your own app & website to pay those costs?

JamesSwift• 6 years ago

You dont "force your site to pay these costs", you design your clients as consumers from day one and get consumable external APIs for free.

WrtCdEvrydy• 6 years ago

I think at the beginning you won't have a client API at the beginning.

That stuff is only done at the end unless you're chargin for that at the beginning.

If you're not B2B positioned, you probably won't have a client library for a while.

wenc• 6 years ago

Wouldn't the issue with client libraries be tight-coupling and having to maintain n libraries instead of one generic server API endpoint?

bananaface3• 6 years ago

Yes

Roybot2• 6 years ago

To do this, we need to write database queries that aggregate over the entirety of our data. We don’t want to run these queries against our production SQL database, because they could put an enormous amount of load on it. We don’t want a huge query issued by an internal analyst to be able to bring our production database to a grinding halt

What kind of query would you have to write to bring down a production db? What makes a solution like hive much better - I guess its optimized for this?

sradman2• 6 years ago

What kind of query would you have to write to bring down a production db?

Scans and Sorts, seen in a query plan, are relatively expensive to run in a production row store. OLAP queries (GROUP BY with aggregate functions like COUNT, SUM, and AVG) do large^/full table scans by definition. They take seconds to run while your goal in a Cloud OLTP system is thousands of requests per second. An automatic sort issued per query in an OLTP system is pathological and represents a vector for a DoS attack.

What makes a solution like hive much better - I guess its optimized for this?

Column stores use compressed bitmap indexes that are optimized for scans over a small number of columns. Hive is SQL over Hadoop, and is inherently slow but it does offload the processing from your Production OLTP server. Hive supports the RCFile format which is partially column oriented. The ORC file format is fully column oriented, replaces RCFile format, but requires Presto (or equivalent). Hive is brownfield for existing Hadoop clusters but it has no place in a discussion about greenfield architecture other than discussing historical systems.

If you have a need for GROUP BY style analytics, a true column store like Presto, Impala, or RedShift is a necessity.

^EDIT: based on zbentley's comment

zbentley2• 6 years ago

OLAP queries (GROUP BY with aggregate functions like COUNT, SUM, and AVG) do full table scans by definition

Isn't it only a full table scan if your query isn't otherwise filtered? Those functions have to read every row of "something", but that something might not always be a whole table.

jordic• 6 years ago

We started evaluating clickhouse, after saying that sentry it's using it

rhlsthrm• 6 years ago

Just recently started messing with AWS Amplify. It's crazy how easy it makes basically all this stuff. It really makes it simple to be a solo founder and manage every piece of the stack.

pantulis• 6 years ago

Great post!

rustamm• 6 years ago

Regarding DB backups, it would be useful to add that it is not enough to simply make backups, one should also test that the backups are actually restorable.

intpx2• 6 years ago

real developers never concern themselves with the realities of operations /s

chasd00• 6 years ago

also the restore/recover process has to be laid out step by step and be as simple as possible. No one is doing a restore in a relaxed, easy going state of mind. Restores are done in full panic mode with the team thinking about how they're going to break the news to their families that they've lost their job.

gmanis6• 6 years ago

I have been doing a greater part of the things elaborated in the post as a solo founder/freelance software person. Albeit the scale of things I need to handle are relatively small.

What kind of a job profile should I be looking at if I am in a position where I absolutely need to have one?

I don’t slot particularly well in any one thing I feel, neither a great developer nor a great systems person. And I did do a bunch of recruiting and tech consulting too:

intpx2• 6 years ago

the mythical 'full stack' engineer? or just sell the AL/ML snake oil and get hired to brute force tech debt.

gmanis6• 6 years ago

There’s nothing mythical about it. Just have been doing this for a long time across different parts of the stack and few indie products under the belt. The newer javascript frameworks are a bit tricky to wrap your head around though.

fapjacks2• 6 years ago

Have you ever considered working technical support? As a lifelong generalist programmer, I never imagined I would work in a support role beyond what every programmer at a startup is expected to do, but I was persuaded to cross over by someone I greatly respect. That support team was the most talented team I ever worked with in almost twenty-five years in tech, and working in that role grew many different skills both in and out of tech in ways I never imagined. Hacking broken production environments back online and writing and pushing bugfixes is exhilarating, and it was the single most valuable experience of my career.

gmanis6• 6 years ago

Never considered support role to be honest. Maybe it feels a little inferior after having those degrees I guess but that’s my own bias. I did think of like a product management role at startups but they all expect you to have pedigree or previous work ex in similar role. Catch-22!

ctrl4• 6 years ago

I once read about devs that get bored with long projects and thrill at starting new things, and the other kind that doesn't want to take new projects until the one that she/he is working on is "finished". Maybe you are the first kind and that's why you are enjoying the unexpected situations you face giving technical support.

tda• 6 years ago

Perhaps a job at a medium size non-tech company. There are plenty of problems that can be solved with small scale (web) apps everywhere, that don't need webscale levels of scalability, but are well suited for a single full stack developer

gmanis6• 6 years ago

I would love to I suppose. Sort of like a mini-cto/dev of a small non-tech company would be ideal. Tough to find such opportunities. Any ideas where?

cvhashim• 6 years ago

Maybe you’d enjoy software consulting work under a firm with good engineering management. You get to touch many different tools and tech depending on project.

wolco• 6 years ago

Easiest path for you:

Frontend/full stack agency developer (node/react).

banq• 6 years ago

no DDD?

avipars• 6 years ago

I love the phrase "Advanced Beginner"

forgotmypwbctbi• 6 years ago

anyone else just getting a blank page here?

withinboredom• 6 years ago

We’re considering either splitting up our cron jobs into multiple servers, or setting up a new system using a modern tool like Kubernetes.

:facepalm: beanstalkd can handle a pretty massive amount of jobs before you need to start worrying about scale. I love the huge jump from simple to complex.

zbentley2• 6 years ago

To be fair they offered a simple option (splitting crontabs) as well.

rataata_jr• 6 years ago

This is amazing. Thank you Robert.

zeckalpha• 6 years ago

Don’t forget a system for organizing contributions to all this code, a system to ensure compliance with the many legal regimes “Steveslist” is offered in, a system for onboarding new hires, a system for expanding market share, a system for keeping money and reporting on it to wouldbe investors, and a system of systems designers that know how to respond to changing and ever expanding requirements.

Don’t sell yourself short: there are systems everywhere you look, not just where there’s convenient precedent to make a web service.

brianzelip• 6 years ago

Really like the prose/Adventure-like writing style of this "tutorial"!

giggl• 6 years ago

Great post! Thanks for sharing.

iblaine2• 6 years ago

Systems Design is a formidable topic because it can go in so many directions. This is a good guide. Might I also recommend something that includes caching, partitioning, indexing, and NoSQL vs SQL. More detail can be found here https://www.educative.io/courses/grokking-the-system-design-...

humanlion87• 6 years ago

I remember using the course you have linked to some time ago - it was very useful. I noticed now that the course is sold on a subscription model. I don't remember being that the case. Would you happen to know when that changed?

iblaine2• 6 years ago

I don't. I came across this in 2018 and recall the course being for sale for $80.

tinalumfoil• 6 years ago

database engines that are quick at small queries are typically unacceptably slow of answering giant queries

Even with proper indexing? I haven't seen this issue with Postgres but maybe I wasn't working on large enough data sets.

zealsham• 6 years ago

This is the best thing I have read in 2020. As a bug bounty Hunter this is insanely helpful to me .

Systems Design for Advanced Beginners

Related Stories