mapme

Who is more impactful, the startup engineer who singlehandedly ships a feature that increases a startup revenue by 25% off a base $5M/yr ($1M extra rev), or a Meta/Google team of 5 engineers who ship a .01% revenue improve off a base of 150B/yr (15M/5 = $3M/engineer).

As an engineer you are thinking about impact as 'scope' or 'features'. Leadership will be thinking marginally on what adding a net new engineer will provide to the business.

“Marginalism is the economic doctrine that we can best understand value by considering the question of how many units of a good or service an individual has, and using that starting point to ask how much an additional – or marginal – unit would be worth in terms of other goods and services.”

What field has interviews without silliness like this. Jobs for lawyers in 1950 when all that mattered was your father and your alma mater? Hard pass.

I would take a interview premised on a know, learnable challenge (even if silly) over one that exclusively relied on what college you went to.

The native tool use is a game changer. When I ask it to debug something it can independently add debug logging to a method, run the tests, collect the output, and code based off that until the tests are fixed.

Living off 20k a year in a Hcol, where these salaries primarily occur, is next to impossible. Rent alone anywhere in SF area, for example San Jose, is 2k at a bare minimum for a 1BR, or 3k for a 2BR split between a roommate at 1.5k each. That means rent is 18k-24k, not to mention other living expenses (car, utilities, food, travel to visit family, etc.)

Is there a OSS license that specifically precludes its use in LLMs or effectively does so?

Twitters data cannot fit on one machine. In 2015 their Hadoop cluster was 30 PB per earlier comments/their blog. How do you fit that on one machine?

They did not spend half their revenue on compute. It’s more like 20-25% for running data enters/staff for DCs. Check their earnings report.

Whats app is not an applicable comparison because messages and videos are stored on the client device. Better to look at Pinterest and snap, which spend a lot on infra as well.

The issue is storage, ads, and ML to name a few. For example, from 2015:

“ Our Hadoop filesystems host over 300PB of data on tens of thousands of servers. We scale HDFS by federating multiple namespaces.”

You can also see their hardware usage broken down by service as put in their blog.

https://blog.twitter.com/engineering/en_us/topics/infrastruc...

https://blog.twitter.com/engineering/en_us/a/2015/hadoop-fil....

Excluding a settlement from 2014 that was paid this year, Twitter would have made 500mil on 5B revenue in 2021. They were also GAAP profitable in 2019.

Additionally, the incentive from wall street is to spend all your money to grow users. Whether that is right or wrong it’s the path they chose which led to users and revenue roughly doubling over 5 years.

https://s22.q4cdn.com/826641620/files/doc_financials/2021/ar...

https://s22.q4cdn.com/826641620/files/doc_financials/2019/Fi...

SRE is a mix of both. The expectation is you are able to write and understand any code the team is responsible for.

Yes that is the normal case. The post was refuting the assertion that one engineer can run these services indefinitely as previously the OP had the help of SWEs oncall and also fixing bugs.

Source?

One SRE, many SWE. Also have fun asking someone to be permanently oncall with one person on the team.

The cache clusters size are also described here for anyone who wants a good technical read over speculation. https://www.usenix.org/system/files/osdi20-yang.pdf

But Tesla and Spacex have notoriously low pay compared to their peers. Eg a Tesla swe makes 40% less than at Google

Holy moly you should relax buster, I was joking!

Nothing, because they don’t understand RecSys which is half of ML at social media companies. Probably could help with prediction for ads but the domains are very different.

Printing code on paper definitely happened. Confirmed by multiple folks. The layoff 75% numbers and layoffs before Nov 1 were false though.

https://twitter.com/leahculver/status/1586145696163373056?s=...

This is false.

Employee salaries is only 1.5B of their 4.5B annual costs.

Edit: per Wapo but you can see the cost structure from 10ks mostly

IME for engineering at these companies is that the social or political side of things is mostly irrelevant for Eng. Some folks want to be involved in internal activism but it’s easy to ignore and basically never comes up in Eng work. Ppl just avoid politics like normies and talk about average stuff like “how’s your kid?” not “let’s go to that protest”. Lots of good smart technical peeps more likely to nerd out on cool tech stuff than be political.

This is assuming that political power comes from influencing all Americans rather than a small fraction who hold true power (politicians, donors, journalists, industry groups, etc.). If the small fraction is all on Twitter then viola the rhetoric is meaningful.

Does graal native perform better than a non-native running with JIT compilation?

I think this is an unhelpful comparison. Simply put, when users exchange messages on WhatsApp it causes minimal traffic to a WhatsApp data enter as it’s mostly p2p. No ads, no persistence.

Snapchat requires semi persistence for stories (viewable for 24hr) and millions of users can view a single story. When a celebrity posts a story and 5mil users view it that requires more engineering effort that p2p what’s app messages. Additionally, serving ads requires a large amount of compute (ad ranking/serving), storage (impressions), offline processing (Hadoop clusters are hard), engineering, salepeople, economists, and auxiliary systems to support advertisers buying ads, etc.

As an example, check out snaps blog post on ML and ads https://eng.snap.com/en-US/machine-learning-snap-ad-ranking

That is very helpful, thank you! Good point on load balancing, and also retries, back pressure, etc. would change it as well.

what are the most common types of queues for large data center applications (eg. Stateless app that handles rpcs, running on N machines). Are they mostly M/M/N? For example, is memcached different than nginx, or Postgres?

Interesting, anecdotally at a faang adjacent company, I do not know of any Seniors who do not have code as their primary output. Even a large majority of staff actively code IME.

Yeah, most of that is backend issues. Part of them dictated by product requirements eg. We require serving 5 year old, <4MB photos with 99th percentile latency of 1 second.

Product and UI has its own complexities, but it seems clear each product is unique and would require different logic.

The front end apps are not all the same. If you built an app that handles 3 users well do you think it would easily handle 3 billion with no tweaks at any layer (app, framework, OS, network, hardware, edge traffic routing, internal dc network fabric, caching layer, storage for binary/ graph/ KV/ relational, cluster scheduler, security )? Another example, ML backed products (eg. Search, any RecSys) requires tuning features based on product specific needs.

You also end up leaving massive efficiencies on the floor by not hiring more specialists as you scale. This is a good read which explains it in detail https://danluu.com/in-house/

I would also suggest reading their published papers on in house systems to understand the challenges that come with operating at a large scale.

https://news.ycombinator.com/item?id=30940595

I’m sure FB could operate with a range of employees 40k-100k(?), but that number scaling with users is far from surprising.

Hmm, I was trying to do the same with KS test on performance data but it seemed extremely sensitive to outliers even when an eyeball test of the two distributions look near exact. Have you ran into any of those issues?

Reminds me a bit of this AWS builders post about doing constant work as variance can lead to instability.

https://aws.amazon.com/builders-library/reliability-and-cons...

Yeah, there’s also a lag sometimes ie. They write the paper 1-2 years after building the system. They’ve published some really interesting papers over time. If you filter by year you can see them all!

https://research.facebook.com/publications/