HN user

nbm

1,053 karma

Make Facebook more reliable by day, learning game development by night.

South African temporarily in the Bay Area 2011-2015, Seattle 2016-.

Formerly tech lead and first employees at startups Yola (2007-2010) and KnowledgeTree (2004-2006).

Contacting/following me:

fb.me/nbm @nxsy on twitter Email: HN username at nxsy dot org

Posts5
Comments292
View on HN
Eden 4 years ago

One thing to keep in mind for how development at larger tech companies works is that you’re often not building on your own desktop, you’re usually building on a development server that’s on a well-connected (effectively production-quality, if not literally the same) network. You don’t see a ton of drops in those cases, so it works well. Not that there hasn’t been effort to recover from networking issues encountered in this and other build tooling - at scale, someone’s development server is going to have a bad day every day.

You also need much better tools than grep and locate for a monorepo - or any sufficiently large repo probably. Just load the full repo into memory in a few places around the world, and use an API to find the text you’re looking for. If you already have expertise with search services in your company, this is not that challenging a step - and you can get fancy by using something like Tree-sitter to make those searches more advanced than text. Hitting disk (especially for whole directory trees for “grep -r”) is a losing approach in a large repo.

Since you mention a big tech company, what you’ll probably find is that there are people that care about these things around you, and they’re probably doing what they can and could do with support - not necessarily to do work, but also just moral support. (I like to pretend I’m one of those people in the quality space in my area in my big tech company…)

As others mention, be careful trying to change the world (and especially saying bad things about what you see) before you grow your credibility. But ask a few questions in team meetings (or, at something like FB, in workplace groups) about what test or staging or related infrastructure is available as part of ramping up, and that might get people that care to notice you’re a potential ally.

There’s even the (admittedly probably very) small chance that that the team wants to improve here, and doesn’t know how/didn’t have the resources to do better before - either way, you’ll learn more without alienating anyone.

Big tech companies also often offer mobility and culture variety, so keep your eye out for teams that align with what you care about. Learn what they do and how they got started at least - or possibly move there.

(If you’re at FB, feel free to reach out to me - same account name.)

Perhaps the right word is “system” - the combination of software (DNS protocol server software, health checker, BGP agent, and so forth) involved in making the DNS service available (or not, in this case). These could be running on the same computer, or separate ones if you are particularly imaginative.

Unfortunately, “DNS system” means something very different than if you said “load balancing system”, so “server” is simpler.

(Usual disclaimer: work at FB, even on this exact stuff, but not representing it.)

Stories (which is the medium used) about how things went right without effort are boring. Which leaves two stories - things that went right that had a lot of effort, and things that went wrong.

In the case of things going right with a lot of effort, there’s the case where things went right because of that effort. If you write stories about those, you can be accused of being self-aggrandizing - but they do give people practical options to consider in similar situations.

In the case of things going wrong, there’s the case where you tried to correct things, but were not successful. There’s value in exploring what you could do better - but that’s not a story. The story is the people and archetypes and systems and so forth. In that case, you can be accused of seeking to blame others.

The other stories could be where you did something wrong (but which at the time seemed like they were good and necessary), but despite that the outcome was successful, or because of that, the outcome was unsuccessful. If you’ve read enough of the back catalog, these do exist.

I like that these are presented as stories, and that they are representative of situations people will recognize and also find themselves in. If you’ve seen them, it’s validating that others perceive the challenges the same. If you haven’t, you’re forewarned about things that may come up.

Her stories gain a sharper edge if you’ve worked in similar sorts of environments before.

If you aren’t aware of how there are teams trying to “own” turf, and prevent alternatives (even one-offs), and also how the entire company tries to funnel anything that matches a keyword to that team, even when it is the most tentative match, then you aren’t aware of the challenges faced to navigate them.

You see one path (going with the flow), but don’t see what happens when you don’t follow it.

Not going with the flow is definitely valuable - it’s something any good senior person should have in their tool belt. And if you read Rachel’s stories, there are many examples of not going with the flow (and comments in the HN posts about how she’d be more effective if she didn’t go against the flow).

The challenge is that you can’t always go against the flow either. It’s celebrated if you cut through some red tape - but at some point you’ll just get a reputation of being contrary. Even if you don’t, it’s tiring to have to be the one trying to course-correct the world. Either way, you have to choose your battles. And a button on a dashboard probably isn’t worth using your capital on…

There’s a degree to which you can just go out and talk to people and build relationships. You’d be mistaken if you think that developing these relationships (as well as a reputation for solving real problems, which puts you on the right foot with many strong engineers) is an avenue that wasn’t explored.

At FB, most ICs are not hired for a specific team in advance, and instead choose between (for SWE, a long list of) teams during the Bootcamp process. You can chat to future team mates (not just manager or lead) and ask them straight out about that. (Some people also ask for the half-ly survey results for that team as well.)

If you’re being hired for a team specifically (at another company, say), ask for a “follow-up” meeting with future team mates after offer as a condition for accept. You have a lot of power at that point.

Keep in mind that the referenced paper that this page is based on is over a decade old now. Many things have changed since then, as you can imagine. Look for later papers and engineering blog posts for more about what’s changed since then.

In some ways, HTTP 3 is the same HTTP messages, just over QUIC.

But as you get into other features, there are differences. And your clients and servers need to both have fallbacks to HTTP 2, since UDP connectivity might not be available, and fallback is expected.

So, you have to build support for having working push, or not. Or having working priorities, or not. Or having long-lived non-HTTP sockets, or using fallbacks like web sockets or even long polling. There’s even more fun on the horizon, and I’m not looking forward to my colleagues thinking of a fallback strategy for some of those...

The main benefit is multiplexing - being able to use the same connection for multiple transactions at the same time. This can have benefits in finding and keeping the congestion window at its calculated maximum size, reduce connection-related start-up, as well as overcome waiting for a currently-used connection to be free if you have a max connection per server model.

The other potential benefits were priorities and server-initiated push, but both I’d say largely went unused and/or were too much trouble to use. Priorities were redesigned in HTTP 3 - more at https://blog.cloudflare.com/adopting-a-new-approach-to-http-... - and Chrome recently decided push in HTTP 2 wasn’t worth keeping around - https://www.ctrl.blog/entry/http2-push-chromium-deprecation....

HTTP 2’s main problem is head-of-line blocking in TCP - basically, if you lose a packet, you wait until you get that packet and acknowledge a maximum amount of packets thereafter - slowing the connection down. With multiplexing, this means that a bunch of in-flight transactions, as well as potentially future ones, are blocked at the same time. With multiple TCP connections, you don’t have this problem of a dropped packet affecting multiple transactions.

HTTP 3 has many more benefits - basically, all the benefits of multiplexing without the head of line blocking (instead, only that stream is affected), as well as ability to negotiate alternative congestion control algorithms when client TCP stacks don’t support newer ones - or come with bad defaults. And the future is bright for non-HTTP and non-reliable streams as well over QUIC, the transport HTTP 3 is built on.

Even in the proxy case (which is my day job), it's often worth the price of the up-front load polling to have sufficiently recent information to take action on. The most obvious cases are those where you have some (or all) requests being expensive, as opposed to thousands of exclusively tiny requests.

This does depend on having a local load balancing layer (not, say, making a choice between two servers that are each 50+ms away from the load balancer), and also having a high-performance RPC stack (C++ on both sides, and Thrift, in my case).

In the two cases mentioned, there isn't any extra _response_ time.

servers cannot periodically advertise their load measures to clients

This is an asynchronous mechanism - you can use polling, push, or queue, or whatever. Individual responses don't pay the cost.

staple current load reports to query responses, which the client can cache and use for a period of time

This is just a few extra bytes in the response, which likely has negligible cost. One can fairly reasonably say that responses don't pay any cost.

Obviously there are other costs - it just depends on what you're trading off between on whether it is the right decision.

Why did Twitter allow “Hang Mike Pence” to trend last night?

I happened to look at that trend. Nobody that I saw were actually suggesting people do it - they were quoting (or referring to) what was being said by those at the Capitol, and then commenting on it.

I don’t see a double standard for those cases, although they did subsequently remove it the trend. Possibly because some people might have been calling for it in the “latest” tab - not a fun place looking at tweets outside the featured tab...

Your job title/level/rank (“Principal Engineer”, “IC6”, ...) is what you’ve been deemed capable of (usually by doing it). But your role on a particular project is going to depend on what’s needed. Sometimes you’ll be the lead of the whole thing. Or the lead of a piece of it. Or just another contributor.

Some performance management systems are poorly designed to handle scenarios like this, admittedly.

But it’s not actually factually true - it’s her money according to the laws and contracts in place. If it were his money, he’d have it. He doesn’t, so it isn’t “factually” his money.

It’s only “his money” in a non-factual basis where we are implying something akin to “he earned it and she didn’t”.

Besides the advice in the other replies about choosing good charities, it’s also useful to separate giving to charities (organizations that have non-trivial resources and focus dedicated to handling and seeking donations and often figuring out which other orgs to send that money to) vs. direct charitable giving to colleges, food banks, clinics, universities and other non-profits with a local scope (which many of those receiving donations here appear to be).

Certainly not a guarantee it’ll be used effectively, and that sort of outsourcing of deciding the best way to use money is part of the appeal of the larger charities.

In general, the two biggest financial benefits have to do with reduced health (and other) insurance rates, and not paying taxes on all or some of gifts, inheritance, or other wealth transfers between spouses (similar to those between parents and children).

The largest financial penalty in many countries is that spouses will pay more income-related taxes in dual-income families vs. if they were single, due to how progressive taxation bands are set up.

The benefits of government recognition for most people who marry aren’t financial - they’re to do with parental, health, estate, citizenship/immigration-related, and other rights that largely can’t fully be reliably created by contracts.

TCP has an explicit client and server (some caveats here). The userspace API for TCP requires you to do a connect, which does the TCP handshake in the TCP stack, before you can send traffic.

For UDP, you just specify the destination IP and port, and send packets.

For TCP, new incoming SYNs to most (S)NAT addresses will just get dropped - especially CGNAT - making it impossible to communicate in that direction. If you're both in that situation (really common, actually), you just can't talk to each other.

For UDP, the packets will also get dropped on the receiver side. However, the act of sending the packet will often cause the originator's side's NAT to register that five-tuple (source and destination IP and port + UDP proto), which would allow the other side to reply. If both sides do this with the same IP/port pairs, then magically they can communicate - some of the time, at least. There's a lot more involved (for example, how do you know your own external IP address behind NAT?) - read up on STUN - https://en.wikipedia.org/wiki/STUN - for more details.

Pretty much any HTTP proxy supports talking a set of negotiated protocol versions/features with the client, and a potentially different set of protocol versions/features with the server behind it.

The high-level flow is pretty much:

1) Set up client connection/negotiate stuff (TLS, alpn, NPN, blah blah) 2) Process requests from that client connection 2.1) Decode request from client connection 2.2) Manipulate request (add/remove headers, ...) 2.3) Send request to server 2.3.1) If necessary, create a new connection to server (TLS, alpn, NPN, ...) 2.3.2) Encode request to server connection 2.4) Decode response from server connection 2.5) Manipulate response (add/remove headers, ...) 2.6) Encode response to client connection

You can talk totally different protocols from Internet-side client to the proxy, and from the proxy to the server - and multiple layers of proxies in between if you like. From an app point of view, there's essentially no difference. If you want to for some reason, you can use various headers to do attempt to indicate to the client to upgrade/downgrade to particular protocols, but most apps won't care about that.

Yes. The Internet-side client and the proxy server can use different protocols and versions that they can negotiate however they like, receive the HTTP request over those, and the proxy can use a separately-negotiated connection to your server to send that request over.

Essentially, the core of standard HTTP requests and responses (method, path, headers, response code) is fairly unchanged since HTTP/1.0, and what's changed is how those requests/responses are encoded and carried.

The “US Tech Workers Union” doesn’t appear to be a union at all. It appears to be a subsidiary of the “Progressives For Immigration Reform” non-profit aimed fairly exclusively at the H-1b issue, and the only spokesperson I can see for it is the executive director of PFIR.

So, I wouldn’t take the “US Tech Workers” as being representative of tech workers or tech workers unions without further research.

It’s easy to take a statistical summary of two groups, and discover that attribute X has some distribution for one group, and another for the other. But figuring out why requires much more effort.

In order to say something is genetic, you need to rule out confounding factors that apply. For example, twin studies attempt to reduce the effects of upbringing (cultural influences, home stability, access to education and other opportunity). This is effective in things like trying to determine whether there may be a genetic component to homosexuality - for example, there’s a statistically significant likelihood (that doesn’t exist for non-twin siblings) that if one twin is gay, the other one is.

However, you can’t aggregate twin studies of one group (“white people”) and another group (“black people”), since you’re not getting around the underlying confounding factors. So, while such twin studies do find that genetics play a role in academic performance (since if one twin does well, the other is statistically significantly more likely to do well too), it isn’t helpful as a mechanism here.

Anyway, jumping to genetics for things that have clearly more local causes is at best lazy. I don’t doubt there’s some genetic factors out there, but we don’t have an easy way to untangle them from other factors.

But we do have several different studies that show a connection between family income and academic performance (even at the same school), parental education levels and academic performance (ie, generational effects), class sizes (in early education) and academic performance, teacher quality, school funding, and so forth.

Or, perhaps, much less magical non-genetic factors like more likely being brought up with less access to higher-quality education, nutrition, stability, and safety.

Here’s an article from the institution whose graph you reference about why this gap exists - https://www.brookings.edu/articles/unequal-opportunity-race-... - which explains that many of the differences in education available. Black children are more likely to go to schools that have less funding, offer fewer advanced classes, have more students in them, and have lower-qualified teachers. And experiments that involved children going to better schools showed they had better outcomes.

So, yeah, let’s control for things we actually know affects these outcomes first before jumping to genetic factors.

Did you see an advert for it? (Just wondering why you're at the ads library for it - it doesn't appear to have any ads.)

As for reporting it, the "Report a problem" button at the bottom of that page will lead to this, but here is is:

* Go to the page - you can click it on the page's name in the advert, or you can search for it, or you can cheat and go to /2261037334224684 (that identifier from the URL you posted), which will take you to the page.

* Next to "Like", "Follow", "Share", there's an ellipsis thing, and under there, there's "Find support or report page", where you can selects "Scams and fake pages".

There usually are procedures. Doesn’t stop people trying to work around them anywhere I’ve heard of.

And I’ve yet to find any set of procedures that cover all possible cases. So you still need people to apply good judgement when the procedures don’t cover situations, or if the procedures currently would allow something that shouldn’t be allowed. Like: “no, even though we have a way to safely roll new libraries out, we won’t allow you to add this library to all front-ends, you should follow best practice and use ...”

Sure, but we’re so far away from that scenario here. In that ideal world you describe above, the project leads would have already talked through the requirements and processes involved with the environment and/or service owners and/or release engineering groups. Not all changes are of the sort where “if the tests pass, it’s safe to ship it”, and that requires learning what is acceptable and not, and talking to the relevant people when you’re out of scope.

Environments like those described often have continuous push and automated slow rollouts with health checks, so the idea of doing something on a Sunday isn’t that strange at all.

That said, there’s something to be said for not trying to locally optimize. If you push bad stuff on Sunday, you’re messing up a bunch of people’s well-earned rest and recovery time from work. You push bad stuff on Monday, and everyone’s there to help you fix it without the stress of lost family or other commitment time.

The difference is 24 hours, which likely isn’t going to make or break anything. It’s easy to get sucked into believing things like that matter when they don’t.

Considering the described situation is someone asking for help understanding how to achieve something (which indicates they don’t have the familiarity with the processes in place to do these things), and they’re asking the team responsible for that infrastructure for the first time on the day before launch, I don’t think one can claim that it’s been well-tested and peer reviewed.

It takes a few hours to learn to use the tools mentioned in the article (and many similar ones) to a level of competence - part of the appeal is how easy they are to use. It takes a few hours to learn how to get new data into and out of them as well. Both are the sort of thing taught in 45 minutes presentations and workshops during the Bootcamp training at FB, and then most people can just explore the rest themselves over time in normal usage.

What takes months is feeling like you understand the most important tools and aspects of the environment to be as effective in them as those around you, or to match how you felt in an environment where the scope is smaller. It’s not like you’re sitting in classes for months - you’re doing your job (maybe even more effectively than normal due to the availability of these tools), but every once in a while you’re encountering something where you need to ask someone else or explore or watch a video about how to do something, and that might break your momentum.

I don't know if this is as standard across the FAANG (and similarly styled companies) as an IC interview is - some companies expect managers to have been able to be successful as mid-level ICs, and some might not. Recruiters will generally happily describe company's process to you if you're applying, or will tell you if you ask if they reach out to you.

Often, you'll still have a coding (or other technical interview) and a system design interview, and a probably very similar general "people skills/career" type interview that an IC gets. But instead of another one or two technical interviews, you'll have another one or two manager-focused interviews (how to build teams, how to run projects, how to grow individuals, how to navigate a particular scenario).