cweld510

We try to spread out sandboxes evenly across the cluster (at least, across the workers which are available to take new sandboxes) to minimize conflict. But in general we don't get close to saturation thresholds so high that conflict becomes a problem, except during massive load tests. I suspect we'd see issues around 90% effective utilization.

We make a probabilistic routing decision based on worker load and attributes of the sandbox request. I compare to a load balancer because it's essentially just forwarding an HTTP request.

I'm a huge scheduling nerd, and the container scheduling system in this post is probably the most impactful system I've worked on. It's quite different than existing solutions, and I personally feel it's at an interesting point in the design space -- very distributed, no strong consistency anywhere, and oriented towards massive scales. Would love to hear feedback and thoughts!

The PM’s job in this situation is to articulate some sort of high level strategy generalizable across customers, based on information gathered from customers, which includes feature requests but is more than just that.

The skillset is different. SAs everywhere I’ve worked are solidly salespeople. They can do AE work in a pinch, they use Salesforce, they know how to understand the state of an account, etc. They tend to be more limited technically and as such are generally deployed for the sale of a product which is “finished”, and as such spend most of their time mapping customers onto existing patterns that are already known to work, and identifying those patterns.

FDEs are different in that they’re deployed to sell and integrate products which are unfinished in some way. For example Palantir sold a fairly low-level generalized product which required bespoke technical work to integrate with a customer. This integration work is more like normal engineering work in that lots of code may have to be written and there’s technical creativity required. The reason why FDE is increasingly in vogue now is because AI companies haven’t yet found a high-level product shape which is generalizable across many customers, because the market is immature and customers probably don’t know what they want. So there’s lots of engineering work required to take a lower-level offering (e.g some sort of agent) and figure out how to use it to generate value for one particular customer.

Is the point of math really to prove the statement? Arguably interesting, unproven statements are interesting often specifically because they are hard to prove with existing techniques.

It definitely is at the youth level. I don’t think any football or basketball pros could be soccer stars, but absolutely there are kids who are star point guards on their youth basketball team but top out at 5’8”, or football players who never make it past high school but could have been great at soccer.

Sure, but then the interesting question becomes how people decide whether or not an action is in their self-interest.

AI is only eating some of that though. For instance, everyone who does performance work knows that perhaps the most important part of optimization is constructing the right benchmark. This is already the thing that makes intractable problems tractable. That effect is now exacerbated — AI can optimize anything given a benchmark —- but AI isn’t making great progress at constructing the benchmark itself.

I agree with this — there is at least some bifurcation by skillset and capabilities. Lots of engineers are overfitted to working on consumer web apps or SaaS products, but those are no longer an area of focus. You need to be adaptable enough to work on other kinds of systems too. Doing so either requires that you’re really good commercially (can lead development of a product over time, drive revenue, etc) or very good technically at a wide breadth of technologies and problems.

What makes this more extreme is that we’re in a paradigm shift, technically. Systems of the future look different than what’s been built before. Building agentic stuff is very different than web apps. The infrastructure side is also different. Moreover, both are uncertain so there’s no plug-and-play set of skills that would fit into any company in the way you could probably get hired reliably in the 2010s if you can operate Kubernetes, design a database schema, write Node.js APIs, etc.

Context windows are a natural improvement, but new architectures are completely speculative and it’s unclear we can make any sort of predictable progress with new, better architectures. Most progress has been made on essentially the same architecture paradigms, although we did move from dense models to MoE at some point.

You said above that our country “needs the best engineers and doctors.” Are the tests you mention really objective, direct measures of which student is likely to be the best engineer or doctor in the future? What does it even mean to be the best engineer, and how do you test that?

Depends on the team — managing can be quite a bit more scope than being a senior IC, depending on expectations for that role. You have broader ownership of technical outcomes over time, even aside from the extra responsibility for growing a team. Managers have all the responsibility of a senior engineer plus more. In that way manager feels to me like a clear promotion to me. Manager vs staff eng, maybe not though.

I don’t agree with that though, plenty of places practice agile well. Maybe big corporations don’t practice it well, but startups often do agile correctly and understand the philosophy.

I think the primary benefit of LLMs for me is as an entrypoint into an area I know nothing about. For instance, if I’m building a new kind of system which I haven’t built before, then I’m missing lots of information about it — like what are the most common ways to approach this problem, is there academic research I should read, what are the common terms/paradigms/etc. For this kind of thing LLMs are good because they just need to be approximately correct to be useful, and they can also provide links to enough primary sources that you can verify what they say. It’s similar if I’m using a new library I haven’t used before, or something like that. I use LLMs much less for things that I am already an expert in.

What matters is whatever he believes the likelihood to be, not what it actually is.

Big companies hired a lot, but I don’t think this specifically is true? In theory a high-value engineer would be productive, or else they aren’t worth stealing.

The simpler explanation seems more correct here — there was a lot of product fluff and a lot of headcount allocated to build that fluff.

What does it mean for someone’s model of the world to be accurate? My experience with mild depression is that you notice many negative things which are true but then lack perspective about how much they matter. When you feel better you just don’t pay any mind to these negative things.

It’s probably also the case that being physically fit and healthy helps one think more clearly. Carlsen notably spends a lot of time on physical health in addition to prep.

Doesn’t that imply an interface is necessary though, so you can compile (and potentially release) the components separately? I don’t use .net but this sounds quite similar to pulling things into separate crates in Rust or different compilation units in C, which is frequently good practice.

Reading between the lines, my guess is that the standup was the only forum for communication that the team had, and lots of communication was required because people weren’t working on the same things. The only real solution to that is to get people talking outside the standup.

I don’t think this is because of AI. Rather, it seems a continuation of a shift which has already been occurring for my entire career, which is that the tech industry continually sheds people whose skills are entirely practical and are tied to a specific era or regime of technology. For instance, at one point, there were webmasters, but now those jobs don’t exist anymore. Sysadmins have gone through a similar struggle with the advent of the cloud. Once there were sysadmin jobs, and now there are no more. Today it is happening to a certain kind of full-stack engineers specializing in technology of the last 15 years. In the future it will probably happen to YAML engineers who specialize in Kubernetes and GitHub Actions.

Consistently the most durable roles seem to be those which require theoretical understanding of the fundamentals —- UI/UX, systems, algorithms, etc. It’s unfortunate that not everyone gets a chance to learn these things.

You're right -- we do relax the integrality constraint, gaining performance at the expense of some precision, and we're generally able to paper over the difference at scheduling time. We've investigated integer linear programming for some use cases, but for solves to run quickly, we have to constrain the inputs significantly.

Great to see this post here -- really enjoyed writing it! I think it's really cool how an algorithm from an operational research context can play a critical role in a high-availability large-scale cloud service.

Sure, but without an LLM, measuring customer retention might require sending a request over to your data scientist because they know how to make dashboards, then they have to balance it with their other work, so who knows when it gets done. You can do this sort of thing faster with an LLM, and the communication cost will be less. So even if you choose the wrong statistic, you can get it built sooner, and find out sooner that it's wrong, and hopefully course-correct sooner as well.

Rust is more a response to C++ than to C. Both C++ and rust are big and complicated languages that are good for large projects that are performance-sensitive. Both have very strong static typing and can be verbose as a result.

C feels substantially different than Rust. It’s much smaller and less complicated. It’s technically statically typed, but also not in that it doesn’t really have robust non-primitive types. It’s a very flexible language and really good for problems where you really do have to read and write to random memory locations, rearrange registers, use raw function pointers, that sort of thing. Writing C to me feels a lot closer to Python sometimes than to Rust or C++. Writing algorithms can be easier because there is less to get in your way. In this way, there’s still a clear place for C. Projects that are small but need to be clever are maybe easier done in C than Rust. Rust is getting used more for big systems projects like VMs (firecracker), low level backends, and that sort of thing. But if I was going to write an interpreter I’d probably do it in C. Now, I’d do it in Zig.

The useful thing about proofs is that they are written in English (or another language), not formal logic. In general they can be mapped to formal logic, though. This means that people can digest them on an intuitive level. The actual goal of a proof is to create new knowledge (via the proof) which can be distributed across the mathematical community. If proofs exist but are not easily comprehensible, then they don’t accomplish this goal.

It makes sense from a financial perspective for a small company that’s trying to grow. The most precious resource at such a company is engineering time. It’s very limited - you only have X number of engineers, it’s hard to hire good engineers, and they’ll only work so much. Engineers create the thing that makes money and they are in the critical path for nearly every single product experiment that a growing startup needs to perform to figure out its PMF. So nearly all of the small startup’s engineering time should be spent on things that differentiate its product, not stuff like serving frontend code, which is a solved problem. Traditional VC-funded startup economics assumes exponential growth after PMF, which means that engineering work required to reach PMF has potentially compounding returns.

Engineers know this too, which is why they choose Vercel over rolling their own infra. Frontend engineers get paid for building nice UXs that people will pay for, not mechanically implementing routing policies. So they’ll outsource that work to Vercel. It’s similar for backend engineers. What compounds this effect is that, at a small startup, you are on call in perpetuity. So you need to aggressively farm out support work for nonessential systems to service providers or else you risk being overwhelmed.

It’s not a matter of excusing or not excusing it. Incidents like this one happen for a reason, though, and the real solution is almost never “just do better.”

Presumably crowdstrike employs some smart engineers. I think it’s reasonable to assume that those engineers know what CI/CD is, they understand its utility, and they’ve used it in the past, hopefully even at Crowdstrike. Assuming that this is the case, then how does a bug like this make it into production? Why aren’t they doing the things that would have prevented this? If they cut corners, why? It’s not useful or productive to throw around accusations or demands for specific improvements without answering questions like these.

I work on an event-based architecture that I think is successful, but that’s because our core primitives are event-based, so there is no impedance mismatch in the way that there can be if you migrate from a request-response architecture to an evented one. Specifically, we aren’t trying to deal with databases and HTTP (both of which are largely synchronous primitives). Instead, I work on a platform for somewhat arbitrary code execution; and the code we are executing depends on our code rather than vice versa. In general, the code we execute on the platform can run for an indeterminate amount of time, and it generally has control and calls back into our code rather than our code calling into it. So our control flow is naturally callback-based rather than request/response; as a result, our system is fundamentally event-based.