NightMKoder

When you measure latency, you’re measuring it based on requests. So in some bucket if you had a request take 2s and a request take 10s, you would say the average is 6s. This answers the question “how long should I expect a single request to take”.

But the articles point is that to the people - it’s not the number of requests that matters - it’s how often they are waiting for them. The question is “how much time am I sitting here waiting?” In that case the 10s request is 5x worse than a 2s request - it takes 5x of the “time spent”.

So you can change the weighing to 1 / 2 2s 1 / 2 10s to 1 * 2/12 & 1 * 10/12 - that gives us 17% and 83%. And the average there is 9.04s.

The difference is the question. If you think about it as road segments, let’s say you have a group of road segments with different lengths. You can ask the “average length of segment” - or you can ask “if you pick a random point among all the segments, how long is the segment that I landed in?” You’re picking very differently there - the second is proportional to the length!

Technically IMO the blog is slightly off - you want to use “mean residual life”. Ie if I pick a random TIME how long do I have to wait for my request to finish. But it’s reasonably close.

This is standard statistics terminology - E(X) is https://en.wikipedia.org/wiki/Expected_value . E_a is presumably Alice's perceived expected value. Var(X) is https://en.wikipedia.org/wiki/Variance . The law of large numbers says the arithmetic average of observations becomes E(X) with enough samples.

I'm pretty sure what the author is saying is:

E(X) =:= \sum_t(t * P(X = t)) is the definition

another important note is P(X^2 = t^2) = P(X = t) - because it's the same distribution.

E_a(X) is a bit sloppy, but consider X_a aka Alice's latency "experience" distribution. The argument is:

P(X_a = t) = t * P(X = t) / \sum_u(u * P(X = u)) - i.e. scale the probability up by t but make it sum to 1.

Then

E(X_a) = \sum_t(t * P(X_a = t)) = \sum_t(t * t * P(X = t) / \sum_u(u * P(X = u))

aka

E(X^2) / E(X)

Then (from wikipedia)

Var(X) = E(X^2) - (E(X))^2

And we get

E(X_a) = (Var(X) + (E(X))^2) / E(X) = E(X) + Var(X) / E(X)

This might speak to the craziness of the gstreamer plugin ecosystem - good/bad/ugly might be a fun maintenance mnemonic, but `voaacenc` is actually in `bad` - not `ugly`. Most plugins you'd want to use aren't in `good`. How are you supposed to actually use "well supported plugins" with gstreamer? Is it just to not use gstreamer at all?

I was recently in the market for one of these! I ended up going with https://github.com/dbohdan/recur due to the nice stdout and stdin handling. Though this has stdout/stderr pattern matching for failures which is nice too!

Facebook’s wormhole seems like a better approach here - just tailing the MySQL bin log gets you commit safety for messages without running into this kind of locking behavior.

IMO mermaid is awesome, but for two somewhat indirect reasons:

- There’s an almost wysiwig editor for mermaid at https://www.mermaidchart.com/play . It’s very convenient and appropriately changes the layout as you draw arrows!

- Notion supports inline mermaid charts in code blocks (with preview!) It’s awesome for putting some architecture diagrams in Eng docs.

Agreed - concretely with-redefs forces single threaded test execution. So eg you can’t use the eftest multithreaded mode.

Explicit dynamic bindings are better if you need something like this since those are thread local.

Usually the controversial decision for Clojure code highlighting is rainbow parens. This color scheme is horrific and unreadable (on mobile at least).

I don’t know about tires, but for brakes we already know how to make lower dust brakes - use drum brakes instead of disc brakes. The friction material is enclosed on drum brakes so much less of it just flies away.

So I might be putting words in your mouth, so please correct me if this is wrong. It seems like you don’t actually control the SIGTERM handler code. Otherwise you could just write something like:

  sigterm_handler() {
    make_healthcheck_fail();
    sleep(20);
    stop_web_server();
    exit(0);
  }

Why the additional SUGUSR1 vs just doing those (failing health, sleeping) on SIGTERM?

Yes sorry for not qualifying - that’s right. IMO the liveness check is only rarely useful - but I've not really run any bleeding edge services on kube. I assume it’s more useful if you actually working on dangerous code - locking, threading, etc. I’ve mostly only run web apps.

This is actually a fascinatingly complex problem. Some notes about the article: * The 20s delay before shutdown is called “lame duck mode.” As implemented it’s close to good, but not perfect. * When in lame duck mode you should fail the pod’s health check. That way you don’t rely on the ALB controller to remove your pod. Your pod is still serving other requests, but gracefully asking everyone to forget about it. * Make an effort to close http keep-alive connections. This is more important if you’re running another proxy that won’t listen to the health checks above (eg AWS -> Node -> kube-proxy -> pod). Note that you can only do that when a request comes in - but it’s as simple as a Connection: close header on the response. * On a fun note, the new-ish kubernetes graceful node shutdown feature won’t remove your pod readiness when shutting down.

Talking of trees and caches, back in school I remember learning about splay trees. I’ve never actually seen one used in a production system though, I assume because of the poor concurrency. Has anyone heard of any systems with tree rebalancing based on the workload (ie reads too not just writes)?

Presumably depends on the country and the laws. Keep in mind that Apple considers this a new interesting use case - not a killer feature for AirPods. They wouldn’t risk AirPod sales with a gray interpretation of the law.

Of course you’re right and you really want to avoid getting to GC thrashing. IMO people still miss the old +UseGCOverheadLimit on the new GCs.

That said trying to enforce overhead limits with RSS limits also won’t end well. Java doesn’t make guarantees around max allocated but unused heap space. You need something like this: https://github.com/bazelbuild/bazel/blob/10060cd638027975480... - but I have rarely seen something like that in production.

If your clojure pods are getting OOMKilled, you have a misconfigured JVM. The code (e.g. eval or not) mostly doesn't matter.

If you have an actual memory leak in a JVM app what you want is an exception called java.lang.OutOfMemoryError . This means the heap is full and has no space for new objects even after a GC run.

An OOMKilled means the JVM attempted to allocate memory from the OS but the OS doesn't have any memory available. The kernel then immediately kills the process. The problem is that the JVM at the time thinks that _it should be able to allocate memory_ - i.e. it's not trying to garbage collect old objects - it's just calling malloc for some unrelated reason. It never gets a chance to say "man I should clear up some space cause I'm running out". The JVM doesn't know the cgroup memory limit.

So how do you convince the JVM that it really shouldn't be using that much memory? It's...complicated. The big answer is -Xmx but there's a ton more flags that matter (-Xss, -XX:MaxMetaspaceSize, etc). Folks think that -XX:+UseContainerSupport fixes this whole thing, but it doesn't; there's no magic bullet. See https://ihor-mutel.medium.com/tracking-jvm-memory-issues-on-... for a good discussion.

Would be great if it included sequential sampling as well: https://www.evanmiller.org/ab-testing/sequential.html . Especially given how A/B tests usually get run in product companies, a peek proof method helps quite a bit.

Easier to explain with coin flips. Let’s say we do 100 flips - we know the “most likely” thing to happen is 50 heads and 50 tails. The actual probability of that is C(100, 50) / 2^100 = 0.079.

So about an 8% chance. You’re significantly more likely (ie 92% chance) to see _something else_. And that’s _the most_ likely outcome.

So tldr - it’s not so much that “you never see an all tails sequence in practice” - you’re actually unlikely to see any particular sequence. All the probabilities get astonishingly low very quickly.

I’ve come to a similar conclusion about “self-service BI” myself but my solution is somewhat different. The solution I have is move the layer of abstraction higher: make extremely customizable dashboards but do not expose SQL to business users.

An example of this might be a dashboard with 20 filters and 20 parameters controlling breakdown dimensions and “assumptions used.” So asking “how did Google ads perform in the last month broken down by age group” is about changing 3-4 preset dropdowns. Parameters are also key here - this way you only expose the knobs that you’ve vetted - not arbitrary SQL.

Obviously this is a hard dashboard to build and requires quite a bit of viz expertise (eg experience with looker or tableau or excel) but the result is 70% of questions do become self service. The other 30% - abandon hope. You will need someone to translate business questions into data questions and that’s a human problem.

I have the same issue with my router - it supports 1gbps but that’s a total over all ports. That total includes the WAN port. So the max throughout you can get from one port to the internet is half the max - 500 mbps.

I’m going to go for the esoteric opinion: MariaDB. Specifically to get system versioned tables. Imagine having the full change history of every single row for any odd task that you need without the performance penalty of keeping historical data in the same table. It can be a huge amount of leverage.

If that’s not your interest, I will admit that Postgres array support is far ahead of any of the MySQLs. Most ORMs don’t use it but you can get optimal query prefetching via array subqueries.

I haven’t tried this framework, but just looking at some examples, the web sockets and sse seem a bit out of place. In a Loom world you’d expect these to use channels or queues or something similar in a loop - not callbacks. Callback hell is what virtual threads try to avoid. I’m not sure if loom has any primitives like go’s multi-channel select that might make this workable though.

In either case Loom and frameworks that use it are super exciting! I’m looking forward to removing the 20 different thread pools we need to avoid deadlocks and just using the common one in our Clojure apps.

Lol yes of course I meant 103. Though I’m ready for an http status code that’s like that: “I can’t give you what you want, but here’s something you didn’t even ask for instead.”

They mention this in the doc, but it seems like the future solution to preloading resources will involve HTTP Status 403 [1]. That seems like a great alternative to server push.

Currently it seems like the best option is to use Transfer-Encoding: chunked and make sure you flush out an HTML header that includes the preload directives before you spend any time rendering the web page. This is a pain - especially if something in the rendering fails and you now have to figure out how to deliver...something to the client. It also limits other headers significantly - e.g. cookies.

[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/103

This may not be a surprise to some, but when folks talk about reliability of the control plane, they usually think failure means their web service goes down. That’s not true. If you shot the kubernetes control plane, the individual servers can’t talk to it anymore - so they do nothing. Docker images that were running stay running. They even get restarted if they crash (via restartPolicy). Services that had specific other pods they were referencing continue referencing those pods. In net: everything except for kubectl and other kubenetes internals keeps working.

That said, one piece that isn’t talked about frequently is the network overlay. Kubernetes virtualizes IPs (so each pod gets an IP), which is awesome to work with when it works. But if your overlay network goes down - god help you. DNS failures are the first to show up, but it’s all downhill from there. Most overlays take great care to degrade well, so they’re not tied to the control plane, but I have yet to find one that’s perfect. The overlay is the part of kube that truly isn’t failure tolerant in my experience.

AWS availability zones (so like us-west-2b rather than us-west-2) are not the same between accounts. us-west-2b for you is something different than us-west-2b for everyone else.

Structural editing is basically a hard requirement with Clojure - I don't think you can realistically live without it. Paredit is fine, but it does take 2-3 months for your fingers to adapt. It would be even better if they didn't have to though.

This is very nonambitious, but I’d love to see a Clojure with less LISP. That is - something like JS syntax but with immutable by default types and const only bindings. Macros would be great too, even if they require a different compilation mode. Top it off with great standard library that has _all_ the collection operations you’ll ever need (e.g. partition-all and dedupe in Clojure). nREPL support or the like is a hard requirement.

Spending the last 6 years writing Clojure has been great. That said, the parens don’t add positively to the experience. They feel like semicolons in C - “the compiler is too dumb to understand this so I have to help it.”

My sense that is that any written record (especially publicly posted records), an admittance of guilt can later be used as evidence in court. That is, if the bar needs to defend itself in court later, admitting guilt now would make that a lot harder.

And they're a bunch of lawyers -- they likely know that.