compumike

I don’t see any other impact-narrowing criteria.

The default kubelet `syncFrequency` is 1 minute (per Pod). (There can be additional event-driven ones, but this is a floor.) Back of the envelope: 30 Pods * 1440 * 7 = 300k calls to startPodSync in a week.

I’d guess a lot of production clusters don’t run the latest release of Kubernetes, and 1.36 was only released in late April, less than two months before I reported the issue.

Given that the issue shows up as linear memory growth on the order of maybe 1 GiB/month per node, there really is a time-based discovery process here. (Could be artificially accelerated for a CI test, I imagine!) And any restarted nodes or clusters that upgraded point releases would restart that clock.

The only unusual contributing cause that made the issue more visible to me was trying to run everything on a tiny 2 GiB RAM node!

It does seem like a serious tripping point.

I took a quick look at "containedctx" and it seems like for this case, it would almost be backwards: it would flag the (not-memory-leaking) struct-stored "status.ctx", but wouldn't flag when there is a stored "status.cancelFn" only (which resulted in the memory leak).

Could Go implement a runtime leaked-context detector, like the data race detector? https://go.dev/doc/articles/race_detector

What kind of health checks? In my case, the kubelet process was staying alive and responsive to queries, I believe due to:

  # cat /proc/$(pgrep kubelet)/oom_score_adj
  -999
  
  (from OOMScoreAdjust=-999 in /etc/systemd/system/kubelet.service)  

Thanks. I know there's a `go vet` tool that's run as part of Kubernetes CI, and one of its checks is:

  lostcancel: check cancel func returned by context.WithCancel is called

Author here! If you're running a Kubernetes cluster, I recommend you check `kubectl version` and see if you're running "Server Version: v1.36.[0,1,2]". If so, you may want to use the one-liner at the end of the article to check your "process_resident_memory_bytes" on each node, and consider restarting kubelet as a temporary workaround to tame the memory leak until v1.36.3 is released.

Here are some N-year rolling total inflation charts to put this datapoint in a longer-term perspective: https://totalrealreturns.com/inflation . Zooming out always smooths the noise.

Redis (and memcache) are memory caches and should be treated like that

If you haven't come across Kvrocks yet, it may be worth a look: https://github.com/apache/kvrocks https://kvrocks.apache.org/ . It's a database with a Redis-compatible wire protocol, but the database is stored on disk. This means your working set is not limited by RAM and can be a few orders of magnitude larger! On modern SSDs this is still very fast. I think it improves the durability story as well. But the big win is the orders of magnitude larger database space.

As I've been improving my side project https://totalrealreturns.com/ recently I've ended up using both Redis and Kvrocks together. Redis is great for small global state that needs to be super fast. Kvrocks is great for larger bulk data storage (large precomputed datasets), but also supports a lot of the Redis data structures as well as Lua scripts.

Re: "page for all 500s": there's a world of difference between "page me with a critical alert at 3am" and "notify me on Monday morning when my normal workday starts". At the extremes:

If my DB health check endpoint is returning 500s for N consecutive checks over M minutes, yeah, please wake me up at 3am!

If one user hit a weird edge case in form validation and got a one-off 500, please don't! We can fix that on Monday.

Not always easy to distinguish those clearly or configure those business hours rules, but for my team at https://heyoncall.com/ that is the goal -- otherwise your team burns out fast. Waking up someone at 3am has a real cost, so you better be sure it's worth it.

You can cheaply and readably give a lot of clues to both agents and humans with some assertions at the start of a method:

  raise ArgumentError.new("...") unless ...

I've also had good results using agents to write Crystal https://crystal-lang.org/ which is Ruby-like but does have the static types and produces blazing fast static binaries. Might be a sweet spot for coding agents if you're building some backend services. But I'd still pick Ruby on Rails for a new full stack project.

Check out this map: https://api.phillymesh.net/map for live data from the Philadelphia area.

The edges drawn are between nodes that have been able to hear each other in the last 24 hours, based on observed traceroute packets.

(Even then, it’s only a subset of the actually-connected nodes: the map only shows nodes that have published their position on the public channel, and have set a flag that their data is okay to uplink to a server over MQTT.)

It can be dangerous to backfeed (which is why you're supposed to have an interlock for a generator inlet, ensuring utility power is disconnected). But:

1. These grid tie inverters are designed and tested to shut off completely if there's no grid power. (This is a big design tradeoff: it means they don't provide any power during a grid power outage, even if it's very sunny out.)

2. Even if I had a beefy generator that was unsafely backfeeding my house while the utility power was still connected, the generator would be trying to power not just my house, but all my neighbors too! And the circuit breaker and/or inverter on the generator would likely trip and shut down almost instantly.

There's still a possible risk from #2, especially if the downed wire being repaired is relatively local (i.e. your house only).

But I think #1 and #2 mitigate this risk very well.

Just wait until you see "kW/h" :)

But I think plug-in / balcony solar will be pretty cool. And I think there's a path to inexpensive, larger, safer grid-tie inverters which never backfeed, but prioritize solar input first and make up the difference with grid power.

For example, I'm imagining a box that would plug in to the wall, have a DC input from solar panels, and a power strip for loads supporting up to, ideally, a full 15A normal US 120V circuit.

Currently this box exists in the form of battery power station units (Bluetti, Ecoflow, Anker etc). But I think there could be a much less expensive form that could exist without the battery.

Schematics also help explain your circuit because the idioms of drawing them communicate intent.

I think this is a key point. One could imagine three layers:

(1) NETLIST: a list of text-only descriptions, like "R1 1 0 1k", or even a sentence description "connect a 1k resistor between node 1 and node 0"

(2) SCHEMATIC: a 2D drawing with canonical symbols, straight lines for wires, and labels/notes

(3) LAYOUT: a 3D (or multi-layer 2D) physical representation for PCB, or in this case breadboarding

All three layers are useful. (Obviously you need layout to make a PCB, and you need a netlist for simulation.)

But for most humans, where we have 2D visual representations baked in, if you're trying to understand or communicate what's going on:

- It's really really hard to keep track of a bunch of text sentences like a netlist and node numbers/names for all but the simplest circuits -- maybe 3-5 elements?

- It's really really hard to follow a 3D layout of PCB tracks that leads to pads, and then having to remember pin orders etc.

- It's easiest to follow a schematic diagram. It's browsable. It contains "idioms", as you say, about signal flow, logical block groupings, etc.: purpose and intent and functionality, in a way that netlists and physical layouts don't.

FYI, for medium-large digital circuits, I don't think this is true: probably just reading VHDL/Verilog, like reading source code, makes more sense. This is closer to the "netlist" level. I think that's because you'd name modules and inputs/outputs in a way similar to how you'd name functions and arguments in software, which doesn't really apply to "Resistor" or "Capacitor" as primitives.

But for a pretty big practical range of mixed-mode and analog things, I'd argue that schematics really are the easiest level for our brains.

(Disclosure: I'm one of the founders of CircuitLab https://www.circuitlab.com/ (YC W13) where we've been building an online circuit simulator & schematic editor for a long time. Although I'm mostly on the simulation engine / netlist side. My cofounder and other teammates have done most of the schematic GUI work.)

IMHO solderless breadboards still have their place for prototyping some slow circuits, ballpark maybe < 1 MHz signals, if you're aware of the extra capacitance and limitations. :)

The big thing that articles like this miss completely is that we are no longer in the brief HTTP/1.0 era (1996) where every request is a new TCP connection (and therefore possibly a new DNS query).

In the HTTP/1.1 (1997) or HTTP/2 era, the TCP connection is made once and then stays open (Connection: Keep-Alive) for multiple requests. This greatly reduces the number of DNS lookups per HTTP request.

If the web server is configured for a sufficiently long Keep-Alive idle period, then this period is far more relevant than a short DNS TTL.

If the server dies or disconnects in the middle of a Keep-Alive, the client/browser will open a new connection, and at this point, a short DNS TTL can make sense.

(I have not investigated how this works with QUIC HTTP/3 over UDP: how often does the client/browser do a DNS lookup? But my suspicion is that it also does a DNS query only on the initial connection and then sends UDP packets to the same resolved IP address for the life of that connection, and so it behaves exactly like the TCP Keep-Alive case.)

It's very local here. I'm in the suburbs of Philadelphia, in one of the highest income counties in the state, two blocks from a major hospital, one block from a suburban downtown. Despite that, I've experienced one or two 4-6 hour long power outages per year the past few years. (Mostly correlated with weather.) One outage in June 2025 was 50 hours long!

Many larger homes in this area have whole-house generators (powered by utility natural gas) with automatic transfer switches. During the 50-hour outage, we "abandoned ship" and stayed with someone who also had an outage, but had a whole-house generator.

Other areas just 5-10 miles away are like what you describe: maybe one outage in the past 10 years.

If something goes wrong, like the pipeline triggering certbot goes wrong, I won't have time to fix this. So I'd be at a two day renewal with a 4 day "debugging" window.

I think a pattern like that is reasonable for a 6-day cert:

- renew every 2 days, and have a "4 day debugging window" - renew every 1 day, and have a "5 day debugging window"

Monitoring options: https://letsencrypt.org/docs/monitoring-options/

This makes me wonder if the scripts I published at https://heyoncall.com/blog/barebone-scripts-to-check-ssl-cer... should have the expiry thresholds defined in units of hours, instead of integer days?

100%, I've run into this too. I wrote some minimal scripts in Bash, Python, Ruby, Node.js (JavaScript), Go, and Powershell to send a request and alert if the expiration is less than 14 days from now: https://heyoncall.com/blog/barebone-scripts-to-check-ssl-cer... because anyone who's operating a TLS-secured website (which is... basically anyone with a website) should have at least that level of automated sanity check. We're talking about ~10 lines of Python!

Thought you might find this story interesting, concerning the Total Real Returns website which originally launched on HN a while ago. https://news.ycombinator.com/item?id=32081943

This is probably the same kind of "one new row per month" assumption that many data pipelines with any sort of primary date/time column make!

Lots of people are speculating that the price spike is AI related. But it might be more mundane:

I'd bet that a good chunk of the apparently sudden demand spike could be last month's Microsoft Windows 10 end-of-support finally happening, pushing companies and individuals to replace many years worth of older laptops and desktops all at once.

There's a tradeoff and the assumption here (which I think is solid) is that there's more benefit from avoiding a supply chain attack by blindly (by default) using a dependency cooldown vs. avoiding a zero-day by blindly (by default) staying on the bleeding edge of new releases.

It's comparing the likelihood of an update introducing a new vulnerability to the likelihood of it fixing a vulnerability.

While the article frames this problem in terms of deliberate, intentional supply chain attacks, I'm sure the majority of bugs and vulnerabilities were never supply chain attacks: they were just ordinary bugs introduced unintentionally in the normal course of software development.

On the unintentional bug/vulnerability side, I think there's a similar argument to be made. Maybe even SemVer can help as a heuristic: a patch version increment is likely safer (less likely to introduce new bugs/regressions/vulnerabilities) than a minor version increment, so a patch version increment could have a shorter cooldown.

If I'm currently running version 2.3.4, and there's a new release 2.4.0, then (unless there's a feature or bugfix I need ASAP), I'm probably better off waiting N days, or until 2.4.1 comes out and fixes the new bugs introduced by 2.4.0!

I appreciate that, thank you! :)

Could always just use a status page that updates itself. For my side project Total Real Returns [1], if you scroll down and look at the page footer, I have a live status/uptime widget [2] (just an <img> tag, no JS) which links to an externally-hosted status page [3]. Obviously not critical for a side project, but kind of neat, and was fun to build. :)

[1] https://totalrealreturns.com/

[2] https://status.heyoncall.com/svg/uptime/zCFGfCmjJN6XBX0pACYY...

[3] https://status.heyoncall.com/o/zCFGfCmjJN6XBX0pACYY

Not sure if it’s a fit for what you’re looking for, but maybe https://ultimateelectronicsbook.com/ (maybe more theoretical than practical).

I’ve heard good things about “Practical Electronics for Inventors” but haven’t gone through it myself.

You can simulate a bunch of these (and edit too) in your browser in CircuitLab:

Diode half-wave rectifier https://www.circuitlab.com/editor/4da864/

Diode full-wave (bridge) rectifier https://www.circuitlab.com/editor/f6ex5x/

Diode turn-off time https://www.circuitlab.com/editor/fwr26m/

LED with resistor biasing https://www.circuitlab.com/editor/z79rqm/

Zener diode voltage reference https://www.circuitlab.com/editor/7f3ndq/

Charge Pump Voltage Doubler https://www.circuitlab.com/editor/24t6h3ypc4e5/

Diode Cascade Voltage Multiplier https://www.circuitlab.com/editor/mh9d8k/

(note: I wrote the simulation engine)

For fun, playing with Meshtastic https://meshtastic.org/ and contributing to the open source firmware and apps. They have something cool but need lots of help. I've patched 3 memory leaks and had a few other PRs merged already.

For work, https://heyoncall.com/ as the best tool for on-call alerting, website monitoring, cron job monitoring, especially for small teams and solo founders.

I guess they both fall under the category of "how do you build reliable systems out of unreliable distributed components" :)

Inflation adjusted chart for the GLD ETF:

https://totalrealreturns.com/s/GLD

(Not quite the same due to the compounding 0.40%/year expense ratio of the ETF, but probably close enough for this conversation.)

Just thinking out loud here: an ACME DNS-01 challenge requires a specific DNS TXT record to be set on _acme-challenge.<YOUR_DOMAIN> as a way of verifying ownership. Currently this is a periodic check every 45 or 90 or 365 days or whatever, which is what everyone's talking about.

Why not encode that TXT record value into the CA-signed certificate metadata? And then at runtime, when a browser requests the page, the browser can verify the TXT record as well, and cache that result for an hour or whatever you like?

Or another set of TXT records for revocation, TXT _acme-challenge-revoked.<YOUR_DOMAIN> etc?

It's not perfect, DNS is not at all secure / relatively easy to spoof for a single client on your LAN, I know that. But realistically, if someone has control of your DNS, they can just issue themselves a legit certificate anyway.

It's strange: SSL certificates (and maybe domain name registrations?) are one of the only "ticking time bomb" elements present in every modern web stack, whether a static site or not. By "ticking time bomb" I mean that there's a hard date N weeks/months from now where your site will definitely stop working, unless some external pile of dependencies work smoothly to extend that date.

Software didn't have that sort of "ticking time bomb" element before, I think?

I think I understand why it's necessary: we have a single, globally shared public namespace of domain names, which we accept will turn over their ownership over the long run, just like real estate changes hands. So we need expiration dates to invalidate "stale" records.

We've already switched over everything to Let's Encrypt. But I don't think anyone should be under the delusion that automation / ACME is failproof:

https://github.com/certbot/certbot/issues?q=is%3Aissue%20ren...

https://github.com/cert-manager/cert-manager/issues?q=is%3Ai...

https://github.com/caddyserver/caddy/issues?q=is%3Aissue%20A...

(These are generally not issues with the software per se, but misconfiguration, third-party DNS API weirdness, IPv6, rate limits, or other weird edge cases.)

Anyway, a gentle reminder that Let's Encrypt suggests monitoring your SSL certificates may be "helpful": https://letsencrypt.org/docs/monitoring-options/ (Full disclosure: I wrote the most recent addition to that list, with the "self-hosted scripts".)

PostgreSQL defaults to: "fsync = ON" and "synchronous_commit = ON".

ClickHouse defaults to: "fsync_after_insert = false".

A more fair comparison might at least do "SET synchronous_commit TO OFF" on the PostgreSQL side. (But for this UPDATE benchmark, I think the results would still largely be the same.)

Oof, you're right, that's rough that it's so soon after they discontinued their email service!

I wrote this blog post a few weeks ago: "Minimal, cron-ready scripts in Bash, Python, Ruby, Node.js (JavaScript), Go, and Powershell to check when your website's SSL certificate expires." https://heiioncall.com/blog/barebone-scripts-to-check-ssl-ce... which may be helpful if you want to roll your own.

(Disclosure: at Heii On-Call we also offer free SSL certificate expiration monitoring, among other things.)