phiresky

I actually first thought this wasn't possible at all because I'm used to zstd which by default uses a 128MB window and I usually set it to the max (2GB window). 32kB is _really_ tiny in comparison. On the other hand though, zstd also compresses in parallel by default and has tools built in to handle these things, so seekable zstd archives are fairly common.

I think you're looking at a different perspective than me. At _build time_ you need to process O(n), yes, and generate O(n) additional data. But I said "The amount of data you need to decompress is a constant". At _read time_, you need to do exactly three steps:

1. Load the file index - this one scales with the number of files unless you do something else smart and get it down to O(log(n)). This gives you an offset into the file. *That same offset/32 is an offset into your gzip index.*

2. take that offset, load 32kB into memory (constant - does not change by number of files, total size, or anything else apart from the actual file you are looking at)

3. decompress a 1MB chunk (or more if necessary)

So yes, it's a constant.

I'm a bit disappointed that this only solves the "find index of file in tar" problem, but not at all the "partially read a tar.gz" file problem. So really you're still reading the whole file into memory, so why not just extract the files properly while you are doing that? Takes the same amount of time (O(n)) and less memory.

The gzip-random-access problem one is a lot more difficult because the gzip has internal state. But in any case, solutions exist! Apparently the internal state is only 32kB, so if you save this at 1MB offsets, you can reduce the amount of data you need to decompress for one file access to a constant. https://github.com/mxmlnkn/ratarmount does this, apparently using https://github.com/pauldmccarthy/indexed_gzip internally. zlib even has an example of this method in its own source tree: https://github.com/gcc-mirror/gcc/blob/master/zlib/examples/...

All depends on the use case of course. Seems like the author here has a pretty specific one - though I still don't see what the advantage of this is vs extracting in JS and adding all files individually to memfs. "Without any copying" doesn't really make sense because the only difference is copying ONE 1MB tar blob into a Uint8Array vs 1000 1kB file blobs

One very valid constraint the author makes is not being able to touch the source file. If you can do that, there's of course a thousand better solutions to all this - like using zip, which compresses each file individually and always has a central index at the end.

There's a bit of an issue with the linked deployment (in my opinion). In the most zoomed out view you should see the first layer of blocks - very big blocks titled "English language", "French language", "German language". See https://phiresky.github.io/isbn-visualization/ maybe. That makes it a bit easier to read.

The point of the visualization is showing different attributes of books in the space of ISBNs. ISBNs correlate with country, publisher, and release date, that's why using it as a space is useful. You can clearly see the history of when blocks were created, which blocks are rarer than others (present in fewer libraries), and (on the AA hosting) which blocks are more present in AA vs not.

In any case though, yes ISBNs as spatial data are clearly not perfect. Do you have any suggestions that would order the 100 million data points better?

Here's my article on how I built it - and also an instance hosted on GitHub pages if the AA domain is blocked for you: https://phiresky.github.io/blog/2025/visualizing-all-books-i...

Happy to answer questions as always :)

Netdata used to be really impressively minimal, performant, and packed with functions. Fully GPL open source. You ran one install command and it started a web-ui at localhost:19999 in a few seconds. The UI loaded instantly and had hundreds of graphs. You could tell the author was a single opinionated person obsessed with the maximum of monitoring with the minimum footprint.

It auto-detects many programs like docker, nginx and postgresql and automatically creates dashboards for them. It also has many dashboards about system internals I didn't even know were great to monitor, so it taught me a lot. For example, seeing a CPU pinned at 100% processing interrupts because of a network interface overload or having time frames with high IOwait during a SQL query clearly meaning there's some larger seq scans happening.

You also needed zero configuration, no login, etc.

Then they added multi-instance monitoring purely client side - the browser remembers other instance domains and links between them - pretty neat and completely uninvasive.

Then they introduced their cloud login, where you can monitor multiple instances remotely/together. They had a `--no-cloud` flag though if you did not want it. But by now they've removed that flag and they say patching out the cloud functionality is bypassing their license [1]. Some functionality is locked behind premium upgrades, and you get prevented from adding more than N metrics or M instances. It's still _possible_ to use netdata without going through their cloud but you have to go through a nag window every time you try to open the local UI. It's clear they don't want you to use it anymore, and I don't really feel comfortable about their default auto-updating local install any more either.

Now it's still impressive and useful, but it's much more an enterprise focusued tool than an "i have this server i want to monitor" tool.

Of course I understand they need to make money, but what used to be trivial to understand (hooks into everything in your system it can and opens a single port to display it) has become a whole huge integrated ecosystem and for me personally it's competing in the space where I'd probably rather spend the time to make a proper Prometheus/Grafana setup instead.

[1] https://github.com/netdata/netdata/discussions/17594#discuss...

A $120M spend on AWS is equivalent to around a $12M spend on Hetzner Dedicated (likely even less, the factor is 10-20x in my experience), so that would be 3% of their revenue from a single customer.

Even then you can just err.downcast_ref::<std::Io::Error>() though to get the underlying IOError, no?

I'm happy to answer any questions! Nice to see this here again :)

This is great! Especially the DB sync part, because that happens before a user interaction, so you actually have to wait for it (the update itself can run in the background).

It always felt like such a waste to me how the DB always downloads tens of megabytes of data when likely only 1kB has changed. I mean I also really appreciate the beauty of how simple it is. But I'd bet even a delta against a monthly baseline file would reduce the data by >90%.

Also, it would be interesting to see how zstd --patch-from compares to the used delta library. That is very fast (as fast as normal zstd) and the code is already there within pacman.

For the recompression issue, there is some hard to find libraries that can do byte-exact reproducible decompression https://github.com/microsoft/preflate-rs but I don't know of any that work for zstd.

There's an extension to ISO8601 that fixes this and is starting to become supported in libraries:

    2019-12-23T12:00:00-02:00[America/Sao_Paulo]

That seems a bit excessive to sandbox a command that really just downloads arbitrary code you are going to execute immediately afterwards anyways?

Also I can recommend pnpm, it has stopped executing lifecycle scripts by default so you can whitelist which ones to run.

I disagree, you can always spend one or two sentences at the top to immediately bring everyone to a good starting point, regardless of how much technical depth the rest of the article has.

For example in this case: "eBPF is a method for user space to add code to the running Linux kernel without compromising security. They have been tied [...]. The GNU toolchain, the historical and still by many preferred system to build Linux currently has no support.

The description of what LWN and Linux is would be in the about page linked in the article.

It costs almost nothing for an expert to skim/skip two sentences while saving loads of time for everyone else.

The article is also completely missing motivation (why do we care whether BPF is supported in the second toolchain?) Which would be helpful for almost everyone, including people who think it is obvious.

Edit: To be clear though, I love LWN. But the articles are very often missing important context that would be easy to add that I suspect would help a large portion of the reader base.

If your cache fits in Redis then it fits in RAM, if your cache fits in RAM then Postgres will serve it from RAM just as well.

Writes will go to RAM as well if you have synchronous=off.

I think relational databases are great, but this is my biggest problem with SQL, even before the ridiculous syntax.

A query like

    select users.*, orders.* from users left join orders on orders.user_id = users.id

    type SingleReturnRow = { users: {id: ..., }[], orders: {user_id: ..., id: ..., }[]}
    type Return = SingleReturnRow[]

I don't think a larger change in the query language would even be needed.

Even better of course would be a return value like

    type SingleReturnRow = { user: User, orders: Order[] }

Of course in PG now you can use

    select users.*, json_agg(orders.*) as orders from users left join orders on orders.user_id = users.id group by users.id

TypeScript does supports all of these - `C implements I` is not necessary but gives compile errors if not fulfilled.

You can use `o satisfies T` wherever you want to ensure that any object/instance o implements T structurally.

To verify a type implements/extends another type from any third-party context (as your third point), you could use `(null! as T1) satisfies T2;`, though usually you'd find a more idiomatic way depending on the context.

Of course it's all type-level - if you are getting untrusted data you'll need a library for verification. And the immutable story in TS (readonly modifier) is not amazing.

If you replace `sh:link` with `sh:clone` instead, it will.

clone: reflink-capable filesystems only. Try to clone both files with the FIDEDUPERANGE ioctl(3p) (or BTRFS_IOC_FILE_EXTENT_SAME on older kernels). This will free up duplicate extents while preserving the metadata of both. Needs at least kernel 4.2.

Wow, amazing how you managed to convert that into a compact shader, including a representation of the visualization!

It is admittedly pretty noisy, which is somewhat intentional because the focus was on high data density. Here's an a bit more minimalistic view (less color, only one text level simultaneously):

https://phiresky.github.io/isbn-visualization/?dataset=all&g...

It could probably be tweaked further to not show some of the texts (the N publishers part), less stuff on hover, etc.

Huh, that text (and barcodes) are very offset from where they should be. Would you mind sharing what OS and browser you are using and if this text weirdness was temporary or all the time?

Did you search by title or ISBN? If you search by title, the search goes through Google Books, which is very incomplete (since I didn't build a search database myself). If you put in an ISBN13 directly, you'll find a lot more books are included (I'd say you can only find 10-30% of books via the Google Books API)

It's a bit misleading I guess the way I added that feature.

Yes! You can go here to have group texts be horizontal, it was just a stylistic choice to better show the layout: https://phiresky.github.io/isbn-visualization/?groupTextVert...

The names of the groups are taken directly from the official allocation XML from the International ISBN Organization. This is what they call it, I assume to distinguish it from "Germany". Maybe "German language publishers" would be appropriate.

Books visible should be fairly complete as far as I know. But the search is pretty limited (and dependent on your location) because that uses the Google Books API. If you put in an ISBN13 directly, that should work more reliably.

That's not entirely accurate since AA has separate databases for books they have as files, and one for books they only know the metadata of. The metadata database comes from various sources and as far as I know is pretty complete.

Black should mostly be sections that have no assigned books

That but also and for me more importantly it does not tell you how to add time - If you add one month of hours to a date in October, it depends on the timezone whether you will end up one hour of local time earlier or not (due to DST), because then +02:00 might be +01:00

Temporal is great. I've been using it for a while in production using a polyfill [1], and it solves all issues I've encountered with the old Date() API (which is a lot).

It clearly takes inspiration from other high-quality time libraries such as chrono in Rust and Joda Time in Java and combines them into a nice API that's pretty comfortable to use.

Yes, it is a bit more complex to handle since it separates time into naive time, instant and zoned time. But by experience, developers only confront complexity when they are forced to, and time _is_ complex.

If you want to do the operation "add one day to this timestamp", you _must_ decide whether that timestamp is local to a specific timezone and which one. Otherwise you'll get a bug twice per year due to DST, or when the user switches time zones, or when you deploy on a server with a different timezone.

It even solves the serialization issue of the difference between a "fixed-offset" timestamp (e.g. 2025-01-01T00:00+02:00) and one in a specific timezone (e.g. Europe/Paris).

[1] https://www.npmjs.com/package/temporal-polyfill

I think one very common situation you are missing in "Things that can go wrong" is repeating events:

If an event repeats every Sunday at 8pm, people expect that to stay at 8pm local time. Which means you need to know the original timezone the event was created in and make sure the interval is counted in local time, respecting DST. A `timestamptz` field is not enough for that, you need the additional `timezone` storage.

Another thing that I think is useful to mention is the new extensions to ISO8601 time formatting that allow you to include the actual time zone specifier, which is now finally being standardized:

https://tc39.es/proposal-temporal/docs/strings.html

    2007-12-03T10:15:30+01:00[Europe/Paris]

Although neither ISO-8601 nor RFC 3339 specifications currently use this syntax, it's on a standards track led by the IETF SEDATE working group which includes ECMAScript Temporal champions as well as other industry participants.

It is very important to use a high quality time library that forces you to think about these things.

I tried using this a bit in an earlier version, when there was an option to have a real SQL client library/ORM, and you declared what part of the schema would exist client side. Is there plans to bring this back?

The part where you could use drizzle client side was really what interested me, I don't want to bother learning another new query language apart from SQL.

The issue with sqlite's synchronous interface is leaving a thread idle while you wait for IO

That's not the only issue. waiting for the result of every read to be able to queue the next read is also an issue, particularily for a VFS that exists on a network (which is a target of theirs, they explicitly mention S3).

I'm not sure if they also are doing work on improving this, but I'm sure that theoretically many reads and writes that SQLite does do not depend on all previous reads and writes, which means you could queue many of them earlier. If your latency to storage is large, this can be a huge performance difference.