tanelpoder

Great article, including the interactive parts that were just simple enough to illustrate the point at high level.

I never got that far, but I once built a little page that just visualized embedding vectors generated from a few hundred cat, dog and plane photos as heatmaps. I used this for demonstrating to database & storage folks what embedding vectors physically are, at low level. The point of the heatmap (of different vectors) was to show that there are visually observable "vertical bands" standing out when plotting out many embeddings of the same types of objects (like different cats with different backgrounds) in a single heatmap.

I then also took a single cat photo, rotated it by one more degree 360 times and created a heatmap of these vectors to illustrate the point of what the embedding models really detect (you have to uncheck the "Normalized" checkbox in the "same cat rotated by 360 degrees" page to see the vertical bands show up).

The web-app is here:

https://tanelpoder.com/catvector/

GitHub: https://github.com/ilum-cloud/duck_lineage

It was put into second-chance pool by moderators. I originally submitted this link a few days ago and today got this (semi?)automated email from HN, an excerpt below:

  The submission "OpenAI Privacy Filter" that you posted to Hacker News (https://news.ycombinator.com/item?id=47870901) looks good, but hasn't had much attention so far. We put it in the second-chance pool, so it will get a random placement on the front page some time in the next day or so.

  This is a way of giving good HN submissions multiple chances at the front page. If you're curious, you can read about it at https://news.ycombinator.com/item?id=26998308 and other links there.

True, I should have been more specific about the context:

Their invention is about pushing down the .eh_frame walking to kernel space, so you don't need to ship large chunks of stack memory to userspace for post-processing. And eBPF code is the executor of that "pushed down" .eh_frame walking.

The GitHub page mentions a patent on this too: https://patents.google.com/patent/US11604718B1/en

The reference implementation of the profiler [1] was originally built by the Optimyze team that Elastic then acquired (and donated to OTEL). That team is very good at what they do. For example, they invented the .eh_frame walking technique to get stack traces from binaries without frame pointers enabled.

Some of the OGs from that team later founded Zymtrace [2] and they're doing the same for profiling what happens inside GPUs now!

[1] https://github.com/open-telemetry/opentelemetry-ebpf-profile...

[2] https://zymtrace.com/article/zero-friction-gpu-profiler/

I was doing over 11M IOPS during that test ;-)

Edit: I wrote about that setup and other Linux/PCIe root complex topology issues I hit back in 2021:

https://news.ycombinator.com/item?id=25956670

Yeah you'd still have the IOMMU DMA translation, but would avoid the interrupt overhead...

I understand that it's the interrupt-based I/O completion workloads that suffered from IOMMU overhead in your tests?

IOMMU may induce some interrupt remapping latency, I'd be interested in seeing:

1) interrupt counts (normalized to IOPS) from /proc/interrupts

2) "hardirqs -d" (bcc-tools) output for IRQ handling latency histograms

3) perf record -g output to see if something inside interrupt handling codepath takes longer (on bare metal you can see inside hardirq handler code too)

Would be interesting to see if with IOMMU each interrupt handling takes longer on CPU (or is the handling time roughly the same, but interrupt delivery takes longer). There may be some interrupt coalescing thing going on as well (don't know exactly what else gets enabled with IOMMU).

Since interrupts are raised "randomly", independently from whatever your app/kernel code is running on CPUs, it's a bit harder to visualize total interrupt overhead in something like flamegraphs, as the interrupt activity is all over the place in the chart. I used flamegraph search/highlight feature to visually identify how much time the interrupt detours took during stress test execution.

Example here (scroll down a little):

https://tanelpoder.com/posts/linux-hiding-interrupt-cpu-usag...

The issue in the enterprise (SAN) storage space is that there are so many layers where things can go wrong. Lost writes due to the OS kernel (like the fsyncgate), in-kernel storage interaction drivers, the storage array software itself, then disk firmware etc. Theoretically you could read back the just-written block and check if it’s what you wrote, but maybe it’s returned from some cache, before the bug happened.

Another scenario (OS, driver bugs) is that a correct block is written to a wrong location. Yes the write is persisted, but overwriting a wrong location. So now you have two incorrect blocks.

Since we are talking cats here, let me plug my CatBench vector search playground app, intended to learn and demonstrate in a visual way how (PyTorch + ViT) image embeddings and (Postgres) vector search work together with your regular OLTP transactional data. I implemented a toy version of a purchase recommendation system based on “cat customer” image similarity and “cat fraud detection” building block too:

https://github.com/tanelpoder/catbench

Also, a simple browser-only page for visually demonstrating what embeddings/vectors physically are, for explaining this to database & storage folks:

https://tanelpoder.com/catvector/

(Initial load of the vector heatmaps may take a few seconds when you click on a pet type)

GitHub: https://github.com/DataDog/IDE-SHEPHERD-extension

I went with an overkill approach at first (as I often do :-) and bought some expensive nicely designed "active chair" / stool that was adjustable high enough so that I could lean on it even when using my desk as a standing desk. It was interesting, but not a game changer at all for me. I don't use standing desks now at all.

But what I have now is this:

https://www.amazon.com/dp/B002FL3LY4

Just don't assemble the backrest at first. If sitting up straight, I just lean wrists on my keyboard wristpad and part of forearms on the desk, no armrests needed either.

Edit: I still use my height-adjustable standing desk, but now it's value is that I could adjust it for the perfect height for my sitting-up-straight position (so no chair armrests needed) and it's been fixed at that height for the last 7 years...

Cool, thanks for the clarification. Indeed it's a good and practical idea for a small app. As other comments have said, (some) people might happily pay for this app.

I luckily won't need such feedback loop anymore, had some mild lower back pain show up over 10 years ago and bought a chair without a backrest that, after 3-4 weeks of struggling, trained me to sit up straight. Now I have some random cheap office chair with a backrest, but I rarely lean back to it. Funnily, I was going to give up using that "backrestless" chair after 2 weeks of inconvenience, but decided to give it one more week and then the magic happened :-) Mild lower back pain automatically gone.

Once launched, Posturr runs in the background and displays a brief "Claude Mode Active" notification.

I haven’t checked the code yet, but what does the “Claude Mode” mean? Is it a poor naming choice? It implies that the local app is somehow connected to Claude (?)

(2024)

There's also a (free) PDF download from Springer

https://link.springer.com/book/10.1007/978-3-031-99489-0

I also wrote a little Python tool that iterates through syscall tracepoint declarations in debugfs (/sys/kernel/debug) and lists available syscalls and their arguments available in your currently running system:

https://tanelpoder.com/posts/list-linux-system-call-argument...

Debugfs does not show platform-specific syscall internal numbers though (but the stable syscall IDs).

Apparently debugfs does not show all syscalls, excluding "some weird ones" as mentioned by mebeim/systrack author in an earlier HN discussion:

https://news.ycombinator.com/item?id=41018135#41020166

And with eBPF iterators you can bail out early and move to next if you see a non-interesting item (or one that should be filtered out) instead of emitting textual data of all items and later grepping/filtering things out in post-processing.

I use early bailout a lot (in 0x.tools xcapture) when iterating through all threads in a system and determining which ones are “active” or interesting

The internals document looks pretty great too. It actually talks of internals and goes pretty wide and deep. Saved for reading later once the coffee kicks in!

https://tidesdb.com/getting-started/how-does-tidesdb-work/

Noticed this at a Linkedin discussion about Planetscale's new pg_strict extension:

https://planetscale.com/docs/postgres/extensions/pg_strict

I never got to comparing the exec plans between PostgreSQL 16 and 17. The latter ran much faster, if you (or anyone) have time, please test on 16 and 17 (and 18) with explain (timing=on) and put it through some online plan explainer (and write a post about it).

Happy to link to it in my blog.

Author here. I had a vague idea of what would be needed for building such QR codes, but not all stages. As I mentioned in my blog, I worked with codex, so I did this:

1) ask codex to list what are the stages/components needed for QR code generation in general

2) told codex to build a QR code generator based on these stages in SQL. - It went off the rails and started addding "plsql" stuff for some matrix operations, etc. - I added guidance saying that you can just use string_agg(), array_agg() and either nested loop driven or recursive iteration through the space

3) Then it generated QR-code looking codes, but they were not readable

4) I then installed the "qr" utility to my machine and instructed codex to iterate through its fixes and compare the SQL output to "qr" output, that got it working for some QRs

5) I then created a test suite that was just a 5 line shell script that ran 1000 QR code generations with random text inputs both with SQL and "qr" and diffed the outputs (and bailed out immediately once it found the first mismatch)

I then instructed codex to always run the full "test suite" when iterating changes and that got me to the final result - all work took just an hour.

From the post:

"This improvement comes from a redesigned Windows storage stack that no longer treats all storage devices as SCSI devices"

And:

"Direct, multi-queue access to NVMe devices means you can finally reach the true limits of your hardware."

This also leaves more power & thermal allowance for the IO Hub on the CPU chip and I guess the CPU is cheaper too.

If your workload is mostly about DMAing large chunks of data around between devices and you still want to examine the chunk/packet headers (but not touch all payload) on the CPU, this could be a good choice. You should have the full PCIe/DRAM bandwidth if all CCDs are active.

Edit: Worth noting that a DMA between PCIe and RAM still goes through the IO Hub (Uncore on Intel) inside the CPU.

At an old startup attempt we once created a nested hierarchy metrics visualization chart that I later ended up calling Bookshelf Charts, as some of the boxes filled with with smaller boxes looked like a bookshelf (if you tilted your head 90 degrees). Something between FlameGraphs and Treemaps. We also picked “random” colors for aesthetics, but it was interactive enough so you could choose a heat map color for the plotted boxes (where red == bad).

The source code got lost ages ago, but here are some screenshots of bookshelf graphs applied to SQL plan node level execution metrics:

https://tanelpoder.com/posts/sql-plan-flamegraph-loop-row-co...

I figure it’s one way to keep your compiler version unchanged for eBPF work, while you might update/upgrade your dev OS packages over time for other reasons. The title of the linked issue is this:

“Checksum code does not work for LLVM-14, but it works for LLVM-13”

Newer compilers might use new optimizations that the verifier won’t be happy with. I guess the other option would be to find some config option to disable that specific incompatible optimization.

If anyone is interested in reading about a similar ”local-NVMe made redundant & shared over network as block devices” engine, last year I did some testing of Silk’s cloud block storage solution (1.3M x 8kB IOPS and 20 GiB/s throughput when reading the block store from a single GCP VM). They’re using iSCSI with multipathing on the client side instead of a userspace driver:

https://tanelpoder.com/posts/testing-the-silk-platform-in-20...

Interesting. I just posted the link from this X post [1] without modifying it (and it wasn’t 0.0.0.0…). I think one thing HN does is that when it detects a HTTP redirect, it will start using the referred to URL instead…

[1] https://x.com/julianhyde/status/1982637782544900243

I submitted the link that Julian posted (to his article) on X yesterday. There’s indeed some refresh action in my browser too, but the article opens up.

(I submitted this link). My interest in this approach in general is about observability infra at scale - thinking about buffering detailed events, metrics and thread samples at the edge and later only extract things of interest, after early filtering at the edge. I’m a SQL & database nerd, thus this approach looks interesting.