HN user
Hey HN, encouraged by the "reawakening of systems programming meetups" in SF, NY, and others, we (Spiral) have decided to spin up the London Systems Meetup to here stories and deep dives into all things systems/databases.
The inaugural event will feature talks from Mistral and ourselves, with plenty of time for mingling over drinks and pizza.
How push-down compute helps Vortex run faster and decompress less
Internally the design of Vortex is very similar. The file consists of a whole bunch of "messages" (your files), which then have some metadata attached, and the read logic decides which messages it needs when.
I actually found that DataDog's ddtrace [1] and datadog [2] libraries are very good for instrumenting Python, even in the absence of any connection to DataDog (and no affiliation to them on my part).
ddtrace patches popular Python libraries to emit metrics which can be collected by any statsd-compatible endpoint. The trace spans may well be more specific to the Datadog API though.
I’ve been building out a PyPi proxy to try and protect against these use-cases: https://artifiction.io
Explicit allow-lists and policies such as requiring "greater than X downloads per week" go a pretty long way to filtering out malicious packages.
You should check out Pact Coffee if you’re in the UK: https://www.pactcoffee.com