HN user
trible.space faculties.sh gorbie.dev erlkonig.ai
Neat! Thank you for the in-depth!
So you're also using succinct datastructures with Elias-Fano, you might want to check out this paper https://aidanhogan.com/docs/ring-graph-wco.pdf on WCO joins over succinct indexes.
Cool stuff!
I still always want to know the numbers per edge. Because there'a a reaon why these graph DBs try to hold the stuff in memory, everything else is slow as heck. We have ~30bytes per edge in our succinct indexes, which makes the 1.5gb wiki dataset clock in at ~50gb.
If I understand correctly you shard the graph and then content adress each chunk:
- How do you decide on the sharding? I'd expect finding good connected subsets with nice memory locality to be extremely difficult computationally?
- How do you canonicalise your graph. Which has also been an extremely difficult problem in RDF land for example. Although that one is at least efficiently solvable.
You don't know how much more vibrant the world is knowing that you still make games in it.
Thank you for all of those hours. (And making the tools that help me teach my apprentices.)
movement can be executed more aggressively and quickly
That's exactly the reason why it's usually a bad idea to run a classical robot on a neural controller. If atlas bumps into something you get a small bump and maybe a broken atlas. Your average industrial robotic arm will happily yeet whatever it bumps into across the room.
These robots operate on completely different principles.
One can lift insane weights, has insane torque, and absurd precision, and can do the same movement millions of times with virtually no deviation. You program these with an exact movement plan, just like you would programm a CnC with a tool path. They are basically cnc machines.
The other one is a inacurate, unstable, dynamic system controlled by neural networks and heuristics. It has massive deviation over each run, but that means that the programming must be able to account for it. Which makes it suitable to operate on problems that are messy, unrepeatable and human-shaped.
I know these strengths, I've written Zig fulltime for ~1 year before switching to Rust, and I do miss comptime pretty much every day.
Still in my experience the strengths do not outweigh the weaknesses.
I'd also push back on the narrative that Rust is not a C replacement. For one because that characterization based on surface level syntactic similarities misses the point of WHY you'd want to have a C replacement in the first place. And also because if this whole situation has shown anything it's that if you want to generate the "extern C" boilerplate in Rust, then these days it requires little more than "hey claude/codex please write the imports for this C library" or even "please port this C library to Rust".
Correct me if I'm wrong, but the three largest Zig project (by far, with a huge gap between them and the rest of the pack) are Bun, Ghostty, and TigerBeetle.
A language so niche that it only has 3 major projects is a liability. Now it has 2 major projects, one of which is yours. Even I as a weird language connoisseur would raise an eyebrow at that.
After switching from Zig to Rust, I felt like the language was helping me improve the correctness of my project, to argue that the fuzzing of your project helps improve the correctness of the language feels backwards and adds to my suspicions.
We both know that fuzzing is great, but that wether you fuzz with 1000 cores or 1.000.000 cores, at an exponentially growing state space it doesn't make (that much of a) difference (I know that you guys are not doing naive fuzzing, which is extremely cool, but the shape of the problem is still O of evil shaped). Most things you can find with fuzzing are shallow-ish, and if you want to go deeper you need formal verification (for which a strong type system is a good first approximation and I'm not aware of something like Kani in Zig).
I like TigerBeetle and I still wish you guys all the success in the world, but I can't help and wonder where you could be by now if your language was lifting you up, instead of you having to lift up your language.
Read through the stuff yourself if you don't believe me: https://github.com/ziglang/zig/issues?q=is%3Aissue%20label%3...
I'm not expecting the whole language to be stable, but I expect certain parts of it to be more stable than others. E.g. control flow vs. async.
I'm not saying that they can't work that way, more power to them. But then having the expectation of anybody using it in a professional setting is also unrealistic. You can't have your cake and eat it too, either it's your personal project and you are fine with nobody using it but you, or you evangelise for people to use it, but then you also need to make at least some effort to not break their stuff on a whim, or to accept their change requests when they put in the work as was apparently the case for bun.
Tbh I don't see Zig hit 1.0 with a meaningful user-base, it's probably going to mostly get eaten by Rust or some other language and will continue to exist as a niche thing, kinda like D.
Having one of the flagship/showcase codebases rewritten to Rust in a week feels like a death knell. Either the community or the language is too unworkable if someone that heavily invested into it jumps ship, and I'm afraid it's kinda both.
not from the language they use for the implementation.
Everybody has a plan until their compiler compiles an if-else wrong, and that's the level of instability that Zig had when I left it and Tigerbeetle committed to it.
No amount of testing is going to save you from that and it's a completely avoidable unnecessary self-inflicted problem, as a technical person that makes me seriously doubt every other decision you make on top of that. So I feel like they've already been fighting an uphill battle, where they have to proof that they have a stable system, despite an unconventional early stage foundation, and I feel like that it just got even harder.
I'm pretty sure they'll miss the full developer salary that Oven used to sponsor them, which they no longer do. I'd wager one doesn't do a rewrite like that, if you are in great personal standing with the language foundation.
That same "just don't use it" attitude was what drove me away from Zig btw. I would have been fine in restricting myself to a somewhat stable subset, e.g. if, loop + function calls, but they didn't want to provide any tiered stability guarantees for the language.
Opinionated is great, no local minima is great, but you have to accept that if you don't want to engage with the needs of your (professional) community then what you do is a hobby project. A very cool hobby project beloved by thousands, but a hobby project.
On one hand I kinda feel validated for having jumped ship on Zig 3+ Years ago[1] and moving everything to Rust[2], with the language simply being too unstable and unsafe in my eyes, despite my love for comptime and people arguing that Bun and Tigerbeetle were proof that it wasn't the languages fault.
But I also feel bad for the Zig project to loose one of their flagship projects, because while I find the project ultimately anachronistic, I know what it's like to pour your sweat, heart and soul into something, and having it replaced within a week is a sobering experience even from afar.
A couple years ago this would have been unthinkable because of how slow legacy codebases and rewrites are.
I wonder if Tigerbeetle will also have problems arguing for their solution now that the other project they can point to for customer assurance is gone. And I wonder if they will follow suit eventually simply due to marketing pressure (after having been bitten by the Zig compiler I was surprised that they were putting their super duper high reliability database on top of it at all, but with another big player using it there was at least some peace of mind for their enterprise customers).
On one hand I kinda feel validated for having jumped ship on Zig 3+ Years ago[1] and moving everything to Rust[2], with the language simply being too unstable and unsafe in my eyes, despite my love for comptime and people arguing that Bun and Tigerbeetle were proof that it wasn't the languages fault.
But I also feel bad for the Zig project to loose one of their flagship projects, because while I find the project ultimately anachronistic, I know what it's like to pour your sweat, heart and soul into something, and having it replaced within a week is a sobering experience even from afar.
A couple years ago this would have been unthinkable because of how slow legacy codebases and rewrites are.
I wonder if Tigerbeetle will also have problems arguing for their solution now that the other project they can point to for customer assurance is gone. And I wonder if they will follow suit eventually simply due to marketing pressure (after having been bitten by the Zig compiler I was surprised that they were putting their super duper high reliability database on top of it at all, but with another big player using it there was at least some peace of mind for their enterprise customers).
Very cool approach! We build something super similar, also going for content addressed storage and compare&swap as fundamental primitives.
Also commit dag based, but we also wrote this whole knowledge graph / triple-store CRDT data format on top.[1]
We also have p2p syncing of the history so you can use it to track your local work but also to have your agents coordinate within your team.
We had our agents build their own tools on top of that substrate, that way we're vendor independent, this stuff works everywhere from claude web, to self hosted openclaw, you only need to tell your agent to use the faculties.
Because the substrate takes care of everything, every new faculty you write on top of that inherits all of the same properties.
Fair, it actually started out in JS, moved to Deno, then Zig and ended in Rust.
If I ever find the time I'd like to back port what I have now, up the chain.
It is supposed to be a RDF replacement so it will eventually have to happen, but it's hard work to make everything extremely idiomatically integrated into the host language.
Spot on!
In one word. Simplicity!
TerminusDB is a RDF database with build-in prolog, a heavy focus on succinct data structure indexes, and has a client server model.
Triblespace is purposefully not RDF, because RDF has horrible DX, and it does not have a good reconciliation and distributed consistency story.
It's virtually impossible to get RDF data into a canonical form [https://www.w3.org/TR/rdf-canon/#how-to-read] and trivial stuff like equality of two datasets is NP-Hard.
Triblespace, while also a data exchange standard like RDF, is closer to Datascript or Datomic. It's a Rust library, and great care has been taken to give it extremely nice DX.
In-memory datasets are cheaply clonable, and support efficient set operations. There's macros that integrate fully into the type system to perform data generation and queries.
// The entity! macro returns a rooted fragment; merge its facts into
// a TribleSet via `+=`.
let herbert = ufoid();
let dune = ufoid();
let mut library = TribleSet::new();
library += entity! { &herbert @
literature::firstname: "Frank",
literature::lastname: "Herbert",
};
library += entity! { &dune @
literature::title: "Dune",
literature::author: &herbert,
literature::quote: ws.put(
"I must not fear. Fear is the mind-killer."
),
};
ws.commit(library, "import dune");
// `checkout(..)` returns a Checkout — a TribleSet paired with the
// commits that produced it, usable for incremental delta queries.
let catalog = ws.checkout(..)?;
let title = "Dune";
// Multi-entity join: find quotes by authors of a given title.
// `_?author` is a pattern-local variable that joins without projecting.
for (f, l, quote) in find!(
(first: String, last: String, quote),
pattern!(&catalog, [
{ _?author @
literature::firstname: ?first,
literature::lastname: ?last
},
{ _?book @
literature::title: title,
literature::author: _?author,
literature::quote: ?quote
}
])
) {
let quote: View<str> = ws.get(quote)?;
let quote = quote.as_ref();
println!("'{quote}'\n - from {title} by {f} {l}.");
}
You can store stuff in either S3 or a single local file (for the local file you can union two databases by concatenating them with `cat`).
We also have just recently added sync through Iroh.
The core idea and main difference between RDF is that RDF is text based and weakly typed, we are binary and strongly typed.
We split everything into two basic structures: - the tribles (a pun on binary triple), 64byte units that are split into [16byte entity id | 16byte attribute id | 32byte Value] where the first two are basically high entropy identifiers like UUIDs, and the last is either a Blake3 hash, or an inlined <32byte value, with the type being disambiguated by metadata on the attribute ID (itself represented as more tribles) - blobs, content addressed, arbitrary length
It's pretty easy to see why canonical representations are pretty easy for us, we just take all of the tribles, sort them lexicographically, dedup them, store the resulting array in a blob. Done.
Everythign else is build up from that. Oh and we also have succinct datastructures, but because those are dense but slower, and immutable, we have a custom 256-ary radix trie to do all of the immutable set operations.
The query engine is also custom, we don't have a query planner which gives us 0.5-2.5microseconds of latency for queries depending on the number of joins, with a query engine that is fully extensible via traits in rust.
Working on exactly that! We're local first, but do distributed sync with iroh. Written in rust and fully open source.
Imho having a graph database that is really easy to use and write new cli applications on top of works much better. You don't need strong schema validation so long as you can gracefully ignore what your schema doesn't expect by viewing queries as type/schema declarations.
My dishes have never been cleaner and my laundry has never been more folded than since the advent of agentic AI.
I build a bunch of tools[1] that give the agents means to communicate and coordinate between each other, with me mostly just deciding what goes into the backlog, and checking their summaries whenever they finish a task.
It's surprising to me how well these go together, the transient concept in clojure is essentially a &mut, couple that with a reference counter check and you get fast transient mutations with cheap persistent clone.
All of rusts persistent immutable datastructure libraries like im make use of this for drastically more efficient operations without loss in capability or programming style.
I used the same principle for my rust re-imagination of datascript [1], and the borrow checker together with some merkle dag magic allows for some really interesting optimisations, like set operations with almost no additional overhead.
Which allows me to do stuff like not have insert be the primary way you get data into a database, you simply create database fragments and union them:
let herbert = ufoid();
let dune = ufoid();
let mut library = TribleSet::new();
library += entity! { &herbert @
literature::firstname: "Frank",
literature::lastname: "Herbert",
};
library += entity! { &dune @
literature::title: "Dune",
literature::author: &herbert,
literature::quote: ws.put(
"I must not fear. Fear is the mind-killer."
),
};
ws.commit(library, "import dune");
In Clojure this would have been too expensive because you would have to make a copy of the tries every time, and not be able to reuse the trie nodes due to borrow checking their unique ownership.
The /clear nudge isn't a solution though. Compacting or clearing just means rebuilding context until Claude is actually productive again. The cost comes either way. I get that 1M context windows cost more than the flat per-token price reflects, because attention scales with context length, but the answer to that is honest pricing or not offering it. Not annoying UX nudges. What’s actually indefensible is that Claude is already pushing users to shrink context via, I presume, system prompt. At maybe 25% fill:
“This seems like a good opportunity to wrap it up and continue in a fresh context window.”
“Want to continue in a fresh context window? We got a lot of work done and this next step seems to deserve a fresh start!”
In my experience a wiki can actually drastically reduce the amount of dead context.
I've handed my local agents a bunch of integrated command line tools (kinda like an office suite for LLMs), including a wiki (https://github.com/triblespace/playground/blob/main/facultie... ) and linkage really helps drastically reduce context bloat because they can pull in fragment by fragment incrementally.
As a German who is very happy with their Gambian apprentice, may I kindly ask you to go fuck yourself.
that's what I thought.
I got 4 more github stars and someone dropping into the tiny tiny discord just from mentioning it, why do you think that is?
When was the last time you created something and put it out to the world? Your only big post on here is a lament of your wife not giving you children as if she was some expired carton of milk that owes you (that's something you discuss with your partner if you respect them and not strangers on the internet, and 39 is completely fine to have children as a woman - https://www.youtube.com/watch?v=6YIz9jZPzvo).
Even your critique isn't an act of creation, neither creative nor substantial and doesn't go beyond an egotistical "I don't like it when people post their project and share their experiences when AI is involved" on _social_ media.
Is there even something you're proud of enough to share and present, or is all this bitterness the result of envy for those that have?
“In many ways, the work of a critic is easy. We risk very little, yet enjoy a position over those who offer up their work and their selves to our judgment. We thrive on negative criticism, which is fun to write and to read. But the bitter truth we critics must face is that, in the grand scheme of things, the average piece of junk is probably more meaningful than our criticism designating it so. But there are times when a critic truly risks something, and that is in the discovery and defense of the new. The world is often unkind to new talent, new creations. The new needs friends. Last night, I experienced something new, an extraordinary meal from a singularly unexpected source. To say that both the meal and its maker have challenged my preconceptions about fine cooking is a gross understatement. They have rocked me to my core. In the past, I have made no secret of my disdain for Chef Gusteau's famous motto: "Anyone can cook." But I realize, only now do I truly understand what he meant. Not everyone can become a great artist, but a great artist can come from anywhere. It is difficult to imagine more humble origins than those of the genius now cooking at Gusteau's, who is, in this critic's opinion, nothing less than the finest chef in France. I will be returning to Gusteau's soon, hungry for more.”Hasn't HN been traditionally a place where makers share the experience they had with building things?
Especially when you have someone working on autonomous research agents it doesn't seem that off to lament how much time you can sink into the underlying substrate. In my particular case the work started long before LLMs to make actual research easier, the fact that it can also be used by agents for research is just a happy accident.
But since you seem to take so much offence as per: https://news.ycombinator.com/item?id=47425470 + your dunning kruger remark
then you seem to be somewhat blinded by your aversion to AI assisted engineering, because if https://github.com/triblespace/triblespace-rs is a "shitty vibecoded project", then I don't know what a good project actually looks like to you. That codebase has years of human blood sweat and tears in it, implements novel data-structures, has it's own WCO optimal join-algorithm, cutting edge succinct data-structures that are hand-rolled to supplement the former, new ideas on graph based RDF-like CRDTs, efficient graph canonicalisation, content addressing and metadata management, implements row types in rust, has really polished typed queries that seamlessly integrate into rusts type system, lockless left-right data structures, a single file database format where concatenation is database union, is orders of magnitude faster than similar databases like oxigraph... does it also have to cure cancer and suck you off to meet your bar?
You just seem like a hater.
Yeah you can sink a lot of time into a system like that[0]. I spend the years simplifying the custom graph database underneath it all and only recently started building it into tools that an agent can actually call[2]. But so far all the groundwork has actually paid off, the rooster basically paints itself.
I found a wiki to be a surprisingly powerful tool for an agent to have. And building a bunch of CLI tools that all interconnect on the same knowledge graph substrate has also had a nice compounding effect. (The agent turns themselves are actually stored in the same system, but I haven't gotten around to use that for cool self-referential meta reasoning capabilities.)
1: https://github.com/triblespace/triblespace-rs
2: https://github.com/triblespace/playground/tree/main/facultie...
Great insights and visualisations!
I build a whole database around the idea of using the smallest plausible random identifiers, because that seems to be the only "golden disk" we have for universal communication, except for maybe some convergence property of latent spaces with large enough embodied foundation models.
It's weird that they are really under appreciated in the scientific data management and library science community, and many issues that require large organisations at the moment could just have been better identifiers.
To me the ship of Theseus question is about extrinsic (random / named) identifiers vs. intrinsic (hash / embedding) identifiers.
https://triblespace.github.io/triblespace-rs/deep-dive/ident...
https://triblespace.github.io/triblespace-rs/deep-dive/tribl...
What does that have to do with the situation in the US? The situation in the middle east is completely orthogonal to that, and observing the rise of faschism there says nothing about my stance on the current german foreign policy in regards to the middle east.
If you want to know: In my personal opinion that conflict is fucked beyond repair because a small group of powerful people on both sides benefit from it, while a huge number of deep interpersonal conflicts and histories fuel it, with any moderates getting squashed by their own side. So I wouldn't send weapons, but I'd send humanitarian aid or the blue helmets. That whole region is thoroughly fucked beyond my pay grade.
The number of deportations under obama was definitely higher, but he had only one concentration camp (guantanamo bay), and didn't use that for his own people.
Learn about the tolerance paradoxon, there is no negotiating, nuance and reasoning with fashists.
Your enlightened centrism is nothing but smoke and mirrors. Get educated.