stuhood

Yes, thank you for your hard work! We rebased recently, and we'll likely talk about those improvements as part of our `0.21.x` release.

An interesting exercise would be to compare this with the (confusingly similarly named) `fsst` string compression strategy: https://github.com/cwida/fsst

Thanks for the questions!

    After reading I don’t get how locks held in memory affect WAL shipping.
    WAL reader reads it in a single thread, updates in-memory data structures
    periodically dumping them on disk. Perhaps you want to read one big
    instruction from WAL and apply it to many buffers using multiple threads?

But as far as I know, even with a custom WAL entry implementation, the maximum in one entry would still be ~8k, which might not be sufficient for a multi-block atomic operation. So the data structure needs to support block-at-a-time atomic updates.

    I guess your implementation generates a lot of dead tuples during
    compaction. You clearly fighting PG here. Could a custom storage
    engine be a better option?

LSM compaction does not generate any dead tuples on its own, as what is dead is controlled by what is "dead" in the heap/table due to deletes/updates. Instead, the LSM is cycling blocks into and out of a custom free space map (that we implemented to reduce WAL traffic).

Thanks for reporting this! I'm having trouble finding the link you are referring to though. Would you mind sharing a link to the file/page containing the dead link?

I think that `inverting` also subsumes async functions/values, which is pretty neat!

In the case where asynchrony was actually necessary, it seems like a great alternative to function coloring.

But whether you should actually use it for something like their `sub_min` example is highly dependent on how good the performance of their implementation is. Creating a graph of references rather than making two passes over an array of integers is not clearly faster ... or clearer, for that matter.

They support lockfiles and tools like `audit`, yes. But they do not support having multiple versions of a dependency.

Tools based on loading libraries from a *PATH (Go, Python, JVM) usually do so by grabbing the first one that they encounter that contains the appropriate symbols. That is incompatible with having multiple versions of a package.

On the other hand, Rust and node.js support this -- each in their own way. In Rust, artifact names are transparently suffixed with a hash to prevent collisions. And in node.js, almost all symbol lookups are accomplished with relative filesystem paths.

Yes. AFAIK, it evolved over time across 3+ package managers (`npm`, `yarn`, `pnpm`, etc), but the current state of that ecosystem is similar (including the behavior of dependabot).

When it comes to understanding the risks involved with having this many dependencies, one thing that folks might not understand is that Rust's support for dependency resolution and lock files is fantastic.

Tools like `cargo audit` can tell you statically based on the lockfile which dependencies have security vulnerabilities reported against them (but you have to run it!). And Github's https://github.com/dependabot/ will do that same thing automatically, just based on the existence of the lockfile in your repo (and will also open PRs to bump deps for you).

And as mentioned elsewhere: Cargo's dependency resolver supports providing multiple versions of a dep in different dependency subgraphs, which all but eliminates the "dependency hell" that folks expect from ecosystems like Python or the JVM. Two copies of a dep at different versions? Totally fine.

Every method in Reboot has a type: reader, writer, transaction, or workflow. Our retry semantics are such that any method can always be retried from the top, but for different reasons:

In readers, no state changes are possible. And in writers and transactions, retry/abort is always safe because no state changes occur until the method completes successfully.

In workflows, retry is always safe, and is in fact required due to the primitives we use to implement durable execution (we will publish more docs on this soon!). The workflow retries durably until it eventually completes, one way or another.

That means that a workflow is always the right spot to execute an external side effect: if a reader/writer/transaction want to execute a side effect, they do so by spawning a task, which is only actually spawned if the method completes successfully. And we do "effect validation" (effectively: running your method twice!) to make it very hard to write a side effect in the wrong place.

https://docs.reboot.dev/develop/side_effects has some more details on our approach to side effects, but feel free to follow up in our discord too: https://discord.com/invite/cRbdcS94Nr

----

I have also read that exceptions can complicate control flow, disallowing some optimizations - but if they are transactional, then we can just add their reverse to the supposedly already slow error path, and enjoy our performance boost!

Somewhat...! When you write a transaction method in Reboot, code that fails with an exception cannot have had a side effect on the outside world, and all state changes will vanish if the transaction aborts. So there is never any need to clean something up, unless you are using exceptions to implement control flow.

They are related, for sure. But one of the biggest differences is that operations affecting multiple Reboot states are transactional, unlike Azure's "entity functions".

Because multiple Azure entity functions are not updated transactionally, you are essentially always implementing the saga pattern: you have to worry about cleaning up after yourself in case of failure.

In Reboot, transactional function calls automatically roll back all state changes if they fail, without any extra boilerplate code. Our hypothesis is that that enables a large portion of an application to skip worrying about failure entirely.

Code that has side-effects impacting the outside world can be isolated using our workflow mechanism (effectively durable execution), which can themselves be encapsulated inside of libraries and composed. But we don't think that that is the default mode that developers should be operating in.

We have built something that hits on points 1, 3, 5, and 7 at https://reboot.dev/ ... but in a multi-language framework (supporting Python and TypeScript to start).

The end result is something that looks a lot like distributed, persistent, transactional memory. Rather than explicit interactions with a database, local variable writes to your state are transactionally persisted if a method call succeeds, even across process/machine boundaries. And that benefits point 7, because transactional method calls compose across team/application boundaries.

[1] Loosen Up The Functions [3] Production-Level Releases [5] Value Database [7] A Language To Encourage Modular Monoliths

Not anymore: PEX now supports including an interpreter in the package: https://docs.pex-tool.org/scie.html

And unfortunately, PyOxidizer went into maintenance mode: https://gregoryszorc.com/blog/2024/03/17/my-shifting-open-so...

The fundamental difference between an ECS and a struct/object layout is that an ECS is column-oriented (aka columnar), while a struct/object layout is row-oriented.

Everything else about how you might query these layouts is more superficial... you can provide the same API with either layout, the same way you can in relational database systems (both layouts can be queried with SQL, but with different performance characteristics.)

I’m more familiar with Bazel. Though I don’t “love it”, I spend plenty of time with it every day. Gazelle like rules for java would be nice.

We'd love your feedback on Pants' support for Java. Pants' "dependency inference" eliminates most BUILD file maintenance: https://blog.pantsbuild.org/automatically-unlocking-concurre...

For those familiar with sbt's incremental Scala compiler library Zinc: we believe that this strategy allows for 90%* of the benefits of Zinc with regard to incremental builds, but with the added benefits of caching and parallelism.

(* The last 10% would come from adding early cutoff when public member signatures haven't changed.)

To be clear though, any such consensus _could_ theoretically be modeled using a smart contract between multiple parties, including how ownership was to be revoked (e.g. "party `X` owns thing `z` unless party `Y` says otherwise").

The challenges are in:

1. knowing ahead of time what the parameters and participants of the contract need to be ("I own the house I paid for... unless 'the government' invokes eminent domain")

2. finding the usecases where it's even beneficial to store proof of ownership in a distributed fashion (rather than at, for example, town hall).

The Long Now [10,000 Year Clock](https://longnow.org/clock/) is another interesting exercise in patience and long term thinking, high in the desert.

`git repack` is an operation that is fairly specific to git's default file format: if they were storing objects in any (other) database, it is very unlikely that they would experience blocking repack operations, as that is an area where databases are highly optimized to execute incrementally.

To spawn a sync function from an async context you use https://docs.rs/tokio/1.3.0/tokio/task/fn.spawn_blocking.htm....

Thanks for the kind words!

What's the performance impact of dependency inference on `pants dependencies`?

There are two components to inferring dependencies:

1) Extracting dependencies from the code being built (in Python this means extracting `import` statements). This involves reading source files, but its runtime is proportional to the amount of code being built, rather than to the repository size.

2) Determining which files throughout the repository provide the various packages (due to how Python's module system requires things to be laid out in directories, this is possible with only file existence checks). This part is proportional to repository size, but in many languages it only requires file existence checks.

Since they're proportional to different things, the amount of time spent on each step depends on what is being built. But the output of both steps is very stable, so with the `pantsd` daemon running (by default), you pay a first build cost that is approximately "listing all files in the repository, and parsing the ASTs of only the files you asked to build". After that, edits to files incur a small incremental cost to parse their ASTs to update imports. Because Pants implements "early cutoff"/cleaning (see https://github.com/pantsbuild/pants/blob/master/src/rust/eng...), if you haven't edited the import statements of a file, the dependencies of that file will not be invalidated.

For comparison, users of Bazel sometimes use tools to generate/update the BUILD files when they modify a source file. You could view the BUILD file as a cache layer in that case.

Yea. Since it isn't integrated, it's a slightly error prone arrangement, since you must always use wrapper tools (with their own daemons) to invoke Bazel.

How Pants accomplishes the second step above depends on the particular inference plugin that is in use: it's entirely possible to cache the indexing, but it would provide little benefit for Python due to the index only requiring file existence checks.

Hey folks! As mentioned in the post: file level dependency inference allows for dependency lists that are 30% smaller (improving cache hit rates), even while removing 90% of boilerplate from `BUILD` files.

Happy to answer any questions!

Hey folks! We'd love to answer any questions you might have, here or in Slack (linked from the post.)

In particular, it's worth noting that 2.0.0 is a ground-up overhaul of Pants with a Rust core (but that wouldn't quite fit in the title!)

Great post.

A point that neither this article nor the recent ninja retrospective (http://neugierig.org/software/blog/2020/05/ninja.html) touch on though is that hashing of files is an important step if you'd like to keep caches locally, but especially if you'd like to keep caches remotely.

A key reason that Bazel-alikes (including http://www.pantsbuild.org, which I work on) hash things is for use in cache keys. And many of those systems use file watching facilities (notify, fsevents) to determine whether files need to be rehashed. If done properly (i.e.: no false negatives), this is significantly faster than checking file mtimes, and it allows for "early cutoff", where if after hashing the digest of a file has not actually changed, you can skip rebuilding its dependents.

There isn't, but that's a good idea. At a very high level, the v2 engine is "Skyframe but using async Rust". All filesystem access, snapshotting, UI, and networking is in Rust, and we use the Bazel remote execution API to allow for remote execution. The v2 plugin API is side-effect free Python 3 async functions (called `@rules`) with a statically checked, monomorphized, dependency-injected graph connecting plugins (ie, we know before executing whether the combination of plugins forms a valid graph). Unlike plugins in Bazel, `@rules` are monadic, so they can inspect the outputs of other `@rules` in order to decide what to do.

This file contains a few good examples of `@rule`s that collectively partition python targets to generate `setup.py` files for them automatically: https://github.com/pantsbuild/pants/blob/5e4f123a1dbc47313fe...

Great experience report: thank you! Wanted to point out that the Pants project has been focusing on widening that happy path recently (...by narrowing its focus to Python-only in the short term), and is ramping up to ship a 2.0.

This page covers some of the differences between v1 and v2 of the engine, and particularly its impact on Python: https://pants.readme.io/docs/pants-v1-vs-v2 ... We're using Rust and haven't bootstrapped yet, so we also appreciate Cargo and think that there is a lot to learn there.

We'd love feedback (via any of these channels: https://pants.readme.io/docs/community) on how to make it even better. Thanks!

Can use `impl Trait` I think?

``` fn foo(s: impl AsRef<str>) ```

Hello!

To add a bit of context: the Pants Build project has been managed by a non-profit 501c6 organization for the last few years, and is wholly independent from Twitter. In the last year, Twitter has contributed less than half of the changes to Pants, with the rest contributed by Toolchain Labs and by the wider Pants community.

Would be happy to answer any questions about the project.

(disclosure: active Pants developer, former Twitter employee, current Toolchain Labs employee)

Hello!

To add a bit of context: the Pants Build project has been managed by a non-profit 501c6 organization for the last few years, and is wholly independent from Twitter. In the last year, Twitter has contributed less than half of the changes to Pants, with the rest contributed by Toolchain Labs and by the wider Pants community.

Would be happy to answer any questions about the project.

(disclosure: active Pants developer, former Twitter employee, current Toolchain Labs employee)

Hello!

To add a bit of context: the Pants Build project has been managed by a non-profit 501c6 organization for the last few years, and is wholly independent from Twitter. In the last year, Twitter has contributed less than half of the changes to Pants, with the rest contributed by Toolchain Labs and by the wider Pants community.

Would be happy to answer any questions about the project.

(disclosure: active Pants developer, former Twitter employee, current Toolchain Labs employee)

I think that that makes it fairly similar to Rust's approach then... compiler support for suspension, and cooperative scheduling.