seddonm1

Based on the now-deprecated Clear Linux it does seem that these optimizations add up [0] and so maybe we should be considering them more broadly?

[0] https://www.phoronix.com/review/clear-linux-48p-ubuntu/6

I would be interested to know if there is a method similar to this one in Rust [0] that allows a single binary to support multiple optimization levels depending on the executing CPU? It feels wasteful to not enable these optimizations but I don't really want to force a user to choose between a complex feature matrix.

[0] https://github.com/ronnychevalier/cargo-multivers

I am getting closer and closer to a full verified rewrite in Rust. I have also moved to a much easier sqlite relational structure for the backend.

I actually sidestepped the annoying btrieve problem by exporting the data using a go binary [0] and I write it to a sqlite instance with raw byte arrays (blobs). btreive is weird because it has a dll but also a a service to interact with the files.

P.s. I have spent a lot of hours on this mainly to learn actual LLM capabilities that have improved a huge amount in the last year.

[0] https://github.com/barchart/go-btrieve

It’s a real problem. I threw it at an old MUD game just to see how hard it is [0] then used differential testing and LLMs to rewrite it [1]. Just seems to be time and money.

[0] https://reorchestrate.com/posts/your-binary-is-no-longer-saf...

[1] https://reorchestrate.com/posts/your-binary-is-no-longer-saf...

No, I would need to find a binary to test on. I suspect it would produce horrible code at the decompiler layer but ultimately I would expect that function signatures are still relatively clean?

Its scary - once you get the differential testing harness set up it seems to be just a matter of time/tokens for it to stubbornly work through it.

I have tried to post this here but it has not got traction.

I have a demonstrated process here on my blog (all hand written without AI).

This bit about how to brute force decompilation: https://reorchestrate.com/posts/your-binary-is-no-longer-saf...

And this about how to do the conversion and address the LLM hallucination problem: https://reorchestrate.com/posts/your-binary-is-no-longer-saf...

Yes, it is absolutely possible.

I am applying differential/property based testing to all the side effects of functions (mutations) and return values. The rust code coverage is also used to steer the LLM as it finds discrepancies in side effects.

It is written up in my link - please bear in mind it is really hard to find the right level to communicate this level of detail at - so I'm happy to answer questions.

My test harness loads up the original DLL then executes that in parallel against the converted code (differential testing). That closes the feedback loop the LLM needs to be able to find and fix discrepancies.

I'm also doing this on an old Win32 DLL so the task is probably much easier than a lot of code bases.

I delivered a talk at Rust Sydney about this exact topic last week:

https://reorchestrate.com/posts/your-binary-is-no-longer-saf...

I am able to translate multi-thousand line c functions - and reproduce bug-for-bug implementation

Hi Ben. I published an article about this problem this week (and did a talk at Rust Sydney).

What you need is differential, property testing. I’m sure it would work for you (you can skip the first half as you already have the source):

https://reorchestrate.com/posts/bringing-a-warhammer-to-a-kn...

What about s3 stored in SQLite? https://github.com/seddonm1/s3ite

This was written to store many thousands of images for machine learning

A process for using LLMs to do brute-force decompilation of binaries and conversion to another programming language - including testing to prove equality.

This process is targeting an old computer game but there is nothing preventing this being run on any binary.

SQLX has an offline mode where it saves the metadata of the SQL database structure but then you run into risk of that being out of sync with the database?

Yeah I just drop this one file [0] into my Tokio projects and I have a SQLite with single writer/multi reader pool done in a few seconds.

[0] https://github.com/seddonm1/s3ite/blob/0.5.0/src/database.rs

Whilst I love the idea of SQLX compile-time checked queries it is not always practical to need a database connection to compile the code in my experience. If it works for you then thats great but we had a few tricky edge cases when dealing with migrations etc.

Also, and more fundamentally, your application state is the most valuable thing you have. Do whatever you feel makes you most comfortable to make sure that state (and state transitions) is as well understood as possible. rusqlite is that for me.

I went through the same mental process as you and also use num_cpus [0] but this is based only on intuition that is likely wrong. More benchmarking is needed as my benchmarks show that more parallelism only works to a point.

You can see how the transactions work in this example[1]. I have a connection `.write()` or `.read()` which decides which queue to use. I am in the process [2] of trying to do a PR against rusqlite to set the default transaction behavior as a result of this benchmarking so hopefully `write()` will default to IMMEDIATE and `read()` remains DEFERRED.

[0] https://docs.rs/num_cpus/latest/num_cpus/ [1] https://github.com/seddonm1/s3ite/blob/0.5.0/src/s3.rs#L147 [2] https://github.com/rusqlite/rusqlite/pull/1532

Thanks.

All good and valid questions.

1. I work mostly in Rust so I'll answer there in terms of async. This library [0] uses queues to manage workload. I run a modified version [1] which creates 1 writer and n reader connections to a WAL backed SQLite and dispatch async transactions against them. The n readers will pull work from a shared common queue.

2. Yes there is not much you can do about file IO but SQLite is still a full database engine with caching. You could use this benchmarking tool to help understand where your limits would be (you can do a run against a ramdisk then against your real storage).

3. As per #1, I keep connections open and distribute transactions across them myself. Checkpointing will only be a problem under considerable sustained write load but you should be able to simulate your load and observe the behavior. The WAL2 branch of SQLite is intended to prevent sustained load problems.

[0] https://github.com/programatik29/tokio-rusqlite [1]: https://github.com/seddonm1/s3ite/blob/0.5.0/src/database.rs

In other abuses of SQLite, I wrote a tool [0] that exposes blobs in SQLite via an Amazon S3 API. It doesn't do expiry (but that would be easy enough to add if S3 does it).

We were using it to manage a millions of images for machine learning as many tools support S3 and the ability to add custom metadata to objects is useful (harder with files). It is one SQLite database per bucket but at the bucket level it is transactional.

0: https://github.com/seddonm1/s3ite

I have been following and playing with this repository: https://github.com/singlestore-labs/python-wasi/

It builds a single Python WASM module with all dependencies included (they use VFS) and a Dockerfile to make the process easy (and actually worked first go). It does produce large files though: wasi-python3.11.wasm 110MB

This post demonstrates how to implement plugins for Rust using QuickJS in WebAssembly for safe arbitrary code execution. It relies heavily on on the work done by Shopify with Javy (https://github.com/Shopify/javy) but makes the deployment process easier. I hope it is useful.

This post demonstrates how to implement plugins for Rust using QuickJS in WebAssembly for safe arbitrary code execution. It relies heavily on on the work done by Shopify with Javy (https://github.com/Shopify/javy) but makes the deployment process easier.

I hope it is useful.

The Materialize team manage a fork of https://github.com/sfackler/rust-postgres with the changes required to consume from the Postgres WAL: https://github.com/materializeInc/rust-postgres.

Here is a comment and link to some code which seems to work: https://github.com/sfackler/rust-postgres/issues/116#issueco...

Months ago I posted a link to Arc (https://news.ycombinator.com/item?id=26573930) a declarative method for defining repeatable data pipelines which execute against Apache Spark (https://spark.apache.org/).

Today I would like to present a proof-of-concept implementation of the Arc declarative ETL framework (https://arc.tripl.ai) against Apache Datafusion (https://arrow.apache.org/datafusion/) which is an Ansi SQL (Postgres) execution engine based upon Apache Arrow and built with Rust.

The idea of providing a declarative 'configuration' language for defining data pipelines was planned from the beginning of the Arc project to allow changing execution engines without having to rewrite the base business logic (the part that is valuable to your business). Instead, by defining an abstraction layer, we can change the execution engine and run the same logic with different execution characteristics.

The benefit of DataFusion over Apache Spark is a significant increase in speed and reduction in execution resource requirements. Even through a Docker-for-Mac inefficiency layer the same job completes in ~4 seconds with DataFusion vs ~24 seconds with Apache Spark (including JVM startup time). Without Docker-for-Mac layer end-to-end execution times of 0.5 second for the same example job (TPC-H) is possible. * the aim is not to start a benchmarking flamewar but to provide some indicative data *.

The purpose of this post is to gather feedback from the community whether you would use a tool like this, what features would be required for you to use it (MVP) or whether you would be interested in contributing to the project. I would also like to highlight the excellent work being done by the DataFusion/Arrow (and Apache) community for providing such amazing tools to us all as open source projects.

Edit: format

Previously I posted a link to Arc (https://news.ycombinator.com/item?id=26573930) a declarative method for defining repeatable data pipelines which execute against Apache Spark (https://spark.apache.org/).

Today I would like to present a proof-of-concept implementation of the Arc declarative ETL framework (https://arc.tripl.ai) against Apache Datafusion (https://arrow.apache.org/datafusion/) which is an Ansi SQL (Postgres) execution engine based upon Apache Arrow and built with Rust.

The idea of providing a declarative 'configuration' language for defining data pipelines was planned from the beginning of the Arc project to allow changing execution engines without having to rewrite the base business logic (the part that is valuable to your business). Instead, by defining an abstraction layer, we can change the execution engine and run the same logic with different execution characteristics.

The benefit of DataFusion over Apache Spark is a significant increase in speed and reduction in execution resource requirements. Even through a Docker-for-Mac inefficiency layer the same job completes in ~4 seconds with DataFusion vs ~24 seconds with Apache Spark (including JVM startup time). Without Docker-for-Mac layer end-to-end execution times of 0.5 second for the same example job (TPC-H) is possible. * the aim is not to start a benchmarking flamewar but to provide some indicative data *.

The purpose of this post is to gather feedback from the community whether you would use a tool like this, what features would be required for you to use it (MVP) or whether you would be interested in contributing to the project. I would also like to highlight the excellent work being done by the DataFusion/Arrow (and Apache) community for providing such amazing tools to us all as open source projects.

Edit: fix links

Hi eduren. Give me a few days and Ill see what i can publish as a WIP repo. The aim of Arc was to always allow swapping the execution engine whilst retaining the logic - hence SQL -so this should hopefully be easy.

Disclosure: I am a contributor to Datafusion.

I have done a lot of work in the ETL space in Apache Spark to build Arc (https://arc.tripl.ai/) and have ported a lot of the basic functionality of Arc to Datafusion as a proof-of-concept. The appeal to me of the Apache Spark and Datafusion engines is the ability to a) seperate compute and storage b) express transformation logic in SQL.

Performance: From those early experiments Datafusion would frequently finish processing an entire job _before_ the SparkContext could be started - even on a local Spark instance. Obviously this is at smaller data sizes but in my experience a lot of ETL is about repeatable processes not necessarily huge datasets.

Compatibility: Those experiments were done a few months ago and the SQL compatibility of the Datafusion engine has improved extremely rapidly (WINDOW functions were recently added). There is still some missing SQL functionality (for example to run all the TPC-H queries https://github.com/apache/arrow-datafusion/tree/master/bench...) but it is moving quickly.

Yes. We usually use a ConfigExecute (https://arc.tripl.ai/execute/#configexecute) stage to dynamically calculate a runtime parameter and pass that into the JDBCExtract query for example. There is an example here: https://arc.tripl.ai/solutions/#delta-processing

The better statement would be that this facilitates the development of idempotent jobs and aims to minimise side-effects.

Fair enough. This really replaces the Notebook experience of Databricks and gives you the option to self host and as it is dockerised it is easy to deploy on Kubernetes (with JupyterHub) or other container hosts. If you want to do low-level RDD operations you can easily build a plugin.

I think previously Spark was difficult to run, now with Docker and supported Kubernetes it is fairly easy.

Edit: I forgot to link to the Deploy repo which does have some scripts to easily deploy to cloud environments (but needs some work): https://github.com/tripl-ai/deploy

Yes. Most of the simple stages just invoke the Spark Scala API - for example MLTransform invokes a pretrained SparkML model against a dataframe and returns a new one. You can see the standard Spark ML call: https://github.com/tripl-ai/arc/blob/master/src/main/scala/a.... You can add any plugin you want via the interface: https://arc.tripl.ai/plugins/

This is really defining a dialect that is more simple for Technical Business Analysts to consume that is safer than code and a notebook environment to interactively build with.

Looks good. It is nice to see how much influence the 'Engineers Shouldn't Write ETL' post had!

With Apache Arrow (https://arrow.apache.org/) I think the future looks very bright for both of our projects. It is important to have standard open source libraries and my early experiments have shown very good performance results.