Oxidizing bmap-tools: rewriting a Python project in Rust

https://www.collabora.com/news-and-blog/blog/2023/03/03/oxidizing-bmap-tools-rewriting-a-python-project-in-rust/

by glenngillen • 3 years ago

70 33 3 years ago

I’m confused… you’re talking about avoid a local copy of sparse regions… Linux already does that at the level of the inode. There’s also a seek operation to move past the next hole. Not sure why you would carry around metadata the filesystem is already tracking for you.

masklinn4• 3 years ago

Not sure why you would carry around metadata the filesystem is already tracking for you.

Because bmap files are independent of the filesystem and OS, and thus would probably like to work even with filesystems which don't support sparse files, and OS which don't expose holes?

For instance until NFS 4.2 in 2016 you could write sparse files to an NFS volume, but there was no way to detect holes when reading. exfat doesn't support sparse files at all. And according to their man pages, OpenBSD and NetBSD have yet to support SEEK_HOLE/SEEK_DATA (which are non-standard extensions of POSIX lseek(2)).

Plus according to its history the bmaptools project was created about a year after the release of kernel 3.1, which introduced support for SEEK_HOLE and SEEK_DATA. Doesn't take much of a leap to assume that the project's creator didn't consider that widespread enough to be reliable (Debian wouldn't release a 3.x-based version until the following year).

andrewshadura• 3 years ago

Seeking through holes also doesn't work very well for compressed images, since usually there is no way to tell apart an insignificant hole from a long sequence of zeroes or other filler data.

agildehaus• 3 years ago

An example from my work:

We have a Yocto build that results in about 120MB worth of files that make up our app and Yocto. Originally we had a script that would write a bootloader, partition and format ext4 our target's eMMC, and decompress a 120MB tarball to that filesystem.

That worked well, but we wanted our script to become OS-independent, as our field team ran Windows laptops. It's quite difficult to get Windows to do an ext4 format, and I wanted our tool to have a minimal number of dependencies (e2fsprogs requirement? some proprietary thing from Paragon? no thanks)

So instead, have Yocto produce an image containing the bootloader and all four pre-formatted ext4 filesystems. No operating system needs to do the format if the filesystem already exits within, it's just a raw block write. But now the image is 4GB, the size of our eMMC, and writing all of it would be painfully slow.

Thankfully Yocto also outputs a bmap file which maps the parts of that 4GB which are empty space -- blocks we don't need to write when commissioning our target device. So our commissioning tool was rewritten in Go, and I wrote a bmap implementation in Go to do the write. Flashing our target is as fast as it used to be, but now that tool can be easily made to work on multiple operating systems.

masklinn4• 3 years ago

There's very little content to the article sadly, aside from links to the artefacts.

It also seems to have been done independently of the upstream, so it's not really an "oxidation" in the usual terms, more of a pseudo-fork of the specific `bmaptool copy` subcommand (though TBF it only has one other subcommand which is `create`, and the implementation in the upstream is about 1/3rd that of copy, so copy is clearly the "meat" of the project).

UncleEntity2• 3 years ago

Usually a project is oxidised into Rust because of many reasons, the main usually being memory safety.

What about python’s memory model is unsafe?

pohl3• 3 years ago

Does Python do anything to enforce mutually exclusive access when mutating? If not, that’s a hole you could drive a truck through, isn’t it?

gpm3• 3 years ago

Isn't Python still run under a single global interpreter lock? Can't have simultaneous access while mutating if only one thing is running at a time...

brundolf• 3 years ago

The article felt kind of disjointed, I think that statement was just meant generally and not meant to suggest it applies here

masklinn4• 3 years ago

Also how I interpreted it, though even there it's quite weird (e.g. better performances is also a common reason to convert things to Rust, especially when "easy binding" tools like pyo3, neon, or rustler are available and take care of the unsafe bits between the two).

UncleEntity2• 3 years ago

Yeah… It’s becoming a pet peeve that the Rustafarians believe they have a monopoly on “memory safety” and need to point it out all the time.

claytonjy3• 3 years ago

Which Rust-written Python tools are folks using?

I know of two big ones: ruff (linting) and pyflow (dependency management). The standard lib crypto module uses rust, too.

Are there other ones I should know about? Maybe replacements for mypy, pre-commit, tox/nox?

1f60c• 3 years ago

The standard lib crypto module uses rust, too.

This couldn't matter less, but I think you're confused with the third-party Cryptography package, which uses Rust.

claytonjy3• 3 years ago

My bad, thanks for the correction!

bogeholm• 3 years ago

polars, pydantic and deltalake come to mind

- https://pypi.org/project/polars/

- https://pypi.org/project/pydantic/

- https://pypi.org/project/deltalake/

jamincan• 3 years ago

Does Pydantic use rust? When I check the github repo, it shows 100% python.

stusmall• 3 years ago

I'm a huge fan of maturin but that might be a bit meta. It's a python build tool for wrapping rust projects. There github page has a list of projects using it and you might find more of what you are looking for: https://github.com/PyO3/maturin#examples

_2d30• 3 years ago

Wow I really like the terminology "oxidizing" for re-writing something in Rust.

Sorry for the unsubstantive comment.

Oxidizing bmap-tools: rewriting a Python project in Rust

Related Stories