A library to assist writing memory-unsafe code in "pure" Python

https://github.com/DavidBuchanan314/unsafe-python

by albertzeyer • 2 years ago

54 16 2 years ago

Galanwe8• 2 years ago

There are some real world use cases for these.

I do use some similar helpers to e.g. create pickable weakrefs for passing large objects to forking multiprocessing pools as arguments, or expose large objects in shared memories. It's not portable against non-CPython implementations, but as long as it's well documented and known, it's fine.

dabacaba• 2 years ago

Sharing Python objects between processes is not safe. Python assumes that access to objects is protected by GIL, but this is not the case when using multiple processes (even if the processes only read from the objects, they still modify the refcounts). Sharing memory buffers can be done safely using mmap.

Galanwe8• 2 years ago

Sharing objects is completely fine upon a forking multiprocessing pool. Each worker process ends up with a (lazy) copy of the parent address space, including the whole python interpreter state.

When you think about it, you already rely on that to access imported modules, global variables, etc in your worker processes. The GIL and reference counting has nothing to care about it either, as they are both copied from the parent process as well, thus you can freely continue to read global state (and even modify, though it will be local to the worker process).

That is, if you have large objects to share with your worker processes, and you don't need to get them back, you can freely just assign them as global variables before starting your multiprocessing pool, and have your workers read them, at zero transfer cost, and safely. This _trick_ has been used since decades in Python.

Now the annoying thing is, that global variable mumbo jumbo is kind of dirty and ad-hoc to maintain. Using custom pickable weakrefs (using something similar to the trick with id() in this repo), you can create proxy weak objects to transfer to your workers, thus achieving the same effect, while keeping a "function argument" interface, instead of proxy8ng through global variables.

hot_gril2• 2 years ago

What do you mean by sharing objects between processes in Python? By default it sends a pickled copy to each subprocess, but there are special things like Manager I haven't used.

owlstuffing• 2 years ago

Sergeant Frank Tree: You shouldn't touch the ordnance at all. But more specifically, you should never pull this hand-operating lever to the rear.

Ward Douglas: Never.

A library to assist writing memory-unsafe code in "pure" Python

Related Stories