Multiprocessing.shared_memory – Shared memory for direct access across processes

https://docs.python.org/3.8/library/multiprocessing.shared_memory.html

by pjmlp • 7 years ago

84 23 7 years ago

docs.python.org

zozbot123• 7 years ago

Both 3.7 and 3.8 have so many performance improvements. But still, what people like most about Python is code readability. Performance isn't as important as in the past, because of how hardware performance has scaled.

AtHeartEngineer• 7 years ago

Very true, but it does let me make stuff faster than my co-workers who are using C++.. without their "well python is so slow" argument from ten years ago.

takeda2• 7 years ago

It is extremely unlikely (I actually wanted to say "never" but many people were bitten for using that word) that Python will be faster than C++.

So if the hope of being fast keeps you with Python, you choose the wrong language.

gpderetta3• 7 years ago

wait, hardware performance hasn't really progressed much in the last 10 year!

anonymou2• 7 years ago

Not CPUwise. But they have increased the number of CPUs available and that's where python has problems because of the GIL. Fortunately the multiprocessing library seems like a good workaround to the GIL issue.

lunchables• 7 years ago

I guess we can argue the semantics of "much" but comparing my current ryzen system to my core2duo system from 10 years ago the performance difference is massive.

imtringued• 7 years ago

You can't take advantage of the hardware improvements if you're not using all cores.

takeda2• 7 years ago

Well, with multiprocessing you actually do.

malux85• 7 years ago

This is great! As someone who codes lots and lots of multiprocessing + queue code I'd love better shared memory support

ldng• 7 years ago

Now combine that with Apache Arrow and you can do some interesting things.

sametmax3• 7 years ago

Great, the support is still basic, but it's a huge improvement over message passing when you need speed. 2 months ago a student asked me exactly that question: how to share data between processes, and when I told him that with the stdlib he had to manipulate mmap by hand (https://blog.schmichael.com/2011/05/15/sharing-python-data-b...), he was not happy.

Now that asyncio is production ready (we really have had a decent doc and asyncio.run() for a short time), and this, the concurrency story in Python is starting to look decent.

Since the 3.6, things are really going nicely.

Maybe we'll see an async django-like framework emerging from now on instead of the myriad of async flask copycat and django-channels. This would open the gate to live settings, manipulating tasks queues in URL enpoints, having RPC/PUB-SUB baked in. After all, I don't need async to just have exactly the same features as a WSGI framework.

m_ke• 7 years ago

Tom Christie (DRF dev) is working on an async django like toolchain with starlette, databases and uvicorn. https://twitter.com/_tomchristie/status/1100064076191940608

Hopefully django will start supporting asgi soon and let us write async views alongside existing sync stuff.

ausjke2• 7 years ago

"decent doc and asyncio.run() for a short time" -- can you give a URL or some pointers? Thanks.

sametmax3• 7 years ago

Starting from 3.7, the documentation is much, much better (https://docs.python.org/3/library/asyncio.html compared to https://docs.python.org/3.6/library/asyncio.html).

As you'll notice, we now can start program with a simple asyncio.run() instead of manually setuping the event loop. Not only it is less verbose and easier, it's also much less error prone. It should have been in asyncio from the begining, but even a long fight on python-ideas didn't work. Fortunatly, Yuri decided to add on his own.

j88439h842• 7 years ago

I like http://trio.rtfd.org way more than asyncio.

sametmax3• 7 years ago

Me too, although the scheduling syntax is not to my taste. But trio is not in the stdlib, and is incompatible with most of the current ecosystem. There are talks to integrate it in the future, meanwhile there are compat layers, but it's still in infancy.

For now on, I just use asyncio with a lib to add nurseries to it.

jmakov• 7 years ago

Doesn't Apache Arrow solve this problem partially already?

neurohacker• 7 years ago

This rules, but until 3.8 is out I've been using http://semanchuk.com/philip/posix_ipc/ which unfortunately doesn't work on Windows.

Multiprocessing.shared_memory – Shared memory for direct access across processes

Related Stories