Python 3.5 and Multitasking

http://brianschrader.com/archive/python-35-and-multitasking/

by sonicrocketman • 11 years ago

84 99 11 years ago

brianschrader.com

gcb0• 11 years ago

so is it a little syntax sugar on top of the multiprocess module to please ios developers?

there's nothing new that i can see under the hood

keypusher3• 11 years ago

no, asyncio is a completely different module and paradigm from multiprocessing. the new standard syntax uses asyncio, and none of it has anything to do with ios

sonicrocketman7• 11 years ago

I mentioned iOS because Apple's Grand Central Dispatch API is something I wish Python had. Python's multitasking is so complicated while Apple's GCD is so elegant. It's sad really.

Lofkin7• 11 years ago

Have you seen this: http://dask.pydata.org/en/latest/

Lofkin7• 11 years ago

Your example is trivially parallelized with some decorators from the dask library.

Example here: http://dask.pydata.org/en/latest/imperative.html

fijal5• 11 years ago

Maybe it's worth noting, maybe not, but this extremely trivial example (e.g. serial.py) gets executed 20x faster just by using pypy which speeds up serial execution. This is more than you would get by any sort of multiprocessing/threading shengenians just by using an optimizing VM (and granted, this example is very simple, but maybe addressing basic performance problems should come first)

sonicrocketman7• 11 years ago

I really wanted to take a look to how PyPy handles this. Now I'm even more so. Thanks!

keypusher3• 11 years ago

if you are blocking on io (such as read/write to disk or over the network), pypy execution speed will not help while threading will.

fijal5• 11 years ago

obviously, but the time of blocking on IO should be the same on all strategies + CPU overhead of doing that

sandGorgon4• 11 years ago

pypy has not been updated in a while (it's compatibility is with Python 3.25).

what's even more worrying is that it had almost run out of funds in July. Currently, it has 5000$ in the fund earmarked for Py3 support (http://pypy.org/py3donate.html)

somehow I thought that Google was supporting Pypy.

fijal5• 11 years ago

PyPy is being update very regularly with 3 releases a year on average. Most of the work however focuses on things our users really want - stability, performance, C extension support, warmup speed, memory consumption - you know the mundane stuff. If someone is willing to put effort into supporting more python 3, that's great! we would welcome the contributions.

PS. We're closing onto 3.3 release soon

PPS. Google donated a bit of money to PyPy when Guido was there, definitely "google supporting pypy" is a bit of a stretch. It was years ago though

vegabook10• 11 years ago

In other words, a lost opportunity. Asyncio and this new syntax is both hard for beginners and experienced python coders alike, and still doesn't do multicore!. We're clocking up 3.x version numbers as if this will magically provide the illusion of progress, but the killer feature is still not there. Instead we get type annotations. In a dynamic language. Which doesn't compile and therefore doesn't need them. With no performance advantage. If I have to type declare everything, I want 10x performance. Okay?

If Python were a listed company, the CEO would have been replaced long ago. I'm tired of watching my favourite language flail around like this. Will Continuum Analytics or Enthought please fork 2.7?

sonicrocketman7• 11 years ago

I think the limitations of the GIL are not all in all a bad thing, but I do think that the Python community needs to seriously look at making Asynchronous execution a priority in future versions. Asyncio is really complicated and provides very little in terms of performance.

Multitasking is the most important issue that Python faces today (and maybe PyPy is the answer).

fijal5• 11 years ago

The problem that python is facing (and pypy is not solving) is that in order to have a working async library ecosystem, everything, especially the stdlib has to be async aware. This is a far cry right now and I don't see any of the python devs really trying to address that. Otherwise you end up with the same situation as for twisted - it works and it's great, but every single library needs to be made twisted aware

vegabook10• 11 years ago

So we're in 3.5 already, and one of its central features, async, is flawed, you say. So not only do we not have a killer feature, but even one of the nice-to-haves is a dog. Timeout. This project needs new leadership. The competition (Golang) is walking all over Python on the async issue.

pdonis2• 11 years ago

Asyncio and this new syntax is both hard for beginners and experienced python coders alike, and still doesn't do multicore!.

The fundamental limitation here isn't unique to Python. Multiple tasks that are CPU bound simply cannot be managed using cooperative multitasking, and that's all asyncio is: cooperative multitasking, with a friendlier syntax than the old asyncore module.

Cooperative multitasking using async I/O works great when you have lots of tasks running that are I/O bound--they spend almost all of their time waiting for network or disk reads/writes. But as soon as you start piling up tasks that require CPU, you need either multiple threads or multiple processes. Python asyncio was never intended to handle that use case.

Multiple threads is where Python has a unique limitation: the GIL prevents multiple threads from running on multiple cores, so if you want multiple CPU bound tasks to use multiple cores, you need to fork each one into a separate process.

It also doesn't help that the multiprocessing module is, IMO, a huge ball of cruft that's overkill for a lot of multitasking use cases. But there are simpler ways to do it, at least on Unix systems, because the fork system call is very lightweight.

Shameless plug: I wrote a library for this some time ago, which I use whenever I have a Python project that needs to fork worker processes; it's the "comm" sub-package in my plib-io Python distribution (this is the Python 3 version):

https://bitbucket.org/pdonis/plib3-io/src

vegabook10• 11 years ago

I know it's not easy to graft multicore onto Python. So why pretend? If you're serious about web programming you're not using Python anyway. You're using JS or Golang. And if you insist on python, we have plenty of options to do async i/o already. Optimising async is like polishing the doorknobs on the titanic. Waste of effort.

What we need, is a concentrated focus on the only two areas where Python beats the competition hands down. Numerical computing, and newby accessibility. The first gets a token "@" operator (great, thanks), the second is moving backwards with 3.x. Asyncio, type annotations, and unicode, are answers to questions nobody is asking (except by those who refuse to use the right tool for the right job).

jaimebuelta• 11 years ago

I fail to see how asyncio should use multicore. Asynchronous programming uses a single thread...

If you use CPU-bound tasks, using non-blocking I/O (which is what asyncio is about) is not going to work. You can use multiple processes (maybe from concurrent.futures[1]) to help that case.

[1] https://docs.python.org/3.5/library/concurrent.futures.html#...

armitron4• 11 years ago

Amen brother.

If you make one mistake in your thinking, it's treating Python as more than a christmas hack (which it started out as). It's not a serious language, it has no design behind it and is decades behind the state of the art.

Use it for throw-away scripts and such is what I say, it's an area where it shines.

The moment you want to _architect_ something substantial, reach for something else cause chewing gum and duct tape is not going to do it.

vegabook10• 11 years ago

Amen buddy. I hack in Python. I discover in Python. I script, and I pre-production in Python. Then I do C or Java or even JS. But none of my pro-code would have been possible without all the learning I accumulated about the problem in good ol' 2.7 interactive and its arsenal of tools.

[edit] Actually I think you're right. I think it's time to graduate to something serious. My mistake is to think that Python can be that serious language. 3.x, by trying to be that, is horribly compromising the basics of what Python is all about, namely accessibility, friendliness, discovery, malleability. Not production.

zeeg• 11 years ago

We are pretty serious about the web, infrastructure, and platforms in general, and we use Python. As does Dropbox, Google, Facebook, Twitter, etc.

I don't think concurrency is great in Python, but you're living in the wrong world if you dont think people are using the language for real things.

Lofkin7• 11 years ago

They pretty much already did, but with 3.x: http://pyparallel.org/

keypusher3• 11 years ago

concurrency in python is kind of a disaster, in my opionion. there are a lot of different options but they all seem to have significant drawbacks, and not just limited to ease of use. i know concurrency is a hard problem, but i wish there was one really good, straightforward solution instead of 3 or 4 different half-baked convoluted solutions (threading, multiprocessing, asyncio, subprocess in stdlib, plus twisted, gevent, pulsar etc as third-party).

sonicrocketman7• 11 years ago

From what I see, the community keeps changing its mind every few years. We need to pick a multitasking method and really embrace it. If we want to keep the GIL, then we need to go whole hog on some other approach.

vegabook10• 11 years ago

A community which changes its mind every few years is a community which does not have effective leadership. Python's leadership is ineffective because 3.x has erased its credibility. You have maximum one opportunity to change a language in mid course. That opportunity has already been used (wasted?) in the case of Python, and while it could have addressed the elephant in the room, multicore, it did not. Instead it indulged the wishes of its powers that be to tinker on the edges to make Python more "serious", in the process adding nothing that anybody needed, while reducing simplicity and accessibility, and, criminal sin, no backward compatibility with no corresponding forward killer feature.

Now, nothing less than a wholesale change of the leadership can right the listing ship.

agentultra2• 11 years ago

This isn't a problem unique to Python. There are a plethora of concurrency solutions in every language because there isn't a one-size-fits-all generic solution for every workload that exists.

armitron4• 11 years ago

It's unique to Python in that most other languages used for more than just scripting have true concurrency via OS preemptive threads with no GIL.

Python is really decades behind in this area and no async hack is going to improve matters. I think it's safe to say that this is an obvious example of how Python is not really a general purpose language (disregarding the fact that it started out as a christmas hack and really has no design behind it). It should be used for scripting and quick prototyping but architecting anything substantial on it is not advisable.

ikken5• 11 years ago

We see people hyping all the time for NodeJS mainly because it can handle millions of concurrently open connections. Now we get a much cleaner (IMHO) way to achieve the same thing in python (of course python is slower) - i.e. see aiohttp as websockets server. I would not call it a disaster.

armitron4• 11 years ago

This is like the cripple making fun of the blind man.

When your competition is NodeJS there is nothing really left to say.

nomel• 11 years ago

It's great that it works for you for this one particular type of application, but the whole world isn't io bound. Some of us just want regular, plain old, concurrency. As this shows, multiprocessing is still the only, unsane, solution right now. I just want to be able to make a thread, and do stuff in it in parallel to other threads, like I can with most other languages.

"Go write a C extension" you tell me, "use something besides cPython" he says, "just use multiprocessing" I hear. Sure...but ffs, we've had multicore processors for almost TWO DECADES now.

One of my biggest, and apparently unchanging, problems with Python is the desire to keep things simple in the interpreter, to the disadvantage of the language. Sure "implementation for interpreters may vary" blah blah, but you have to target the bottom end in performance, and most widely installed, which is definitely cPython for both points.

mangecoeur5• 11 years ago

And ironically, nodejs is also single threaded with it's own GIL, which somehow no one ever mentions. It's great for async IO bound stuff but is really worse that Python for mutlicore CPU bound work - at least Python has concurrent.futures which makes throwing together a parallel map operation a few lines of code.

mangecoeur5• 11 years ago

Concurrency is just difficult, with no single solution fully addressing every use case. There are a few mature approaches to python concurrency, their weaknesses lie in the fundamental tradeoffs rather than buggy code. Not that there isn't room for improvement, but its almost impossible to have one solution that solves everything.

vegabook10• 11 years ago

Then why do we have an official concurrency module? If there is "no single solution fully addressing every use case", why do we have one official solution? Which the main post criticises? If you are correct that there is no single solution, then surely there should be no single solution put forward as official?

Yet I believe your premise is incorrect in the first place. Why has Golang, by most commentators' comments, "got concurrency right". Is that true? Tell me. If it is, why can't Python get concurrency right also? If it's not true, why is there an official solution, being criticised before our eyes?

On this case, I will quote you: "I suspect you don't really know what you are talking about...".

ikken5• 11 years ago

I've been using the new async/await syntax to write beautiful asynchronous websockets server and I fell in love with it. It handles hundreds of thousands of concurrent connections (on top of aiohttp) and the code is so much cleaner than it would be with i.e. NodeJS and Express and Promises. It reads like a serial code.

I think benchmarking asyncio with any type of CPU bound tasks misses the point. Previously we were relying on hacks like monkeypatching with gevent, but now we've been presented with clean, explicit and beautiful way to write massivly parralel servers in Python.

findjashua2• 11 years ago

fyi: ES2016 javascript also supports async/await, and you can use them today w babel

Kudos2• 11 years ago

No it doesn't. It just hit stage 3 in TC39 yesterday, meaning browsers should be considering implementing it experimentally in order to solicit feedback from developers. When it progresses beyond that you can start calling it ES2016.

tomchristie• 11 years ago

Anything you can share? Some good examples of networking code with async/await would be incredibly helpful. The documentation covers the primatives but I have a hard time putting it all together.

ikken5• 11 years ago

Unfortunately the code is not open source - I'll try to open parts of it in the future.

But please do check this simple gist I found some time ago that helped me understand how powerful asyncio is:

https://gist.github.com/gdamjan/d7333a4d9069af96fa4d

1st15• 11 years ago

Thanks for writing this!

[..] It handles hundreds of thousands of concurrent connections [..]

Is it a single process application, or you use a few OS processes behind some load balancer?

ikken5• 11 years ago

Yes, it is a single process application. But to utilize all cores I use Gunicorn to spawn multiple processes.

http://aiohttp.readthedocs.org/en/stable/gunicorn.html

ak217• 11 years ago

This mirrors my experience precisely. I was very excited about async/await, hoping that this would integrate coroutines into regular Python scripts, without the need to manage some complex dispatch engine. I was equally disappointed to learn that it's business as usual, with painful and inadequate semantics out of the box.

At least we have pypy. The community should really be rallying behind that project.

1st15• 11 years ago

This mirrors my experience precisely. I was very excited about async/await, hoping that this would integrate coroutines into regular Python scripts, without the need to manage some complex dispatch engine. I was equally disappointed to learn that it's business as usual, with painful and inadequate semantics out of the box.

You're not supposed to use async/await without a framework like asyncio or tornado.

One way to provide a better UX is to merge asyncio into Python on a deeper level, but this is something that many people won't like.

btreecat• 11 years ago

I did this for two reasons, the first being that I cannot, for the life of me, figure out how to use asyncio to do local file IO and not a network request, but maybe I'm just an idiot.

I don't think you are an idiot I just think you didn't search well enough. There is a reason there is no local file IO with asyncio.

Check these links for more info:

* https://stackoverflow.com/questions/87892/what-is-the-status...

* http://blog.libtorrent.org/2012/10/asynchronous-disk-io/

From what I understand the way libuv (what node.js uses) gets around OS limits is with a thread pool.

Also, this documentation might be helpful:

* https://docs.python.org/3.5/library/asyncio-dev.html#handle-...

My experiences don't seem to mirror your frustration. I found asyncio quite useful for tinkering with a simple web scraper that hit multiple sites at once (each with a different response time) and munged all the data together into one data set.

Thanks for the write-up!

sonicrocketman7• 11 years ago

Thanks for the suggestions!

sandGorgon4• 11 years ago

have you used a database in a non-blocking way with asyncio ? I'm thinking of the way Psycogreen works or node-postgres works.

ikken5• 11 years ago

Please check aiopg [1]. There's also an interesting list of packages for asyncio [2].

[1] https://aiopg.readthedocs.org/en/stable/

[2] http://asyncio.org/

csytan• 11 years ago

While I haven't had a chance to use 3.5's async/await syntax, I have used AsyncIO pretty heavily to deal with multiple sensor inputs/outputs on a Raspberry Pi.

The author is right. If you write a coroutine, any code that uses it must also be a coroutine. This is pretty annoying when you're trying to test something manually. It bubbles up this way until you eventually hit the event loop.

If you're trying to debug a coroutine in the interactive shell you've got to do something like this:

  loop = asyncio.get_event_loop()
  # Blocking call which returns when the hello_world() coroutine is done
  loop.run_until_complete(hello_world())
  loop.close()

That's my main beef with it. Debugging can also be painful because when you hit an exception, your stack trace will also involve the asyncio library. Aside from those complaints, I'm a fan. It works fine and reads better than callback-style code.

mangecoeur5• 11 years ago

You have a similar problem in nodejs - if the api you want to use is async it has to take callbacks, so if you have to do something with that result you have to nest in another callback etc... it gets ridiculous. But it's pretty much a fundamental issue with async code - to keep things async then you have to set up the whole interdependent network of functions to be able to work async, since you don't know when anything is going to return its value.

It takes some getting used to - I think Python people are likely to have more trouble with this precisely because Python is a very clear, explicit, and mostly imperative language -you can read Python code as a sequence of instructions and that will be pretty much the way it gets executed, which makes understanding programs very easy. Async is simply a less intuitive way to program, so to adopt it you have to be sure it's worth the hassle of giving up easily understood code.

n0us• 11 years ago

When he brings up requests in comparison to urllib he seems to not know about aiohttp https://github.com/KeepSafe/aiohttp

justinlardinois• 11 years ago

Tangentially related: of the people and projects that are using Python 3, why? I've found that aside from syntax and a few features here and there, Python 2 and 3 are more or less the same technology-wise, especially since 2.7 has a lot of features backported from 3. Thus to me it seems better to stick with 2 because there's so many existing libraries and CPython is the only implementation with complete Python 3 support.

If Python 3 had good concurrency and optimization (something neither version has right now), I'd consider using it, but is there an already existing reason that I'm just not seeing?

fweespeech• 11 years ago

Python 2 has an EOL in 2020.

Why start a new project in a language that will have support dropped in ~4 years?

If it isn't going to be around in less than 5 years, I'm not going to use it for new projects which have lifespans measured in decades. I've got projects still in production, almost unchanged [only changes were to update business rules], for 4 years. It moves millions of dollars a year. I sure as hell don't want to worry about OS support for it another 4 years.

antoinealb• 11 years ago

The amount of libraries working on Python 2 only is really small and not a problem for a lot of us: http://py3readiness.org/

The syntax is really nicer IMHO. Unicode strings are really nice to have too. Plus Python 2.x will be EOL'd soon. I just don't see the point of working with it anymore.

mangecoeur5• 11 years ago

Simple answer - It's the current (and currently updated) version. Why would you not use it when starting a new project?

More fancy answer - There's plenty of nice improvements to the language (better handling of iterators, tidied up std lib, modern objects, etc), though it's probably worth it for unicode support alone - trying to convince python2 to properly handle international text is just a huge pain (also - no you can't just replace all accented characters with non accented ones if you want to preserve the correct meaning of text and not piss off all your customers by misspelling their names). Yes you can eventually solve all problems with python2. But why bother when you can just use the latest version, there's basically no cost for new projects.

maw• 11 years ago

it's probably worth it for unicode support alone

Yes. I'd probably accept a slightly worse language than python 2 but with reasonable unicode support. Fortunately, there's no need to.

vegabook10• 11 years ago

There is a very clear cost for many new projects: infinite... for the simple reason that some key libraries still do not do Python 3 at all. Let's be clear. Cassandra is the best large ingest nosql database out there. Python 2 only on CQLSH. Bloomberg. The default terminal used by the 300 000 most important financial people on earth. API Python 2 only. Theano. 3.x via 2to3 only. Anaconda, 2 still the default download with 3 an afterthought (in other words, even for new downloads, 2 still the majority!). This is more than about stuff not being ported yet. It's about people preferring 2.x.

This idea that Python 3 is cost free needs to be expunged. For large classes of users, Python 3 is not possible even if they want to (which they don't). Stop this erroneous propaganda. Unicode is nice if you're a web guy, and if you're a web guy, why are you using Python already? Unicode is completely irrelevant for everybody else and is most definitely not a core reason to move to Python 3. You're living in cloud cuckoo land on your 3.x magic mushroom trip.

Earth to web jockeys. Python's hardcore is numerical computing, and that hardcore is not on Python 3, and is not moving anytime soon, and certainly not for the dubious benefit of unicode. Moreover the web stack is much less important to Python than is numerical and scientific computing, for the simple reason that while the former has multiple better competitors in the form of golang, JS et al, the latter does not.

[sidebar: since when does ascii not cater for accents?? I am bilingual french / english and I have never had a problem typing french accents in ascii? You're creating misleading propaganda again. Once again, EatHeart, I quote you: "I suspect you don't really know what you are talking about...", or worse, you have an agenda to mislead.]

e1ven• 11 years ago

For me personally, I've had much better experiences with Py3 than I ever did with Py2. I've used Py3 for most of my projects since 2012, and I've never run into any major problems.

Despite what people like to say on forums, pretty much every library I'd want to use works fine. And it has for a long time.

When I've tried to go back and use Py2, I keep getting annoyed by simple things. The unicode is a pita. Not having keyword-only arguments is annoying. Not having "yield from" means more dancing. Py2 forces me to manually make a lru_cache decorator, instead of just working..

From my perspective, sure, I _could_ use Python 2.x.. But Python3 works better and is easier.

The question isn't "Why Python3", it's "Why the fsck wouldn't you?"

SiVal• 11 years ago

Python 3 has the advantage of having a future. It is the living branch of the language.

The advantage of Python 2 is an evaporating pool of legacy Python2-only libraries and legacy Python2-only programmers. Python2 is dying at the rate that pool is evaporating.

Python 3 has the advantage of having a future, plus all the advantages of Python 2 and more with the exception of that shrinking pool. I have already replaced all my Python 2 code and won't intentionally write more code that I know will have to be replaced. I'd like anything I build now to have the greatest likelihood of still being something I can continue building on in the future. Python 3 is more likely to fit that requirement than Python 2.

I'm teaching my kids to program. Python 3 is a good choice for doing so. Python 2 would set them up to be teenage maintenance programmers. Doing that to them would be child abuse. I upgrade my own skills, too, so wouldn't choosing Python 2 for my own upgrades be a form of self-abuse?

keyneus• 11 years ago

Type annotations. They're the sole reason we moved from Python2 to Python3, and it was completely worth it. With mypy, we now have at least some semblance of a type system and type checking, which is a huge win.

Unicode support is vastly improved.

Finally there are have specific exceptions like FileNotFoundError so you don't have to catch OSError and then check errno. I got so tired of that.

No more old-style classes, which means you know you can use super(), which is now less annoying since you don't need to “remind” Python which class you're in.

Lots of other small changes, most of which do lead to improvements, however minuscule. I would argue that unless you are must use a Python2-only library, just go with 3.

laurencerowe• 11 years ago

I held of switching to Python3 until last year because I was worried about library support. But we're at a point now where the vast majority of currently maintained libraries are Python 3 compatible.

In the end what pushed me over were the improvements to the standard library multiprocessing module only available in Python 3.

antod• 11 years ago

I've been slow to move to 3.x, but have just recently started using 3.4 as my first Python 3.

I never had any problem with 3.x conceptually, I was just slow/lazy and because I could see how each new 3.x release got better and better there wasn't much rush. I always thought I would move to 3.x one day. And 2.7 was a nice place to wait for awhile across distro upgrades.

I haven't bothered with any migration of old code yet - just starting new stuff in 3.4. I've got one project that aims to be 2.7/3.4+ bilingual and it's going well so far.

Now that I've tried it I do like 3.4 - four releases of small feature improvements do add up eventually. I think Python's strategy of keeping 2.7 around for a long time to let stuff migrate when ready is a good one. Each new 3.x release also makes migration from 2.7 a bit easier. Not worrying about supporting 3.0, 3.1 or 3.2 (and 3.3 even?) also makes migration easier too.

I'd recommend moving to 3 for new projects. Migrating existing projects is more of a case by case deal where it may or may not be worth it yet or even ever.

aidenn02• 11 years ago

I don't use python all that much, but out of curiosity, what's the problem with multiprocessing? In the languages I do develop in, I find it much easier to reason about than multithreading.

andor• 11 years ago

Data exchanged between the processes is serialized with pickle. Pickling is slow, adds latency, and doesn't work on all objects.

https://docs.python.org/2/library/multiprocessing.html#pipes...

aidenn02• 11 years ago

That sucks; there's no shared-memory message-queue implementation?

1st15• 11 years ago

First of all, see this pic: https://pbs.twimg.com/media/COLLg0TUAAA4j79.jpg:large

asyncio doesn't provide any nio abstractions for files because (a) it's not really needed, and (b) there is no easy way to implement it.

(a) basically you shouldn't expect your code to block on disk io. But even if it does block for a very short amount of time it's probably fine.

(b) one way to implement nio for files is to use a threadpool. Maybe we'll add this in later versions of asyncio, but it will require to write some pretty low level code in C (and reimplement big chunks of asyncio in C too). Another option is to use modern APIs like aio in Linux, but as far as I know almost nobody uses it for real.

Bottom line -- you don't need coroutines or asyncio to do file io. What you need asyncio (and frameworks like aiohttp) is to do network programming in Python efficiently.

sonicrocketman7• 11 years ago

After playing with Asyncio more, I certainly see your point. I guess my larger point was that, even with the new async/await syntax, Python's general multitasking is still rough and incomplete imo. I thought async/await was a solution, but it's not. I just imagine it being so much simpler.

athrun• 11 years ago

"you shouldn't expect your code to block on disk io"

uh?! In what world are you living in? Blocking the main thread of your app on disk I/O makes for a poor user experience.

Disk I/O is a big deal for a whole range of applications. I would even say almost all apps have to deal with disk I/O at some point, as opposed to network I/O.

beagle35• 11 years ago

basically you shouldn't expect your code to block on disk io. But even if it does block for a very short amount of time it's probably fine.

No, it's very often not fine. Magnetic disks, still the norm for many, and definitely with large storage, often go as low as 5KB/s for random access (or even sequential access to very fragmented files). Reading a 1MB file can easily take 5-10 seconds in some setups - which is not acceptable for any interactive service. It's not fine for a web server to not service any requests for 5 seconds.

Another option is to use modern APIs like aio in Linux, but as far as I know almost nobody uses it for real.

Anyone I know who tried came back screaming. There is no way to do an async file open, for example - which means that if you rely on aio, you can block for 10 minutes waiting for an NFS or SMB mounted file to open.

The only sane, portable thing to do for Unix/Posix is use a threadpool for async file io - or just use something like libuv which already abstracted async operations this way.

Lofkin7• 11 years ago

Better multitasking and concurrency syntax, with task scheduling and out of core support here: http://dask.pydata.org/

Python 3.5 and Multitasking

Related Stories