A Conversation with Guido about Callbacks

http://oubiwann.blogspot.com/2012/03/conversation-with-guido-about-callbacks.html

by vgnet • 14 years ago

68 45 14 years ago

oubiwann.blogspot.com

ef4• 14 years ago

Callback-heavy programming is what you get when your language doesn't support better ways of controlling context. It's analogous to manual memory management: it's work that's better to outsource to a compiler or runtime, except in specialized circumstances.

For Python, you can get the benefits of asynchronous IO without callback hell if you use the Stackless fork.

A similar idea for Node is node-fibers.

Both let you build exception-safe coroutines.

anonymoushn• 14 years ago

The library referenced in the OP, gevent, uses greenlet rather than stackless to provide this.

j2labs10• 14 years ago

Or just use gevent or eventlet. It's not necessary to use a whole different branch of python.

On top of that, gevent outperforms both twisted and tornado because the eventloop is libev, meaning it's written in C instead of pure python.

gauravk92• 14 years ago

Can you find a source for that, I'd love to read about actual performance statistics between anything and the likes ok twisted and such.

sausagefeet12• 14 years ago

The main problem I see with gevent/eventlet is it has many of the weaknesses of callback and preemptive threaded code with none of the strengths.

- It can't use multiple cores, even if Python got the ability in the future, gevent code simply wouldn't work as intended.

- While gevent is not preemptive, context switches are implicit, that means you cannot write code without locks unless you want to do some handwaving. You have to know the entire system to know that a multistep operation that calls functions is actually safe.

On the net, gevent may be better than Twisted or Tornado, but I think that is saying more about how much those suck than how good gevent is.

glyph3• 14 years ago

Python outperforms C in many cases. http://morepypy.blogspot.com/2011/08/pypy-is-faster-than-c-a...

voidr• 14 years ago

> Callback-heavy programming is what you get when your language doesn't support better ways of controlling context.

UI programming is callback heavy, but I doubt that any language will support "better" ways for that.

So basically UI programmers are used to callbacks, others not that much, and now that event-loop servers are becoming popular, we see confusion from people not familiar with events.

Server-side people just need to figure out how to use callbacks properly and everything is gonna be fine.

All people need to understand is to nest callbacks properly, nested callbacks are the things that confuse most people.

> For Python, you can get the benefits of asynchronous IO without callback hell if you use the Stackless fork.

You can't just abstract away every programing feature that seems hard and expect everything to be fine, it just doesn't work that way, Java developers are still suffering because Java tries real hard to abstract memory management.

cdavid• 14 years ago

I don't think the comparison with UI is so apt. Sure, they both use callbacks, but the granularity is quite different. In UI callbacks, you don't need to insert your own "scheduling points" as you need in explicity asynchronous programming. Mostly, you write functions that reacts to the events generated by the UI, but within those event handler, your own code is mostly synchronous.

I wonder why you mean by using callbacks properly ? The critical point mentioned by Guido is error/corner-case handling: the idiomatic callback-based code I have seen becomes difficult very quickly. I have not seen a case where e.g. deferred-based code ala twisted has been simpler than gevent.

gruseom• 14 years ago

So basically UI programmers are used to callbacks, others not that much, and now that event-loop servers are becoming popular, we see confusion from people not familiar with events.

I don't think that's right. There is a huge difference. UI callbacks ("onClick" and the like) execute synchronously. But the callbacks in event servers execute at completely different times. Indeed, they're typically nested inside code that has long since terminated. This is what causes the learning curve, not grokking events or functions-as-values, which any half-decent programmer can easily do in the unlikely event that they didn't a long time ago. Indeed, the way we write asynchronous callbacks is cognitively dissonant with everything we've been trained to understand about scoping. The idea of a nested scope that doesn't mean "contained within" or "covered by", but rather "at some unknown time later", is inconsistent enough with how programming languages normally work that it can easily trip you up even after you get good at it.

Asynchronous programming is a very different thing than e.g. browser UIs in Javascript. I've programmed those for years. It still took me a long time to get my head around async servers.

luriel• 14 years ago

UI programming doesn't need to be callback heavy.

Rob Pike designed one of the spiritual ancestors of Go, Squeak ( http://doc.cat-v.org/bell_labs/squeak/ ) using the CSP model of concurrency specially to build (G)UIs.

joezydeco• 14 years ago

Qt uses signals/slots instead of callbacks and it works just fine. That's a "better way" as far as I'm concerned.

chipsy• 14 years ago

Agreed, roughly.

Nobody particularly likes reading callbacky code, but it has a lot of power. What we don't have enough of is "structured alternatives" for our "goto."

jQueryIsAwesome• 14 years ago

Slightly on-topic, a post of mine about how to have zero nested callbacks in Javascript: http://javascriptisawesome.blogspot.com/2012/03/zero-nested-...

sausagefeet12• 14 years ago

Seems like one has to be very out of touch with modern approaches to concurrency to like callbacks. Sounds flamey, but I'm just not sure how one can look at Go, or Erlang, or even Ocaml/Lwt with syntax extensions, and think "yes, callbacks are superior to this".

lobster_johnson4• 14 years ago

Surely Erlang uses callbacks heavily, in the sense that you pass functions around. Supervisors, for example, depend heavily on callbacks to receive notifications when something happens to the processes it supervises.

sausagefeet12• 14 years ago

You're confusing things. Callbacks are not the unit of concurrency in Erlang, a process is, which is a sequence of steps. You implement callbacks for behaviors in Erlang, but that is an interface system, not a concurrency unit.

yorhel2• 14 years ago

Rather flamey indeed, but let me take the bait. I've been using callback-style programming in many applications, ranging from IRC bots to application servers and whatnot. I've also been playing around with Go for several months and tried very hard to implement something that is inherently asynchronous. I gave up after about two months, I just couldn't get used to the concurrency model. I always end up with either data races, deadlocks and have otherwise no good overview of the program logic. This is exactly what the article describes, and seems to apply to me as well.

Incidentally, while figuring out how to implement something inherently asynchronous in Go I looked at some IRC client implementations, and noticed how all of those used callback-style programming one way or another. That doesn't inspire much confidence. :(

All in all, I am not sure yet whether this is purely due to my inexperience with concurrent programming or whether I am just inherently incompatible with it. I'd love to be able to play with alternatives to callbacks in the future, but for now it's just way over my head for the things I want to do.

sausagefeet12• 14 years ago

It appears, much like FORTH, callback coding rots the brain. I am surprised you had such issues with deadlocks in Go, it takes some effort to write code that deadlocks in Go (I don't mean that a deadlock cannot be shown with a trivial amount of Go code, but that the style of programming tends to lend itself to not having deadlocks).

> and noticed how all of those used callback-style programming one way or another.

What do you mean precisely here, because I think you are conflating two concepts. There is a distinction between using a callback as a unit of concurrency (do this async task and call THIS function when you're done), and genuinely being an event handler (call THIS function when you get receive a particular IRC event). I would expect to see the latter in Go for IRC code, not the former.

But there is another aspect of this too, beyond code looking prettier. Go/Erlang/Haskell scale to multiple cores as the runtime evolves to support this. Callback codes doesn't, it can't. This doesn't seem to be because callback developers don't see the value in multiple cores, Node has built-in support for spawning a VM per core. This is quite restrictive. Callback developers have to either limit themselves to solving problems that don't need to communicate with each other or use some other means to communicate between VMs. That doesn't sound like progress to me. But shrug, plenty of people don't neat things in Node so maybe it doesn't matter.

aufreak32• 14 years ago

Not sure whether what you saw was an artifact of your long exposure to callbacks. The Go/Erlang kinds of approaches need you to think at a slightly higher level than callbacks. A small step is the concept of a "promise" .. which, in Go, can be modeled (roughly) as a channel on which some process waits to receive a value promised to it that is being computed in another process. At a higher level are streams of data. Do you have an example you found hard to express in, say, Go without thinking "callback"?

aufreak32• 14 years ago

Given how much callback based stuff exists in the Javascript world -- GUIs, Node.js and some recent W3C specs like the FileSystem API -- I thought it ought to be possible to write a simple sequencing function that can take care of this in the JS world. The key to a useful sequencing function is that it should let you customize the sequencing of the actions and recovery techniques pretty easily.

Here is my attempt at it - https://gist.github.com/2192413

Thoughts?

bluesnowmonkey• 14 years ago

I'm waiting for the day when everybody realizes that callbacks and preemptive threading both suck, relative to automatic CPS transformations.

http://matt.might.net/articles/cps-conversion/

A Conversation with Guido about Callbacks

Related Stories