Ask HN: Why is concurrent programming so hard ?

https://news.ycombinator.com/item?id=768220

by jacquesm • 17 years ago

8 11 17 years ago

news.ycombinator.com

This is one of those things that keep coming up over and over again, compared to 'regular' programming programming concurrent (massively parallel) systems is hard.

Lots harder than the simple relation between the number of threads of execution and a single thread should make you suspect.

Unless your problem is one that is easily parallelized you are left with limited alternatives.

Usually this is put down to either architectural issues (von Neumann vs alternatives), sometimes to the difficulties of locking and such, most times to how easy a program can be parallelized.

But even the latter is an effort on the part of the programmer, rarely of the environment the program is written in (though there are some interesting developments in that area).

I have this weird theory that it has to do with something completely different:

Our brains are not wired to have 'multiple parallel threads' of consciousness.

So we have a hard time envisioning in our sequentially wired brains what is going on in a parallel program, which means we feel much more 'at home' in a sequential environment.

It takes a good bit of training to get around that limitation and even then it is not easy.

So we tend to 'break up' our programs into sequential pieces of code, each running in parallel with the rest, and the spots where they meet, like trains using a railroad track, have to be scheduled to make things execute in the proper sequence.

This makes me wonder if we are truly capable of devising a method of parallel execution that will feel 'natural'.

Even in the real world, when we 'concentrate' we single out a thread from all the possible inputs and go step-by-step to analyze that one single process.

What are your thoughts on this ?

ErrantX• 17 years ago

> So we tend to 'break up' our programs into sequential pieces of code, each running in parallel with the rest, and the spots where they meet, like trains using a railroad track, have to be scheduled to make things execute in the proper sequence.

Dead on. We've been working on single threads for SO long and then just adapted that onto multi-threaded architecture now that we have to scale outwards.

I spent the last year of university writing a research project (final year project) trying to come up with a viable form of concurrent programming that didnt just look like a fancy way to multi thread.

But no luck :D

The best I can think of is to create some rock solid standard and an underlying architecture. With the idea being you can write your basic code and the compiler, vm, interpreter does the parallel work for you. But that's basically the same thing and not really solving the problem :)

makmanalp• 17 years ago

I think it's rather not that our brains are inherently that way, it's just that we've (stereotypically) thought that way all our lives.

It's like learning formulas for 2D geometry and then trying to extrapolate to 3D formulas, such as the sum of all the sides of a square vs a cube. If we learned about generalized formulas in n-D, we could easily apply to any dimension at all. But it's harder to think so abstractly in the beginning.

Same thing here, now you're trying to think of algorithms that work with n threads instead of just one.

makecheck• 17 years ago

It's probably more accurate to think of concurrent programming as what's going on in the brains of multiple people, instead of looking at it as how one person's brain would handle parallel threads.

Whenever you need access to information you don't have available, there's a barrier before you can continue that task: maybe you have to go look for somebody, or send a message and wait for a response, or buy a book.

While these dependencies exist, you have some options. You could decide to work on something else; you could stop everything; or you could speculate on what the answer is likely to be, proceed as if that is the correct course, and expect to backtrack if you're wrong. To learn from the experience and not waste as much time in the future, you can remember key facts.

Like any dependency graph, it can explode as more points are involved. If you have many people (or tasks), there's a greater risk that many of them will essentially be unproductive while waiting directly or indirectly for something else. The challenge is to keep everybody in tune and to give them enough to work on so that they're never really wasting time.

mbrubeck• 17 years ago

A program that use both mutable state and concurrency is nondeterministic: when run twice with the same input, it might produce completely different results (based on the order in which concurrent parts execute).

This is one reason that languages good for concurrency tend toward functional programming or other more-declarative paradigms. By eliminating observable state mutation, you can write concurrent programs that are still guaranteed to be deterministic.

sybreon• 17 years ago

Try sheet music?

tocomment• 17 years ago

I tried to make an SVD algorithm run in parallel one time, but for the life of me I just couldn't figure out how to do it. Maybe I just needed a deeper understanding of the linear algebra.

alexgartrell• 17 years ago

We all learned to program in a non-concurrent manner and we're trying to reuse knowledge from that. Instead, we should focus on re-learning programming, namely what it means to have side-effects and global memory and what we should use instead to operate on those things in a safe way.

Concurrent programming isn't hard, debugging concurrent programming is hard. Eliminate side effects, use Software Transactional Memory, and figure out how easy concurrent programming _can_ be.

obvious Clojure plug

andyleclair• 17 years ago

Yes. I hear many a person bitch about concurrent programming. Concurrency is hard ONLY when you have mutable state. Otherwise, you just (call/cc ...) your continuations.

I thank my lucky stars every day that I go to a school where the first thing you learn is Scheme. Thanks, Matthias!

clord• 17 years ago

Humans are immensely good at concurrent thought, and really bad at sequential thought.

Consider driving. Your brain is processing millions of things in an out-of-order manner, and producing continuous motions in your hands and feet that guide the car so perfectly that few would be surprised to hear of someone driving for a lifetime without a major accident.

Same deal with many concurrent activities, like having a conversation, or playing sports, or listening to music, writing, thinking about the future, or fixing the fridge. In every case, we set our massively parallel architecture on many problems simultaneously, and then integrate a solution in real time without deadlocks or timeouts.

Humans have a corresponding weakness for sequential thought. We find it very hard to write a piece of sequential software that lasts more than a week without crashing hard or suffering from a major design flaw. We need to be taught for decades in the most advanced institutions to think sequentially and logically. Only the most elite of us ever manage to produce something that others would actually pay for, and even then we may solve a different problem for people than the one we set out to solve. (Think of startups like twitter that find that the market likes them, but for completely different reason than what they thought.)

Then we try to make people trained in sequential thought solve parallel problems, and they come up with steamy piles of tangled mess, full of deadlocks, races, and side-effects.

Our DEFAULT state is concurrent thought. We've all been trained to think otherwise, by our computers, languages, and education. We're hard on the poor folks who were carefully trained as sequentialists when they fail to make a parallel problem easy.

I work on a compiler for sequential source at a major corp by day, but my personal long-term goal is to create a compiler for a language that can form the kernel for a new school of thought separate from the sequentialists and their half-measures towards solving this problem. As hardware evolves, it will only become more like our own brains, and less like calculators and simple programmable machines.

jacquesm• 17 years ago

I see what you're getting at but I would call that unconscious thought. There is a great deal of that going on.

When you learn something it is conscious. Every bit of the activity that you are learning is new and requires lots of hard work. But at some points you have 'mastered' the skill, and you can do it 'without thinking', which is really just another way of saying that your unconscious has enough information to do the thinking without direction.

For instance, most of the people reading this are probably good typists (to me that's a part of 'hacking', the sound a typewriter keyboard would make when you 'hack' out a piece of text). All of us had to learn that skill at some point, one conscious and slow stroke at the time. But right now I'm typing this just a little slower than I can think it, or maybe even faster in spurts, depending on how frequently I've used those patterns.

If I tried to consciously think of every letter that I'm typing immediately I slow down to a crawl.

So we're good at parallel thought, no doubt about that, we multitask like we're made for it. But we have but one state of consciousness and we can not fork that even for the simplest of tasks.

Take chess players that can play on many boards. That's as close to human 'multitasking' as you can get when making a computer analogy. The state of each board is the thread of execution, the consciousness is doing 'multiple-instruction-single-data' across each board. On a 'task switch' (move to another player) the state gets reloaded and another move is produced.

If you could consciously think multiple threads then it should be much easier to play on many boards at the same time.

I wonder how well octopuses would perform at that task...

vicaya• 17 years ago

General concurrent programming is hard because of the shared mutable states and their interdependencies, which are ripe grounds for deadlocks and race conditions. However, there are many successful attempts to solve large chunks of problems by avoiding shared states in user/application code: e.g., actor models (popularized by Erlang) and map-reduce frameworks, which can be used to solve a large class of parallel programming problems.

Ask HN: Why is concurrent programming so hard ?

Related Stories