Understanding Real-World Concurrency Bugs in Go [pdf]

https://golangweekly.com/link/59972/b208593eda

by oldgun • 7 years ago

165 70 7 years ago

The terminology could use some refinement. Not sure "message passing" is the right phrase to describe Go channels.

Most encounters I've had with "message passing"--whether it was AJAX, Win32 PostMessage, or grade school--have been non-blocking. Go channels block.

edit: Here's a pretty good write-up from 2016 on why channels are an anti-pattern:

https://www.jtolio.com/2016/03/go-channels-are-bad-and-you-s...

zzzcpan2• 7 years ago

It's fine. It's still message passing and it's obvious they refer to Go flavored CSP style message passing whenever they talk about it, not anything else.

eternalban• 7 years ago

(I've been using Go since it was released and like the language.)

Go tried CSP in the beginning but in CSP the 'sequential process' preempts only on blocking (IO) or until finished.

Also message passing is just that. You can't reach out and modify a message after you have posted it, IRL. This means messages are 'safe' to use concurrently since they are copies. That, in conjuction with strict CSP ssemantics (processes always run to completion with no preemptive cooperation) gets you 'safe concurrency' at the price of performance and at scale 'reasoning' about what is happening.

So, the practical decisions were (a) honors system message passing via references, (b) preemption at IO and elsewhere, and (c) introduction of mutex, etc. to fall back to [more performant] 'light weight threads' and 'locks' paradigm.

api• 7 years ago

I don't see why blocking is an issue given that goroutines are light weight threads. They are really cheap and can be created as needed, eliminating much of the need for async patterns. It's one of the nicest things about Go given that async is just manually implemented lightweight threading.

I would say though that go channels have not turned out to be as useful as I thought they would be. I dont find myself using them that much. But maybe it's just my style or the type of stuff I am writing.

ec109685• 7 years ago

Lightweight threads are different than async models. With the latter work continues until io or a manual yield happens. With threads, your workload can be preempted outside your control.

jstewartmobile3• 7 years ago

Blocking is a pretty big deal.

With something like pipes or sockets, there are timeouts and saner handling of breakages. With Go channels, one false move, and you've got a panic or an infinite wait.

I like Go, just not a big fan of channels.

pcwalton8• 7 years ago

It's one of the nicest things about Go given that async is just manually implemented lightweight threading.

No, async/await is typically stackless, while goroutines have stacks. That's a significant difference.

littlestymaar2• 7 years ago

Blocking implies you can get deadlocks if you're not cautious, whereas with async you just can't.

zzzcpan2• 7 years ago

This is pretty good and deserves more attention. Github makes bug studies so much more approachable. Hopefully the study will make people more cautious about Go's concurrency model, which is as error prone as shared memory multithreading, if not more.

weberc23• 7 years ago

What concurrency model do you prefer? Seems like it’s easy enough to opt into a share-nothing model in Go, and this is generally what I do unless I have performance concerns (and as a consequence, I very rarely see concurrency bugs). I’m also not sure how much static analysis a la Rust could help the problem without accepting Rust levels of productivity.

ricketycricket• 7 years ago

I wouldn't say I'm an expert in this area, but I do find Erlang/Elixir's actor model quite compelling.

pcwalton8• 7 years ago

The only foolproof way to opt into shared nothing in Go is not to spawn any goroutines. Global variables (which are all mutable, because there aren't any other kinds of variables in Go) are used all over the place in Go, including in the standard library.

deepsun• 7 years ago

I prefer ReactiveX, because you're still thinking in terms of data flows, parallelization comes as a bonus.

rurban• 7 years ago

I would not cite Rust as proper static concurrency analysis, but Pony. Pony is the only fast language (in use) with safe and proper static concurrency analysis and support for shared memory.

And there exist many other concurrency-safe slower languages with message-passing only, which don't have the Go problems. Go has at least the tools to detect deadlocks at run-time in testing, but these tools do not replace proper static type-checks for concurrency bugs.

littlestymaar2• 7 years ago

I’m also not sure how much static analysis a la Rust could help the problem without accepting Rust levels of productivity.

You're implying that Rust has productivity issues, which is not true. I'd say I'm as productive in Rust as I was in C# and more productive that I was in JavaScript. Productivity isn't a problem in Rust, the true issue is the level of involvement and effort needed to learn the language in the beginning.

tapirl• 7 years ago

is there a html version?

c612• 7 years ago

The two key takeaways for me are:

(1) "Contrary to the common belief that message passing is less error-prone, more blocking bugs in our studied Go applications are caused by wrong message passing than by wrong shared memory protection."

(2) "Shared memory synchronization operations are used more often than message passing, ..."

So basically, (1) people have more trouble writing correct multithreaded routines using message passing, and (2) in large production applications people tend to fall back to using shared-memory primitives rather than message-passing anyway.

It seems the intention of the Go language designers was to make a language that would be simple and easy for programmers to pick up. Given that most programmers are already accustomed to multithreaded programming using shared memory and that this is conceptually simpler, I think the language designers made a mistake by throwing channels, a relatively new and experimental approach (yes, I know CSP is from 1978, but I'm talking about widely-adopted industrial languages), into the language. I think it was the single biggest mistake in the design of Go.

tedunangst3• 7 years ago

Channels are appealing, but they're trickier than they seem. Especially when you're selecting on two channels waiting for one event to happen first but then both happen at the same time. Eventually you end up with special code in both cases to check whether the othe thing happened and try to stop it, and then what if you're late, ... The code is easy to write, but hard to make correct.

bb88• 7 years ago

I feel like Golang is turning out to be like C++ with all of it's gotchas. It's not that it didn't have good intentions, it's just they didn't nail the corner cases for the end coder.

There's a lot of features of C++ that would have been awesome, had it not been for those corner cases.

naikrovek2• 7 years ago

The ratio of gotchas in Go compared to C++ has to be on the order of 1:10,000

shereadsthenews• 7 years ago

It's like C++ but without metaprogramming, without a huge library of standard algorithms, and with much worse performance, yes.

coldtea3• 7 years ago

I feel like Golang is turning out to be like C++ with all of it's gotchas. It's not that it didn't have good intentions, it's just they didn't nail the corner cases for the end coder.

In many cases in Go they left footguns and open corner cases for the end coder in favor of an easier compiler implementation.

naikrovek2• 7 years ago

most programmers are already accustomed to multithreaded programming using shared memory

Aren't most programmers JavaScript programmers accustomed to single threaded applications passing variables around?

I've been programming for 20 years, as a career, and I've never once used shared memory in the way that I think when I think "shared memory". Maybe some languages I've used do things internally with shared memory, I don't know.

My point here is to be aware that the people you work with and know aren't necessarily representative of "most programmers" even though it feels that way to each of us. We expose ourselves to what we know more than what we don't know, and that influences how we each see the world.

atombender• 7 years ago

You've never needed to share a cache (like a map of ID -> object) across threads? Or implement a queue for distributing work to threaded workers? Or maintain a shared rate limiter across threads, to prevent saturation of some API or other shared resource? Those are all shared memory examples.

Avoiding sharing memory for concurrent workers is rare, in my experience, unless the workers are completely isolated -- which they rarely are. You can move some things outside (for example, Redis as a cache), at the expense of performance or simplicity.

c612• 7 years ago

My point here is to be aware that the people you work with and know aren't necessarily representative of "most programmers" even though it feels that way to each of us.

Definitely a good point and something we all should keep in mind.

With that being said:

Aren't most programmers JavaScript programmers

I would say no. Outside of Silicon Valley, most programmers are employed working on internal enterprise applications using backend languages like Java, C#, Ruby, and Python. Shared memory is the standard way multithreaded programming is done is all these languages. It is also the standard way multithreaded programming is done in C and C++ and therefore most of the serious software out there. I think this definitely covers "most programmers," or at least most application developers, the subset of programmers most likely to try to learn Go.

Someone• 7 years ago

”people have more trouble writing correct multithreaded routines using message passing”

The paper makes that claim for blocking bugs, but it also says ”Message passing causes less non-blocking bugs than shared memory synchronization”

Also, having more blocking bugs in message passing could be explained by users using message passing only for the harder cases. I don’t see that discussed in the paper, but ”Message passing […] was even used to fix bugs that are caused by wrong shared memory synchronization.” might hint at it.

Finally, programmers having less familiarity with message passing code might explain the difference in number of blocking bugs, rather than writing message passing code being more difficult.

MauranKilom• 7 years ago

How appropriate that the other thread covers this exact situation: https://news.ycombinator.com/item?id=19278839

ecnahc5152• 7 years ago

Channels I think make sense in more nuanced cases where you want to be design a more complex, but isolated concurrency problem. A good example is timers/tickers, which heavily lean on channels in their implementation. Same for things in the sync package.

I think using channels for every-day synchronization is often reached for too early when concurrency likely isn't even needed in the first place, and when it is, it's probably going to work well enough with shared memory until it becomes more complex.

tedunangst3• 7 years ago

Timers are kind of my go to for how channels quickly get difficult. Oh, it's just a channel? I can totally compose that with other channels in select. Go offers conditional variables, but not with timeouts. You want cond.Wait(timeout)? You're going to build it yourself with a timer and a wake channel. But now you've got the problem where you need to stop the timer. Except you might miss the race, so you have to remember to drain the channel. And your wakeup channel is more like a semaphore than a condvar. If you get multiple wakeups, you need to remember to drain that too. And unlike cond.Signal(), wakers can block on the channel send if you're not careful. So now you've decided to put the timer in its own goroutine, which will simply cond.Signal at intervals, but you still need to manage stopping and resetting the timer in the cond.wait() calling function. Edge cases abound if you want precise consistent results. The documentation for timer.Reset spends more words telling you how not to use it than what it does.

gtowey• 7 years ago

The one thing I would note is that most of the problems with channels seem to fall into the type where incorrect ordering causes a blocking channel operation that will never complete.

Channels are definitely hard things to get right in go consistently, and they can be a bit non-intuitive. They result in deadlocks/stalls and leaked goroutines. However what they don't result in is memory corruption issues -- none of the cases wind up reading or writing the wrong values. So in that sense I think you can say that concurrency fails with channels tend to be "safer" in that they will not cause operations to proceed with the wrong data. Which is one advantage over many cases of shared-memory concurrency bugs.

crimsonalucard4• 7 years ago

"Surprisingly, our study shows that it is as easy to make concurrency bugs with message passing as with shared memory, sometimes even more. For example, around 58% of blocking bugs are caused by message passing. In addition to the violation of Go’s channel usage rules (e.g., waiting on a channel that no one sends data to or close), many concurrency bugs are caused by the mixed usage of message passing and other new semantics and new libraries in Go, which can easily be overlooked but hard to detect."

jstewartmobile3• 7 years ago

On the one hand, they're absolutely right.

On the other hand, statistical methods don't give quality it's due.

I'd rather fight a blocking bug than a data-updated-via-race-condition bug.

crimsonalucard4• 7 years ago

As much as I don't like javascript, there is a whole swathe of concurrency bugs that exists in golang and not in nodejs.

Thaxll2• 7 years ago

Makes sense since Nodejs has no parallelism features.

beatgammit• 7 years ago

And that's a large reason why I use Go: scaling to multiple cores works out of the box if you build it with goroutines.

pcwalton8• 7 years ago

Go doesn't really have parallelism features either. A language designed for parallelism needs to have data-parallel features like parallel iterators, good support for SIMD, and concurrent data structures, neither of which Go has. (In fact, due to lack of generics, you can't even build your own generic concurrent data structures!)

merb2• 7 years ago

doesn't node have workers ?

crimsonalucard4• 7 years ago

Python threads don't either due to the GIL, yet they hit the same concurrency bugs.

It's the granularity of the concurrency.

crimsonalucard4• 7 years ago

Actually the event loop in nodejs uses several threads. There is parallelism. It's the granularity of the atomic operations that causes certain types of data race bugs.

In node, var x = 5; is atomic. In golang var x: Int = 5 is not atomic.

mirceal2• 7 years ago

confirms some of my suspicions to claims on how sexy and easy go is when it comes to concurrency. if anything it will give you a false a sense of security but god help you in case things go south and you’ve abused these features.

mirceal2• 7 years ago

reminded me of: https://twitter.com/pasiphae_goals/status/923835427995508741...

Animats• 7 years ago

Well, yes. Go's shared memory model is essentially the same as that of C and C++. You have locks, and you have data objects, and the language has no idea which lock covers what data. That's the big thing Rust got right.

Go's message passing model is somewhat error prone. It lets you pass references. So you can create shared data through the channels. It's easy to do this accidentally by passing slices, which are references to the underlying array, across a channel.

More generally, Go channels are one-way. Since you often want an answer back, you have to create some two-way mechanism from the raw one-way channels. It seems to be common to do this at the raw channel level, rather than using some "object" like encapsulation. Creating a mechanism which shuts down cleanly is somewhat tricky, too.

Understanding Real-World Concurrency Bugs in Go [pdf]

Related Stories