Go has a debugger and it's awesome

https://blog.cloudflare.com/go-has-a-debugger-and-its-awesome/

by FiloSottile • 11 years ago

124 69 11 years ago

blog.cloudflare.com

wumbernang• 11 years ago

No no no no. This is not right.

1. If you debug with this, you're debugging different code to what blew up in production for example. Subtle timing issues wiped out instantly and memory ballooning masked etc.

2. If you leave it in for prod, it's going to have masses of call overhead.

You can debug well with assertions, logging and unit tests. There is no need for this. I rarely have to spin up a debugger these days.

danparsonson2• 11 years ago

Indeed, it's like they've never heard of the term 'Heisenbug'

gohrt• 11 years ago

It's like they know that not all bugs are Heisenbugs.

alfonsodev• 11 years ago

I use de Node.js debugger a lot, I think is very useful when you don't know exactly what to log, and you don't what to start adding logging lines everywhere. You have one assumption but if you are wrong you can check different var, within the debugger step by step. Sometimes I even use the debugger inside a unit test.

simi_• 11 years ago

The last time I used a debugger was when with VS/C# or IntelliJ/Java, many years ago. For the coding I do now, mostly Go/Python/JS, I find manual debugging techniques much more effective.

0xdeadbeefbabe3• 11 years ago

No no no no. This is not right.

It's as worthwhile as the bugs they can find with it. I wish the author would tell us if he found the bug in the dns program.

thoman23• 11 years ago

"Go has a debugger"

The very fact that that is a headline make me think I'll wait a bit before introducing Go into my tech stack.

timclark• 11 years ago

You can use gdb - https://golang.org/doc/gdb

codingvelocity• 11 years ago

It's largely accepted that while you can use gdb, it's not really good at debugging go code. from your own link.. "GDB does not understand Go programs well"

https://github.com/derekparker/delve delve as mentioned other places is more of a true go debugger

4ad3• 11 years ago

You can also use mdb[1] though mdb is somewhat more suited to debugging the Go runtime itself rather then user programs.

Mdb support for Go will be significantly improved in the short term.

Also note that since Go has (optional) frame pointers, tools like DTrace, ktap, kprobes and uprobes, etc work much better (almost as well as for C). SystemTap works too, just that it chokes on Go DWARF without a patch. I will probably fix this soon.

[1] https://www.joyent.com/blog/mdb-support-for-go

_ak2• 11 years ago

I think you should ask yourself how all these tech companies that use Go in production ever reached the confidence to use Go in production without a "proper" debugger available.

pjmlp• 11 years ago

It is made by Google, and has Plan9 guys on the team, what else do you need?!

zak_mc_kracken3• 11 years ago

Replace "Go" with "PHP" and ask yourself if such a claim is a strong validation of a technology...'

Yes, a lot of people have gone to production with crappy technologies, this is meaningless. The real question is: could they have done it more easily using a better technology?

jlarocco• 11 years ago

I'm not sure that's a very good argument.

50 years ago there were a ton of companies that were confidently using hex machine code for large systems in production. That doesn't mean things wouldn't have been 100x easier for them with higher level languages and more modern tools.

Of course I can get by without a debugger if I really need to, but that doesn't mean I won't be a lot more productive with one.

PopsiclePete• 11 years ago

If you can't write code without a debugger, then absolutely, wait. I'd straight-up refuse to write C#/Java without a debugger as well - too many layers and obfuscation.

Go, however, is kind of like a memory-safe C...it's almost impossible to actually create super high-level abstractions where you can't tell what the code is doing.

For me, unit tests and the occasional log/print cover 99% of my Go debugging needs.

metafex• 11 years ago

That's no debugger. That's a debugger: https://github.com/derekparker/delve

If you want to attach to running processes, use delve, it's awesome. For me personally, even with Go's short build times, you can't just deploy a new binary into a running system to see where a problem lies. Sometimes there is just no way around a traditional debugger.

SixSigma• 11 years ago

No, this is a debugger

http://plan9.bell-labs.com/sys/doc/acidpaper.html

or on Unix

https://swtch.com/plan9port/man/man1/acid.html

Watch it in action

https://www.youtube.com/watch?v=ZlGFdflPcFo

4ad3• 11 years ago

Acid is great, the best debugger I know, unfortunately the Unix port in plan9port is completely non-functional.

Go is really annoying to debug in acid though, because symbols contain dot characters. Somewhat ironically this was done because Unix debuggers were getting confused by the unicode middle dot character, so the unicode middle dot is replaced in the symbol and debug tables with a regular dot.

But on Plan 9 we should not do this process and we should keep the original middle dot.

0xdeadbeefbabe3• 11 years ago

cool, it reminds me os DOS's debug.

flippinburgers• 11 years ago

I fundamentally dislike this approach to solving the problem. It feels very crude.

shawnps• 11 years ago

I also use a well-known debugger called fmt.Println

bcantrill6• 11 years ago

As someone who has dedicated their career to understanding and debugging software in production[1], it's hard to not be depressed about this -- not only because it (emphatically) isn't "awesome", but also because of the failure to couch this in its proper context: this is -- at best -- a kludge to enable small programs to be debugged in development. While this is fine at some level (debugging is, after all, a good thing), small programs in development aren't where the most pernicious bugs are found; sophisticated tooling should (in my opinion) always focus on the production systems in which the nastiest, most undebuggable problems arise.

Note that this needn't mean that the tools themselves run in production: for example, Valgrind is incredibly useful because it finds really nasty, production-grade issues -- even if it is only meaningfully usable in development. Similarly, postmortem debugging tooling is extraordinarily valuable for production problems -- and requires no tooling in the production environment itself. We have developed such tooling for node.js[2], and found it to be invaluable. There is nascent postmortem debugging support for Go[3], but there is much more to be done -- just hoping that some in the Go community see the value of debugging production problems!

[1] http://www.infoq.com/presentations/Debugging-Production-Syst...

[2] http://www.slideshare.net/bcantrill/node-summit2013

[3] http://www.joyent.com/blog/mdb-support-for-go

4ad3• 11 years ago

just hoping that some in the Go community see the value of debugging production problems

Some of us do. As I said in another post, Solaris support will only get better. As for Linux, some people obviously care, Go had some form of gdb support from quite early on. Unfortunately, it's not that people don't care about making the gdb support better, it's just that this is all that can be done with the gdb-python interface right now, due to the limitations of gdb...

At least, now with frame pointers, dynamic instrumentation works better.

vruiz• 11 years ago

just hoping that some in the Go community see the value of debugging production problems!

I just can not remember where I heard it (some podcast maybe), but is my understanding that the Go Team hinted the possibility of something like JMX coming to the Go runtime.

FiloSottile• 11 years ago

Author here. I noticed before both mdb support and the Delve project that other have mentioned, and I find them great. The point of my excitement for godebug resides in the fact that it's an easy to work with, zero complexity added tool to replace most tedious println() debugging sessions. That is, the non-exciting stuff.

Not all bugs only happen on production load, architecture, etc. Many times (with DNS servers even more so, I'd say) bugs are simple state or logic bugs. The Heisenbugs exist. The complex resource-constrained races exist. And I find it fun to track them down, but many bugs are much more mundane, and godebug is a great tool for catching those.

Speaking instead about hard to trace production bugs, I think the Go design and library deserve their daily share of praise here, since they solve many of those problems. Take Valgrind (or ASAN). Amazing tools. Completely useless in Go -- the language is memory-safe and will make that kind of bugs both hard to introduce and immediately detectable.

And about concurrency, first of all the simple design makes bugs harder to introduce and easier to reason about. But even when they make their way in I find provisions like net/http/pprof and its various profile a pleasure to work with. And finally there's obviously the race detector.

Still, I can't wait to see more tools developed to cover different Go debugging needs. Today I'm just happy about making my println() session much more pleasant and elegant :)

bcantrill6• 11 years ago

The enthusiasm is fine -- I just wish it were couched in the understanding of the limitations of the technique. I also think it's naive to say that Go's design solves "many [production] problems" -- memory-safe languages solve one set of problems, but often introduce or exacerbate another. (The most obvious example being memory exhaustion: postmortem heap analysis is often much easier on a C-based system than in memory-safe languages.[1][2]) More generally, do not fall into the trap of correctness from smugness: nasty production bugs can exist in any language, and time is much better spent on the production tooling to debug such problems rather than simply asserting that such bugs are tautologically impossible.

[1] http://dtrace.org/blogs/ahl/2004/07/13/number-11-of-20-libum...

[2] https://queue.acm.org/detail.cfm?id=2039361

Edit: added a much better reference.

pron2• 11 years ago

See also Debugging at Full Speed http://medianetwork.oracle.com/video/player/3731019771001

rayiner2• 11 years ago

Debugging software in production is a hard unsolved problem, the available tools are pretty primitive, and it doesn't help that people often don't plan it into their development and deployment cycle.

We had a client once using our system, who would see anomalies only after long uptimes at sustained high loads. It took six months to find the cause (a race condition in the database code) because we had so few tools at our disposal to look at the post-mortem state of the program. Running the software at those loads for the weeks or months it took to reproduce the problem under a debugger was obviously not a tenable proposition, and reproducing the variability of the particular real-world traffic was difficult to do in a controlled environment. And the software wasn't written with any eye towards making state available for post-mortem debugging.

I guess "don't write complicated multi-threaded software in C" is a pretty good response to the above too. That particular bug couldn't have happened in say Rust (though it could have in Java).

TylerE2• 11 years ago

In languages like C it's an unsolved problem. Doesn't mean it's an unsolved problem everywhere (see, e.g. Erlang, which even allows for hot patching without even taking the system down).

mwcampbell3• 11 years ago

In the QCon talk you referenced in [1], you said that it's impossible to debug a Python or Ruby application from a core file. Why is this? Would an mdb module for Python or Ruby, to delve into the interpreter state and expose it in the debugger, not help or not be feasible?

bcantrill6• 11 years ago

"Impossible" is a bit strong, but yes, it requires deep and unholy knowledge of the interpreter in the debugger. For an example of what this looks like, see some of the monster block comments in mdb_v8.c[1].

[1] https://github.com/joyent/illumos-joyent/blob/master/usr/src...

WalterBright• 11 years ago

My not-so-humble opinion is that valgrind has saved C (and C++) from oblivion, at least for large scale programs.

zamalek2• 11 years ago

postmortem debugging tooling is extraordinarily valuable for production problems

When all else fails this is really all you are going to have to go by. Coming from the Microsoft side of the story I have come to a conclusion that any sufficiently well designed postmortem debugger is a few steps away from being a live/production debugger. Windbg, as a prime example of this, can be used to debug memory dumps, live processes and remote (TCP) processes using exactly the same debugging tools binaries (and scripts/plugins that you write).

Debuggers have to be the biggest source of friction, for me, when adopting things like Go. If you are in charge of "production" you can throw in a few printlns, however, you can't just approach someone like the HP and say "can I put this debug binary on your production system?" No, the absolute best you can hope for is a memory dump and those memory dump had better count as you are only going to get maybe a few.

Maybe it's because not many devs deal with this kind of stuff, which is probably why it isn't a top priority for things like e.g. Go. To some of us, though, the most important language of a feature is easily the quality of the debugger associated with it.

Either way, great slides Bryan - it's nice to see at least one "modern" language taking these issues seriously.

pstuart• 11 years ago

Wasn't ogle supposed to be the official debugger?

https://godoc.org/golang.org/x/debug/ogle/program

tenfingers2• 11 years ago

A debugger that cannot be attached to a living process, that cannot change or execute data dynamically, is not a debugger. Unfortunately, many languages provide "debuggers" that fall into that category.

istvan__2• 11 years ago

Well I guess the Go community is a little bit too passionate about any update regarding Go, even if the introduced new thing is subpar compare to other languages, like this one in the post.

tenfingers2• 11 years ago

I think that the main issue is that debuggers are often dismissed in terms of importance, and many don't see the true value of debugging beyond simple step-by-step execution.

This is not unique to Go: python's several debuggers require cooperation with the inspected process. This is exactly the opposite of a buggy situation. gdb's python support is very limited in terms of debugging capabilities, yet it's the only option you have if you're using an external event loop handler, such as gtk/qt/glib (not so uncommon). I guess I should be grateful I have those at least.

alkonaut2• 11 years ago

While I understand that this is a fairly young language, it still makes me sad that we seem to erode what's actually cool, what's a good language, what good tools are. It took us two decades to massage the "last" generation of tools, the managed/OO ones, into shape. I've been whining about javas lack of value types, or that java tools didn't allow editing and reloading the debugged code in-place. I have whined for years about how the C# debugger doesn't allow edit-and-continue in 64bit, or how watch expressions can't have lambdas in them.

Meanwhile, outside my windows, it seems the world pretty much just rejected types, debuggers...

"You can do println debugging and generics will just make the language too hard to learn and the runtime too complex".

It used to be the other way around. I was young and the old guys used to say "I'll use my punchcard/command line debugger/whatever, you kids can use your fancy IDE all you want".

Now I sound like a dinosaur for wanting to edit my code in place while debugging, and for being whiny about type systems. What the hell happened?

Also, get off my lawn.

Someone1234• 11 years ago

C# debugger doesn't allow edit-and-continue in 64bit

They added that two years ago:

http://blogs.msdn.com/b/visualstudioalm/archive/2013/06/26/d...

or how watch expressions can't have lambdas in them.

That's a limitation on how lambdas work. Doubt it could be solved.

stcredzero5• 11 years ago

That's a limitation on how lambdas work. Doubt it could be solved.

AFAIK, this wasn't a problem with Smalltalk. Watch expressions would have worked in somewhat the same way as conditional breakpoints, and you couldn't have those without allowing Blocks, which are the equivalent of Smalltalk lambdas.

This may be a problem with the CLR, though.

alkonaut2• 11 years ago

They add watch lambdas in vs2015. Maybe Roslyn enabled it, not sure.

zamalek2• 11 years ago

Doubt it could be solved.

Solved in VS 2015.[1]

[1] http://blogs.msdn.com/b/visualstudioalm/archive/2014/11/12/s...

stcredzero5• 11 years ago

There is something to be said for ease of distribution and deployment, as well as playing well with the larger infrastructure/software world. Smalltalk, which was one of the major progenitors of the managed environments you were talking about, didn't do the above. I think that had a lot to do with its downfall.

When your system interacts well with the larger world, this results in more "commerce" with the larger world, which tends to result in greater recognition and larger and healthier communities. It's a virtuous cycle.

Having the best, most well developed developer tools is a win for the individual developer, but fitting in well with the larger world is a win for the entire developer community. The latter is going to be 10X more visible than the former. (Also, think about which communities manage to have both!)

Also, get off my lawn.

Yeah, kids, get off my lawn! At least do enough research to know what's gone before! (Re: Alan Kay's quip about "almost a field.")

blub• 11 years ago

Fair enough, but I don't think you addressed the parent's concern.

Most mainstream programming languages fit in well with the larger world, that's why a lot of people use them. They also have working debuggers, a ton of libraries and tools.

dsl• 11 years ago

Could a mod please update this to point to the original announcement and not the CloudFlare regurgitation? The Mailgun folks deserve the love for their hard work.

http://blog.mailgun.com/introducing-a-new-cross-platform-deb...

dang• 11 years ago

I was just looking into that. Normally we would do so, but that post already had a major thread: https://news.ycombinator.com/item?id=9409423.

stcredzero5• 11 years ago

Before closing, I'd like to say a few words about the technique of source rewriting in general. It powers many different Go tools, like test coverage, fuzzing and, indeed, debugging. It's made possible primarily by Go’s blazing-fast compiles, and it enables amazing cross-platform tools to be built easily.

Source rewriting is indeed quite powerful. You can use this technique to write a code coverage tool in Smalltalk. (Grab the source code of a method, apply parser-transformations, compile an instrumented version, swap the bytecode of the original method with the instrumented one. Make the instrumented method know how to swap itself back when everything is triggered.) A typical developer can write such a tool and have it become functional in one or two days. An exceptional one can do it in hours.

Iterating more produces more innovative applications and software tools. This is why fast compiles and parser tools are a winning combination.

Go has a debugger and it's awesome

Related Stories