Ask HN: Best debugging war stories?

https://news.ycombinator.com/item?id=350174

by jacobscott • 18 years ago

4 4 18 years ago

news.ycombinator.com

I just finished spending two days with the Eclipse debugger and WireShark to fix a crash that would happen every 100th Spring RPC call I made. Fix turned out to be removing a single line of code.

This got me to thinking that HN probably has a good trove of debugging war stories. What are your favorites? Bonus points if they happened to you.

prakash• 18 years ago

http://www.google.com/search?hl=en&q=debugging+site%3Afo...

jlouis• 18 years ago

Sysadmin debugging: Many years ago I had the responsibility of maintaing a qmail setup for SMTP. We had around 1500 accounts and a mail-feed of some 80-90k mails a day. We were happily chewing off a 400mhz P-II with some 256Mb of RAM. Suddenly, the performance of the system began dropping. We only served 2 mails every 2-5 seconds. Tuning up the number of sending qmail processes did not help anything.

At this point, your mail spool begins to fill quickly. You can't process mails and get rid of them in the spool and with 80-90k mails a day combined with qmails primitive handling of bodies, your disk is going low quickly.

The key observation was that the qmail disk-log service used plenty of CPU-time. Restarting it didn't help. At this point, the next key observation is that most of the time is spent in the kernel. strace(1)! We are using blocking disk-writes to sync log changes to disk. Hmmm. It turned out that the log file was so big that many indirections in the inodes were needed. Qmail logs an insane number of lines per mail, so this combined with a hefty disk fragmentation killed the system. rotating the log file made the system jump to 20 processes sending again and we cleared the some 400k mails in the queue in a matter of 80 minutes.

kqr2• 18 years ago

Not my bug, but my favorite story is the Mars Pathfinder bug caused by priority inversion:

http://research.microsoft.com/~mbj/Mars_Pathfinder/Mars_Path...

antigravity• 18 years ago

http://catb.org/jargon/html/magic-story.html

Ask HN: Best debugging war stories?

Related Stories