The Magic CmdLine and how I got it back

http://users.softlab.ece.ntua.gr/~ttsiod/bashheimer.html

by ttsiodras • 12 years ago

165 76 12 years ago

users.softlab.ece.ntua.gr

metafex• 12 years ago

The tip about getting the binary from /proc is also helpful if you upgrade screen/tmux and the newer version doesn't allow you to reattach to it. With that you can use the old one to reattach, work your magic, and restart the session.

zilog805• 12 years ago

You just explained why I could not attach to an already existing screen last month - it all fits now...

jarin• 12 years ago

Er, why not just use the `history` command (or hit the up arrow)?

ttsiodras18• 12 years ago

Good question. Well, when you have forgotten everything about a 10 lines long command line you wrote 6 months ago, you don't know what side effects a signal may cause... You don't have the heart to signal anything, except surrender. Luckily, I found a workaround :-)

albertoleal• 12 years ago

While I'm unable to come with this on my own, the article was pretty easy to digest and understand.

It seems that the take-away lesson from the article is that you should save it in a bash script.

ttsiodras18• 12 years ago

Since the shell where I wrote the command is still executing, the .bash_history has no mention of it (bash saves the commands in the history file when it exits). As for the up arrow, it would mean interrupting the commands - and I didn't want to :-)

sp3323• 12 years ago

Just curious about one of the last steps:

  cat /proc/53165/exe > /tmp/oldBash

I would have reached for 'cp /proc/53165/exe /tmp/oldBash', instead of using cat. Is there an advantage to using cat here, or is it just the same?

ttsiodras18• 12 years ago

Indeed! No idea what went through my brain when I wrote that :-)

sp3323• 12 years ago

Hey, working code is the only justification you need :)

zokier• 12 years ago

Could you just use `gdb --pid 53165 /proc/53165/exe`?

emotionalcode• 12 years ago

I recently did exactly the same, it's those times when everything with similar functionality depending on what context you are in blurs together because you should probably be outside, taking a walk, or staring at a tree.

However, reaching for one way of doing things when the justification for the other way of doing things is because you've seen them done that way more, is not really good justification. So maybe that blurry brain is a good brain sometimes, in retrospect.

Someone• 12 years ago

I would have tried running 'strings'on the process's memory instead of gdb. Ugly, but likely would have worked with some manual intervention.

As to 'cp' vs 'cat': neither is a builtin, so they both are risky. What if you also accidentally did a rm -rf / on that system?

Mac OS X doesn't have a /proc filesystem, so I cannot test it, but from reading the man page

   read -N 1000000000 /proc/53165/exe $bash

should read all of bash into $bash, if it isn't ridiculously large, and

   echo -nE $bash > myBash

should then complete the copy.

michaelmior2• 12 years ago

Why are cp and cat "risky"?

dietrichepp• 12 years ago

    read -N 1000000000 /proc/53165/exe $bash

That shouldn't work, due to NUL bytes.

cssandjs2• 12 years ago

And cp would have also saved permissions (the executable bit). Not that GDB would have an issue without it, as the OP showed it can look for symbols inside an ELF regardless of the inode bits.

claar• 12 years ago

This was a fun read and great hack. Thanks for sharing.

ttsiodras18• 12 years ago

My pleasure! It was a lot of fun hunting all this down, I just had to share it :-)

PeterWhittaker• 12 years ago

You, sir, are hardcore. A steely eyed missile man, as they say. Awesome bit of forensic debuggery.

slowmovintarget• 12 years ago

It is indeed cool, but the single best lesson to come of it might be to start the work in a script in the first place. Still, cool beans. :)

_lce0• 12 years ago

the best I've read since a long time!!

kelvin0• 12 years ago

Cool tricks, but reads like a how-to of things not to do when writing any type of code and deploying it.

A Jedi? Maybe at getting himself out of the quicksand he lays himself into ...

walshemj• 12 years ago

wouldn't a simple Perl script and cron do the job I recall back I 2000 doing 20 line script in an afternoon that tied together one of the major exhibits at the millennium dome that my employer was sponsoring - the 3d scanner booth thing from memory.

gerner• 12 years ago

You're right, quicksand all around.

However, the post illustrates an important feature I often see overlooked: the systems we build on top of are not just black boxes. We should not be afraid to really understand how they work, opening up source code, asking questions like, "what the heck is this /proc/NUMBER section of the filesystem?" And I think this post illuminates that in a pretty fun way.

ttsiodras18• 12 years ago

Oh come on :-)

You're telling me that you've never - under tremendous pressure - hacked things that quickly "handle" something - and then forgotten about them till much later?

Jeez, tough crowd :-)

mst• 12 years ago

Those who do, blog. Those who don't, snark at those blog posts on HN.

andmarios• 12 years ago

Why not ctrl+c, then arrow up and enter? If the command can't be stopped even for a few milliseconds then there is something wrong with it.

ttsiodras18• 12 years ago

I didn't want to send any signal - because I could not remember what I had done, and was afraid for sideeffects (I did remember that the logic had to do with database actions).

I wanted to see the command without "messing" with the execution in any way - after all, it was running fine for 6 months, why interrupt it (and face potential effort in fixing messed up state) if I could see the command "from the outside"?

And that I did :-)

__david__2• 12 years ago

C-z is actually your friend here. I've never ever seen SIGSTOP affect anything negatively. I even SIGSTOP Mac OS X apps sometimes. They just beachball until you SIGCONT them, then they happily continue, oblivious.

If you're worried about timing out of the process being unresponsive, just C-z and then "bg" real quick. Then you can up arrow at your leisure and "fg" when you've copied the command away...

vhdlGuru• 12 years ago

I know a lot of programs that would have issues with Ctrl-C - I understand why the OP didn't want to do this.

kazinator2• 12 years ago

GDB can almost be a dynamic computing environment:

  $ gdb ./txr
  GNU gdb [ ... ]
  [ ... ]
  Reading symbols from /home/kaz/txr/txr...done.
  (gdb) b eval
  Breakpoint 1 at 0x808f6b0: file eval.c, line 969.
  (gdb) r -p '(+ 2 2)'
  Starting program: /home/kaz/txr/txr -p '(+ 2 2)'

  Breakpoint 1, eval (form=0xb7fb919c, env=0xb7fb92ac, ctx_form=0xb7fb919c)
      at eval.c:969
  969     {
  (gdb) p d(form)
  (+ 2 2)
  $1 = void
  (gdb) p d(cons(form, form))
  ((+ 2 2) + 2 2)
  $2 = void
  (gdb) p d(cdr(form))
  (2 2)
  $3 = void
  (gdb) p d(cons(0x1, 0x1))
  (0 . 0)
  $4 = void
  (gdb) p cons(0x1, 0x0)
  $5 = (obj_t *) 0xb7fb921c
  (gdb) p cons(0x9, $5)
  $6 = (obj_t *) 0xb7fb920c
  (gdb) p cons(0x19, $6)
  $7 = (obj_t *) 0xb7fb91fc
  (gdb) p d($7)
  (6 2 0)

You can test functions that are not even called anywhere from the C code. The d function is such a function; it's only there for use out of GDB.

  (gdb) p (0)
  $10 = 0
  (gdb) p d(0)
  nil
  $11 = void
  (gdb) p t
  $12 = (val) 0xb7fe5eac
  (gdb) p d(t)
  t
  $13 = void
  (gdb) p typeof(t)
  $14 = (obj_t *) 0xb7fe5d3c
  (gdb) p d(typeof(t))
  sym
  $15 = void
  (gdb) p d(symbol_name(t))
  "t"
  $16 = void
  (gdb) p d(symbol_name(0))
  "nil"
  $17 = void

Take out the trash, and show it:

  (gdb) p gc()
  $18 = void
  (gdb) p d($7)
  #<garbage: 0xb7fb91fc>

ttsiodras18• 12 years ago

Awesome, indeed.

Other classics: dumping the memory of an active process (check my GitHub utils repo, I have it there) and redirecting the standard output AFTER you've started a process.

kazinator2• 12 years ago

Ah, dumping memory. I once took the dump-and-restart code from GNU Emacs, and planted it into GNU Make. I was then able to read a large and complex Makefile (consisting of included sub-Makefiles throughout a large tree) which took 30 seconds to process, dump the image to disk (with make --dump), and then use the dump to run incremental builds which were able to kick off build recipes almost instantly!

zilog805• 12 years ago

And in combination with conditional breakpoints executing commands, you can have GDB "dance" any application to whatever tune you wish:

commands ... call d(cons(form)) call symbol_name(t) cont end

It really is a marvelous tool - unfortunately, people don't know anything except the basics (if that)

voltagex_• 12 years ago

Is there anywhere I can learn specifically about the commands you mentioned?

MDCore2• 12 years ago

Nice hack! I hope you learn to start save your notes/commands/snippets to a text file as you go.

ttsiodras18• 12 years ago

I do, indeed - and in Git repos, usually :-)

How that one got by me is a mystery - let's chalk it up to stress and pretend it never happened :-)

MDCore2• 12 years ago

Hehe. And to contradict myself, if you HAD written it all down you wouldn't have the cool story :)

vacri• 12 years ago

I saw things being saved to /tmp, and kept expecting a turn for the worse from a sudden reboot...

Fantastic article though - I've learned a couple of things from it, thank you.

ttsiodras18• 12 years ago

My pleasure :-)

bogomipz• 12 years ago

you're a jedi because you discovered the /proc file system? Right, OK.

ricket• 12 years ago

My mom has Alzheimer's disease. I either sympathize with your situation, or if you were using it as an expression, I find the misuse a bit offensive. Particularly in this context for me, it brings up a painful memory of one of the first signs of Alzheimer's in my mom, when she called me because she had entirely forgotten her computer password.

As a sidenote to anyone reading this, I really appreciate that the Hacker News community occasionally posts and upvotes Alzheimer's articles. I read every one that I see.

Anyway, I beg that you might not use "Alzheimer's" as an expression for forgetfulness, just as you might avoid calling someone "ADD" when they multitask to a fault.

mgraczyk• 12 years ago

It's worth noting that the author appears to be in Greece. Assuming he's a non-native speaker, it doesn't really make sense to be "offended" by his misuse of the term.

Maybe a better way to put it would be "We don't typically use 'Alzheimer's' to refer to forgetfulness because it brings up negative emotions for some people."

ttsiodras18• 12 years ago

It's considered 100% normal in everyday speech here in Greece, to use the name of the disease as an excuse when you forget something. I had absolutely no intent to offend anyone, and my heart goes out to the people fighting this disease - and the ones taking care of them.

I guess it's yet another cultural difference I was not aware of...

oxioxi• 12 years ago

Greek language is more about emotion than content. Hyperbole is used to emphasize meaning. I cannot forget he first time I went to Greece and my aunt kissed me, bit me, and said she was going to eat me. I was 6. I was petrified. It is common in Greece to express disappointment affectionately by saying ' I am going to kill you'. It is our cultural ignorance in the US that makes Americans easy targets for ridicule. If we project ourselves as a superpower to the world, then where indeed are our superpowers of understanding?

q3k• 12 years ago

On the other hand, the Internet is a multicultural and multinational community. It does make some sense to shed the typically American “right to not be offended” when you browse the Web.

Just as much as you ask for your culture to be respected, respect other cultures by acknowledging their more offensive and black humour and not taking everything personally.

ErikRogneby• 12 years ago

screen doesn't have history?

Aloha• 12 years ago

he didnt invoke it from within screen it looks like - it was invoked with the invocation of screen

keeperofdakeys• 12 years ago

It has a finite buffer, so I imagine 6 months worth of logs would have easily blown away the history of you running the command.

ttsiodras18• 12 years ago

Exactly.

zilog805• 12 years ago

You can configure it though, to be much larger than its default settings. Still, 6 months... he'd probably have to hack it as he did, no matter what :-)

eponeponepon2• 12 years ago

Great story... but I read through it hoping to see the fabled command in its entirety at the end! Any chance..? :)

ttsiodras18• 12 years ago

God, no :-)

Some people here are already rather mean in their comments - imagine what would happen if I revealed The Magic CmdLine (TM)...

No, that privilege is reserved for the Inner Circle :-)

But I'll give you a teaser... It involved a lot of ImageMagick and "pdfimages" and "zxing" and "tesseract" and "pdfjoin" invocations... and the uploaded files were PDFs with scanned barcoded pages.

eponeponepon2• 12 years ago

Aw, there's nothing worse than a mystery! ;)

(I can imagine the gory details tbh - it sounds like there is every chance I have had a very similar command for a pretty similar purpose running for a good long while now... except mine's in a VBScript..!)

hammerandtongs• 12 years ago

I appreciate the spirit and the knowledge that you wrote this blog post with BUT -

The extra few seconds to put this in a .sh file in the first place ... priceless. Even for the basic case of debugging it and getting it working.

I don't think in the 20-25 years I've been "doing" unix that I've ever done this (I doubt anyone that has worked with me considers me ocd fwiw but this is beyond the pale).

Behaviors like this lead to "magic" work environments and are just pants on head places to work at/with.

I would be really mad the first time I found you doing this and I'd look for a different work situation for one of us the second time :)

Thanks for the cool post.

krick• 12 years ago

Now this is both interesting and well written post indeed.

barrkel• 12 years ago

When I read the problem description, I decided to try and solve it myself.

I used gcore to dump the running bash process, then ran strings on the core dump, then grepped for likely command line stuff. 'while' is reasonable choice for a loop. There aren't that many strings in the dump file in any case, most of the sizeable ones are function definitions and environment variables.

Relying on gdb makes getting to the right data much more precise, but my approach took about 20 seconds.

The Magic CmdLine and how I got it back

Related Stories