How to kill an unresponsive SSH session

http://www.laszlo.nu/post/553591402/how-to-kill-an-unresponsive-ssh-session

by oarmstrong • 13 years ago

209 100 13 years ago

If you happen to be a few sessions deep, ~~ will send ~ to the next session along. A casual ~~~~~~~~~~~. or so later and everything is wonderful again!

oarmstrong2• 13 years ago

Thanks for the tip! I don't tend to nest sessions more than two deep but this could be useful for those who have deep nestings.

makomk• 13 years ago

Though you have to be nested really deeply to do ~~~~~~~~~~~. - unlike some other escaping schemes, it only requires one extra tilde for every layer of nesting rather than doubling each time.

hahainternet2• 13 years ago

The worst problem is trying to figure out just how many layers deep you are. I was being a little ridiculous though.

kps• 13 years ago

The Telnet Song from CACM, April 1984: http://delivery.acm.org/10.1145/1040000/1035691/p347-steele.... Plain text at http://alcor.concordia.ca/~smw/home/telnet_song.html and no doubt elsewhere.

(This does not refer to the Unix version of telnet, which defaults to ^] as the default escape character.)

oarmstrong2• 13 years ago

In case anyone is having difficulty with the font used on the page, the escape sequence is: newline followed by tilde (~) and then period (.).

Sharlin2• 13 years ago

The default escape character ~ does not work if the tilde key in your keyboard layout is a dead key [1], like it is in many European layouts. It can be changed via the EscapeChar config option or the -e command line parameter. It seems, though, that not just any old character is accepted - I tried to use §, which, in the Finnish layout, is in the same physical position as ~ is in the US version, but ssh complains about "bad escape character".

EDIT - I suppose it must be an ASCII character, which is not an entirely unreasonable requirement.

[1] http://en.wikipedia.org/wiki/Dead_key

lzm• 13 years ago

This must be why mosh's escape sequence never worked for me (ctrl-^ or ctrl-^.)

hansjorg• 13 years ago

Seems to be fixed (to use the same as SSH): https://github.com/keithw/mosh/issues/215

cpa• 13 years ago

I use a french mac layout and it works well. But you have to type ~ then press space, then press shift+; (because shift+; is .)

cnvogel2• 13 years ago

It works, you'll just have to type

  <tilde> <space> .

Sharlin2• 13 years ago

Weird - I did try that before writing the comment and it didn't work. May depend on the terminal or something.

rlvesco7• 13 years ago

You can change what the default escape character is in the ssh config file.

adaml_623• 13 years ago

Another useful trick to remember if you're using Putty and you ever accidentally hit Ctrl-S and find that you've frozen the terminal.

Just type Ctrl-Q and you will unfreeze the connection.

Credit due to: http://raamdev.com/2007/recovering-from-ctrls-in-putty/

ojbyrne2• 13 years ago

This is not news to anyone (in tech) over 40.

https://en.wikipedia.org/wiki/Software_flow_control

delinka2• 13 years ago

But the world is full of neophytes, re-inventors, etc. It's good to remind people periodically and educate the next generation.

emmelaich• 13 years ago

That wiki page doesn't mention tilde does it?

I know tilde escapes from cu(1) (part of uucp) http://www.delorie.com/gnu/docs/uucp/cu.1.html

But it may well have originated before then.

beachstartup2• 13 years ago

i'm turning 30 this year, have been dealing with this issue for 15 years, and still mini-panic for about 2 seconds whenever i hit ctrl-s by accident. it's good to read about.

chill out man.

cnvogel2• 13 years ago

If you frequently hit Ctrl-S by accident, and don't consider this a feature, you can turn off XON/XOFF flow control using

   $ stty -ixon -ixoff

then Ctrl-S/Ctrl-Q will be normal characters without side-effects in your terminal, you'll probably just see (depending on shell, terminal, ...) ^S on your screen when you hit it.

To turn it on again, use

   $ stty ixon ixoff

voltagex_5• 13 years ago

I feel like I should have learned this years ago.

harrytuttle2• 13 years ago

Thanks for that. In over 20 years I've never worked that one out :)

thangalin• 13 years ago

http://en.wikipedia.org/wiki/Control_character

A bit more history.

spindritf• 13 years ago

Or use mosh[1] on top of SSH and stop worrying about that stuff.

It works much better over high-latency links (mobile). It is not bothered by saturated links, tolerates IP changes and losing the underlying connection like when you suspend your laptop and take it elsewhere.

I now have mosh connect to several servers in tabs when I run gnome-terminal the first time, and only disconnect on reboot. I also run a mosh-capable Irssi Connectbot fork on the phone[2].

It's a massive improvement, fixing many of the little annoyances of ssh.

[1] http://mosh.mit.edu/

[2] http://dan.drown.org/android/mosh/

oijaf888• 13 years ago

Has anyone audited mosh's security model/encryption? Just curious since ssh is pretty tried and tested.

qznc• 13 years ago

Authentication and initial key exchange is via ssh, so nothing to audit here. Afterwards data is sent AES-encrypted, which relatively simple.

ReidZB2• 13 years ago

It uses ssh to set up the connection, so that setup is fine. The actual traffic is encrypted using AES-128-OCB, which is a great mode of operation.

I haven't digged too deeply into their security, but from what the main page says, it seems pretty trustworthy. Hopefully they're using the appropriate authenticated data, with packet numbers and so forth, to prevent silly things like replay and reordering attacks.

namarkiv• 13 years ago

It still lacks ssh agent forwarding.

scoates• 13 years ago

…and performs scrollback very poorly.

I still use it, though. Can't wait for them to iron out these issues.

stormbrew2• 13 years ago

The server I connect to most often is behind NAT and I'm usually also behind NAT, and mosh doesn't support that config, alas. I'd use it in a heartbeat if I could, though.

https://github.com/keithw/mosh/issues/48

Freaky2• 13 years ago

I love mosh, but do note it's a lot slower than plain ssh. I run everything through a 266x188 tmux session, and mosh badly struggles to keep up even with light use. Resizing panes is juddery, vim is visibly laggy, and if anything spams a lot of output to a term (e.g. find /) it basically grinds to a halt. ssh is butter smooth in comparison.

No mouse support in mosh either.

(gnome-terminal/vte (so also Terminator, Xfce Terminal and others) is similarly slow, but as with mosh you don't really notice with smaller terms. I switched to urxvt).

gbog• 13 years ago

Does it work too with ctrl-\ ? This has been my process killer recently and it's powerful (and the only way I know to get out of xtail.)

Nursie• 13 years ago

enter-tilde-dot

It is useful, yes. Here's another thing I picked up last week - how do you reboot a remote linux box that's somehow lost its root drive but you still have a shell open (because you left ssh running on another machine)?

  echo 1 > /proc/sys/kernel/sysrq
  echo b > /proc/sysrq-trigger

kdazzle• 13 years ago

Another great solution is to just use the ServerAliveInterval option.

XorNot• 13 years ago

I can never decide how I want to set this. Most sites recommend 180 seconds or so, but with the default ServerAliveCountMax set to 3, this is 9 minutes before a dead terminal is actually disconnected.

I've started to set it really really low personally - like, 5 seconds, so the connection drops after 15 seconds. I'm tempted to go down to 1, but I have a lot of long running sessions and I start to worry about the traffic counts.

But, after all this there's still a problem: none of it seems to work with any of the connection mux'ing options - once the background session dies, I still have to manually kill it to get anything working again.

epo• 13 years ago

Please don't let HN become a substitute for RTFM. This should be known by all SSH users who have skimmed the man page. Fair enough as a blog post but for this trivia to get 46 points so far is deeply depressing.

Maybe I should write a blog post about the use of CTRL-Z in the shell and post that here, should get me Kilo-karma points if this is anything to go by.

peterwwillis2• 13 years ago

It should also be known by anyone who has used Telnet, since that's where it originated. (For more shortcuts, use '~?' from the ssh session)

__david__• 13 years ago

Not any telnet I've ever used. Telnet is ^] to get to the command prompt and then "quit". It even says so right when you connect:

    $ telnet google.com 80
    Trying 74.125.239.103...
    Connected to google.com.
    Escape character is '^]'.

donutdan4114• 13 years ago

Not everyone on HN is a terminal expert, and not sure why you would assume such. It got a lot of points because people found this interesting and helpful. Also, you should write a blog post about CTRL-Z, if people find the info valuable, it'll get the upvotes.

davidw3• 13 years ago

When you have a toolset that includes, but is not limited to:

* Ruby

* Rails

* Postgres

* C

* Erlang

* Emacs

* Bash/Zsh/whatever

* Linux Kernel

* GNU C library

* Postfix

and on and on and on, these kinds of tricks are bound to be useful to someone who is not a complete expert with whichever system is being discussed.

Also, realistically speaking, they are far less prevalent than nakedly political articles lately, which do weight on the quality of the site.

hhw9• 13 years ago

This is like not knowing the return or exit function, and is also used for telnet or serial console connections. If knowledge is so shallow on a particular tool, it should not count as being part of the toolset. There's nothing more basic or essential to SSH than connecting and disconnecting. And it's right there on the man page, not exactly a hidden feature. This is more beginner level knowledge, maybe novice if being very very generous, but nowhere near expert level.

srean2• 13 years ago

Do so if you wish to. Interested to know what you intend to do with that kilo, or how someone else earning a few hurts your cause.

prodigal_erik• 13 years ago

If it becomes acceptable to use HN for gamified man pages instead of expecting people to do their homework, well, there are more than enough man pages in and out of print to fill HN for years. We don't have downvotes on stories, so the only flow control is flagging and complaining about stuff that isn't timely and isn't useful for most of the audience.

jerf• 13 years ago

Read the SSH man pages every so often, even if you think you know how to use it. There's a lot of features in there. Don't miss the "AUTHORIZED_KEYS FILE FORMAT" in sshd's man page for the uber-cool "command='command'" options for authorized keys (restricts a given key to just be able to run a certain command, very useful). See also SSH's port forwarding, -D, learn how to use ssh-agent, and "man ssh_config".

voltagex_5• 13 years ago

the command= syntax is (was?) how gitolite works - I'm not sure what the difference between that and a jail is, though.

stormbrew2• 13 years ago

They're not really related concepts. A jail is allows you to run arbitrary things in a restricted environment while ssh's command feature just forces a particular command to run no matter what's passed.

If your authorized_keys line for a key has command="blah" on it and you go:

    ssh server ls

the server will, rather than running ls as it normally would, run blah and pass the command you specified in as an environment variable. Then blah can parse it and do whatever it wants with it.

You could use command to establish a jail or virtual environment of some sort in which to run the program specified if you wanted to.

MrDOS• 13 years ago

A true BSD chroot jail restricts the user's mobility around the file system, effectively limiting them to running only what binaries have been explicitly placed within that jail. It means, among other things, that the fork hacks that command= is vulnerable to (e.g., vim running other programs) don't work because non-permitted programs basically don't exist as far as the user can see. The downside to jails is that, without automation, they're tedious to maintain, and disk overhead is not insignificant as you have to have a duplicate set of system binaries for each user. In practice, chroot jails end up being very similar to OpenVZ “virtualization” on Linux.

txutxu• 13 years ago

The command= simply changes "what" gets executed when the user logs in.

A jail (ChrootDirectory in ssh) changes "where" the user gets when logs in.

The most creative thing I've do with command= was a "select" menu in bash (with some actions in the shudoers).

Other interesting tool I've discovered recently is rrsync. I'm doing the backups of my systems isolated with this. It's distributed with the rsync sources, you put it like:

    command="rrsync /path/to/chroot/the/remote/rsync/client/"

jvoorhis2• 13 years ago

They work in totally different spheres. The command directive can assign different commands per ssh key. Think git@example.com for all of your users, a la Github.

Jails limit the capabilities of users of the OS, instead of users of the network.

kbenson• 13 years ago

While I've frequently used this to kill connections, my favorite thing I've done with it is to list existing and dynamically add new forwarding ports through SSH.

tankenmate• 13 years ago

The thing that amazes me about this is that people don't realise that this comes from BSD 4.2 rsh released in 1983.

revscat• 13 years ago

Why does that amaze you? That is a pretty trivial piece of knowledge. Interesting, though.

verbatim• 13 years ago

<enter>~Ctrl-Z will suspend the ssh session, too.

I've also found it useful to do <enter>~C - then you can configure port forwarding without having to open a new ssh session.

(~C opens a command line, enter "help" for available commands.)

microcolonel• 13 years ago

I'm surprised/apalled at how many people upvote this, considering how this place is supposed to be "hacker news"...

lobo_tuerto• 13 years ago

Sometimes I upvote articles where the comments are good, present new information or just have better information than the article itself.

badmadrad• 13 years ago

you really don't HAVE to know this to be a good "hacker"

That being said, some people are new to the game. No reason to be elitist.

spudlyo• 13 years ago

Hitting '.' at a prompt used to be a common idiom for exiting a program. I first saw it when I was a kid working on an HP-3000 system where the system programming language was BASIC and all programs followed this convention. Don't know where it came from originally, but you can still see it in places like rsh/SSH etc.

smutticus• 13 years ago

And SMTP where a single "." followed by CR ends the session.

jlkinsel• 13 years ago

I'm a little surprised this is on HN? To me this is the equivalent of a blog post about using %d with printf.

Not complaining, just a little surprised something so novice would get attention...

recursive• 13 years ago

You should post it if you think it will gain traction. Not everyone is experienced with C.

bostonvaulter2• 13 years ago

This usually doesn't work for me. Perhaps it's because my sessions are usually multiplexed via "ControlMaster auto"?

Nick_C• 13 years ago

On a side note, does anyone know what ~B actually does? Does it send a SIGINT to the remote terminal? What does ssh mean by the phrase "send a BREAK to a remote system"?

I've tried to use it without success to kill a runaway listing of megabytes of scrolling text, but frantically hitting ctrl-C seems to work much better.

thristian• 13 years ago

In the beginning, there was RS232, where each character was a fixed number of bits, optionally with a stop bit and a parity bit. If the sender transmitted too many 0 bits to represent a valid character, that protocol failure was called a 'break', and some receiving equipment would detect such a failure and do something about it, like reset itself to a known-good state. Thus, a 'break' was occasionally a useful thing to send, so sending equipment would often have a special keystroke to cause a 'break' condition.

The Unix 'tty' subsystem was basically designed to support simple serial terminals, and so it had a bunch of behaviour designed to interoperate with the pre-existing 'break' conventions. If a Unix system's serial port received a break, Unix would (optionally) send any processes running via that connection a SIGINT, to represent the 'reset to known-good state' behaviour (this is controlled by the stty command's 'brkint' flag). Also, if the user's terminal didn't provide a specific 'send a break signal' command, Unix could be configured to send a break signal when it received some particular character (^C by default; this is controlled by stty's 'intr' setting).

Of course, nobody uses physical RS232 terminals anymore, but for compatibility reasons the Unix tty API lives on, and the "psuedotty" implementation used for things like terminal emulators maintains compatibility. ssh is basically a tool for exporting the tty API over the network, and so for compatibility it too must have a way to transmit the information "pretend a break condition has occurred on the RS232 connection we're pretending to use."

To summarise: yes, unless you've messed with the stty command, ~B will probably result in a SIGINT. Ctrl-C is probably more reliable since you can hit it much faster than you can type <Enter>~B, but ~B is still useful if your terminal is in 'raw' mode, where ^C is not converted to SIGINT (for example, if you're running an app that wants to bind ^C to some other function).

How to kill an unresponsive SSH session

Related Stories