It's all fun and games until someone [XOFF]

http://rachelbythebay.com/w/2015/06/20/blocking/

by luu • 11 years ago

150 50 11 years ago

phil21• 11 years ago

Nice!

This is absolutely one of those things I've had happen once or twice a year, poke around in puzzlement for 5 minutes, and then just restart and get on with my day. You knew it was a keystroke but never quite followed through on tracking down what/how to fix. Quite nice to know what the hell was going on all this time :)

krylon• 11 years ago

I have been there... The problem is that this happens to just rarely enough I don't remember ^Q and randomly poke at the keyboard for a while until I hit the right combination.

fit2rule5• 11 years ago

Remember that holding control and any alphabet letter is supposed to have a function, and then remember "Quick" and "Stop".

Probably you've used Control-C a few times, maybe Control-Z too .. well, these basic characters have a control function over your process' terminal input/output, just like ^M/^N (carriage return/line-feed), ^L (clear screen) and so on .. Just like Control-C is a shortcut for "Cancel" process.. C-S-Q is sort of the 'gearing' over process output, where Z, which is 'detach process i/o from console', otherwise known as "background the process", ( * ) might be considered the 'clutch'.

Anyway, C-S-Q .. those 4 keys control the process. Cancel, Stop, Quick.

Practice it a few times with some cat /somelargefile.txt paging and you'll see there's a use for it beyond perplexity. When 'less' and even 'more' are not around, Control-S/-Q are your trusty pals.

(* - Another "oh, fudge, dunno what happened, guess I'll just reboot.." moment can occur with errant Control-Z usage over a process, where you accidentally put the process in the background and don't know how to re-attach it. In that case, the command 'fg' is your friend .. I've see far too many experienced people not know they can put their processes back in place during debug/admin sessions, opting indeed to reboot, or at least restart the term session, that I can't help but feel that, like Stop/Quick, Control-Z+'fg' is somewhat of a lost lore..)

bostik• 11 years ago

Well that's interesting. There are two nearly identical keyboard mishaps that can wedge a screened process, and they both require ^Q to recover.

I've always attributed these hangs to the sad fact that hitting ^A can fairly easily come off as "^A ^S", especially when you do the keyboard hit with a slightly rolling hand motion. And as that key combination means "send ^S to the process inside screen", it results the inner process receiving SIGSTOP. Of course, the recovery is nearly the same as in the article: ^A ^Q.

For a screen user both have the same cargo-cult solution - just send enough SIGCONTs to the screen to get the application responding again.

I suspect that tmux chose ^B as the default command key to prevent accidental process hangs.

smosher_• 11 years ago

^A in bash will go to the beginning of the line. I don't know if this is why tmux changed the key, but it certainly makes a bigger difference to me because I use that function a lot. I rarely press ^S by accident (less often than I press it on purpose.)

jon-wood• 11 years ago

It's a nice theory, but I seem to remember ^B being Tmux's escape key because it was developed within a Screen session.

tonyarkles• 11 years ago

Either way, ^A is supposed to be "go to the start of the line" and it drove me absolutely nuts when I was using screen and had to ^A^A. There was probably a way to remap that, but I was introduced to tmux via byobu and now just use straight tmux on everything (works great on OSX too!)

yuubi• 11 years ago

^S and ^Q don't send signals; they make the tty driver stop delivering output. If you ^S a process that's not generating output, it will continue to run until it generates some output (until it fills an output buffer? not sure; the original intent, to match a fast process with a slow output device, would be well served by buffering some output).

SIGSTOP is like SIGTSTP, the signal usually generated by ^Z, except (like SIGKILL, signal number 9) it's not possible to catch it. It makes the process stop doing anything at all until something sends SIGCONT (which shows up in ps output as state T).

shabble• 11 years ago

And that's why I've had '/bin/stty -ixon' in my shell init scripts for a long time now.

"But the docs say that C-r searches backwards through history and C-s is forward"...test....um.

joshu2• 11 years ago

I'm using an Arduino running GRBL for a project. It can take up to ~128 bytes of gcode in its buffer, and anything more gets dropped. So you have to carefully shepherd new statements in as they are executed (scanning for "ok".) Would be a perfect use for flow control, if arduino didn't already use it for something else. Dammit.

ambrop7• 11 years ago

I'm not sure if you're already doing this, or if it's relevant to you, but a possible way to improve performance is to allow a "window" of commands that you haven't received an "ok" for. I believe common host software for RepRaps just allows one command at a time. How to define the window correctly depends on how the firmware does the buffering (in terms of bytes, or complete commands, or both).

joshu2• 11 years ago

If you read what I wrote a little more carefully, you will see that I already understood that.

yunesj• 11 years ago

If you didn't know what was going on behind the scenes, it might just seem like "one of those things that happens sometimes".

Yeah, I'd be like, $!@*$%# Ubuntu :-)

Permit• 11 years ago

When using Putty to remote into Linux machines this would happen to me ALL OF THE TIME. I'd be happily chugging along, editing some text in vi and my Windows instincts would kick in and I'd hit Ctr-S to save. Everything would freeze and I'd be sitting there wondering what in the hell had happened...

Apparently this is a fairly common problem. Take a look at: https://raam.org/2007/recovering-from-ctrls-in-putty/ The scrollbar should give some indication of how many "Thank You's" this article warranted!

Too• 11 years ago

This reminds me of a problem we had with running processes from python with subprocess.POpen. Some times after running the program for some time it just froze, none of us could figure out why and we reviewed the source of the invoked program and debugged it over and over again without finding anything that could get locked up. Eventually we found out that the problem was not in the invoked program, it was within python; piping the stdout into POpen was done to a fixed size buffer that was only being read/cleared after the invoked program exits, and since it can't continue to run it will never exit, stdout was getting blocked so just by writing to it the program froze. The solution to the problem was to start a new thread that kept reading the piped stdout and move it to another buffer. Today the documentation has a big warning about this behavior and a convenience-wrapper, communicate(), that does the threaded read for you but back then figuring this out was not exactly obvious.

d0mine• 11 years ago

Just to be clear, the issue is not in Python: the deadlock will happen if you won't consume the pipe whatever language you use to spawn a subprocess -- the buffer is not in your application -- it is the OS pipe buffer.

I imagine your broken code did something like:

  #XXX WRONG, DON'T DO IT
  p = Popen(cmd, stdout=PIPE)
  p.wait() #XXX deadlock is possible if cmd generates enough output
  output = p.stdout.read()

To avoid the deadlock, it is enough to swap the last two statements:

  p = Popen(cmd, stdout=PIPE)
  output = p.stdout.read() 
  p.wait()

that you would normally write as:

  output = check_output(cmd) # 2.7+

If you need to consume more than one pipe separately then you should consume them concurrently and therefore you need threads or async.io.

`Popen.communicate()` can read from both pipes concurrently (as you've mentioned) and it is always available. It was available even in Python 2.4 that introduced the subprocess module.

pjungwir2• 11 years ago

Yep, I've hit this in Perl and Ruby and even once wrote a class to do something like Python's `communicate`. It's especially trappy if you want to run a command and both send stuff to its stdin and read stuff from its stdout. Bash makes it so easy, but it's a fraught with danger anywhere else!

swinglock• 11 years ago

Funny, I fixed a bug caused by exactly this recently in a program in production that had been humming along nicely until the output of another utility had grown larger than it ever was in testing or previously in production, causing the deadlock. The big warning in the Python docs sure was helpful in finding the problem.

mitchty4• 11 years ago

While this wouldn't have fixed this issue, this is why you should have:

    stty -ixon

In your .profile/shrc for your shell of choice. I've never needed flow control and never probably will, this stops that junk.

david-given• 11 years ago

Conversely: I use ^S and ^Q _all the time_. I've got a long-running program that's spewing output to a terminal? ^S and it stops so I can look at it, copy output elsewhere, etc. ^Q and it resumes again seamlessly.

It occurs to me that I grew up on the BBC Micro, where pressing Ctrl+Shift would pausing scrolling, so being able to do this is tightly engrained in muscle memory...

mitchty4• 11 years ago

Fair enough, though I tend to just use tee and/or tmux for things like that. Just turn logging on if I'm curious and let things continue on their merry way.

Only time i've ever used it was when booting a system to stop output on a serial console. That was years ago and a one off basically.

JadeNB• 11 years ago

Conversely: I use ^S and ^Q _all the time_. I've got a long-running program that's spewing output to a terminal? ^S and it stops so I can look at it, copy output elsewhere, etc. ^Q and it resumes again seamlessly.

Why not ^Z and `fg`? (I'm sure that there's a subtle or not-so-subtle difference, but I've only ever suspended, never XOFF'd, so I don't know it.)

_yosefk• 11 years ago

I had a problem with scripts (reeeeeally hairy scripts with intertwined tcl and shell code, the tcl being used to drive some really obscure software) that failed when called from tcsh scripts which didn't have "-f" in them, and the user's .tcshrc had stty -ixon in it. I never figured out what the problem was...

fit2rule5• 11 years ago

Never needed flow control? Like, never?

Not much of a console user then, maybe? Because, like, cat /something is useless without page control.

Oh, maybe you're a speedreader?

na85• 11 years ago

I notice you are consistently a jerk when you write replies. Please stop being so snarky/sarcastic.

I never use flow control either, I just pipe it to less.

dlitz• 11 years ago

I've been aware of XON/XOFF since forever, but I mostly just use my terminal's scrollback buffer.

mitchty4• 11 years ago

I've only used flow control when booting systems like solaris on a serial console or other embedded hardware. Almost never for a tty session. I'm almost always saving output to a file and looking at that. Or using less +F or tmux etc... so yeah.

So my use of never might be hyperbolic, but it is a figure of speech. I can count on one hand with fingers leftover how often I've "needed" flow control. So just like in math, 0.99999 is equivalent functionally to 1 in this case.

yvdriess• 11 years ago

Note that this enables you to use the bash 'forward search', CTRL+R's forgotten twin.

I use CTRL+R and CTRL+S often in emacs and was frustrated that CTRL+S would not work on a command line terminal. As the article mentions, CTRL+S is typically interpreted as XON. Turning that off will enable forward search.

gbin• 11 years ago

Kill this thing with fire ! Put in your .zshrc/.bashrc: stty -ixon -ixoff

fit2rule5• 11 years ago

I find it really cute that XON/XOFF are still with us, and people still marvel at the discovery from the deep dungeons of present time, since its pretty much everywhere in modern systems ..

Also of note, besides "TSTOP/TSTART" (^S/^Q) - as I knew it before XOFF/XON became the standard nomenclature courtesy of ASCII (X = Transmission), is of course the good ol' ^L, which is affectionately known as the boss-key in some parts. Life can't persist without some ^M and ^N business as well, and lets never forget the favourite "wake up your co-worker and go get coffee", ^G, although for some this may require your terminal have a buzzer or other suitable charge-based co-worker wake-up circuitry.

Well, I'm sure there are more in the alphabet to discover, but I'll just leave you here with the reminder that ^C and a few others of the happy little family of ^W control sequences are really but a few bytes away. ^Z

boot13• 11 years ago

It really is still everywhere. For example, you can use ^S to pause your PC's POST display if you're fast enough. The Windows command line still uses it. And so on.

quadhome• 11 years ago

I don't even know how to quit most command-line programs.

lsiebert• 11 years ago

Personally I use Ctrl-S for tmux for the command key, since I use Ctrl-A in vim. If I really need to send a signal I can Ctrl-s Ctrl-s, but otherwise this works fine.

andmarios• 11 years ago

I never thought anything of it because in KDE terminals (konsole, yakuake) when you press CTRL+S, a yellow warning appears on the terminal stating that output is suspended and may be resumed by CTRL+Q. :)

amenghra• 11 years ago

That's called good UI design. Kudos to whoever decided to implement that!

ris• 11 years ago

Oh my god it all makes sense now. Thankyou. -a screen user

userbinator• 11 years ago

There is a good historical background article on the whole TTY subsystem here:

http://www.linusakesson.net/programming/tty/

It mentions what XON/XOFF, as well as what all the other control characters do.

nutate• 11 years ago

I always seem like a genius when I ctrl-q out of someone's inexplicably locked terminal. Lord knows I discovered it through some serious pain and wonderment.

icedchai• 11 years ago

Nice reminder. I used to use ctrl-s / ctrl-q a lot back in the dialup / BBS days.

ezy• 11 years ago

Didn't everyone use this all the time to pause 300 & 1200 bps output in the days before screen and scrollback terminals? :-)

otterpro• 11 years ago

I always disable Ctrl-S with "stty -ixon" in .zshrc / .bashrc. Then, I remap Vim's Ctrl-S with "save/ write"

in .vimrc

nn <C-s> :w<CR>

vn <C-s> :w<CR>

ino <C-s> <ESC>:w<CR>

laacz• 11 years ago

When coming from windows world, I quickly learned that ctrl+s can be undone with ctrl+q. And that's all, never went into more detail. Wrote it off as a quirky stuff linux has plenty of.

pjungwir2• 11 years ago

I hit the issue with blocking log IO all the time, not because of ^S, but because I develop in tmux and hit ^A[^B^B^B to read some stack trace (my ^A is tmux's ^B), then Alt-Tab to the browser, hit ^R, and it just spins. Then I wack my head again, Alt-Tab to tmux, hit Enter, and the page loads.

It's all fun and games until someone [XOFF]

Related Stories