What happens when you press a key in your terminal?

https://jvns.ca/blog/2022/07/20/pseudoterminals/

by robenkleene • 4 years ago

408 101 4 years ago

ratsmack• 4 years ago

The picture of the VT100 brings back memories. I had one at home and would connect to our Vax 11/780 to program in FORTRAN using the EDT editor... using a 1200 baud modem.

(about 1981 or so)

bjelkeman-again• 4 years ago

I wish I had kept my amber terminal. Great for working late at night, very focused. Having a modern display projecting amber colours just isn’t the same.

lambic• 4 years ago

I shared a flat in London with two other geeks in the 90s and we wired up vt100 terminals in each of our rooms so we could talk to each other without leaving our rooms.

They didn't last long, they were quickly replaced by PCs on a lan so we could play Doom.

andreskytt• 4 years ago

And this, kids, is why it was possible to do fun things to computers of other people when their os did not tell the modem to stop accepting incoming bytes when the line was dropped and then picked up again. It just accepted bytes from wherever they might come…

oars4• 4 years ago

So I didn't know that when I press backspace in the terminal, "x08" is being sent, not "^H".

What is the purpose of caret notation then? Is it just for human readability? e.g. my terminal shows ^H sometimes when I press backspace

yjftsjthsd-h• 4 years ago

Pretty sure it's for the human, yes; how else would you represent "hey human, something just echoed the thing you get from ctrl-h"? (Keeping in mind that this almost certainly predates unicode and fancy fonts, and probably colors in the terminal. I'm open to the idea that we could do better today, but it's hard to overcome 50 years of tradition.)

IncRnd• 4 years ago

So I didn't know that when I press backspace in the terminal, "x08" is being sent, not "^H".

"0x08" is "^H". H is the eighth letter of the alphabet. There is no difference between those except for the notation. In the ASCII character set, the first 32 characters are called control characters. This is why many keyboards have ^ on their control keys. The 26 control codes 1 through 0x1A correspond to ^A through ^Z.

PS This relates to the 7-bit ASCII character set, American Standard Code for Information Interchange, composed of control characters for communication control, letters, numbers, and punctuation.

vlovich123• 4 years ago

Usually that’s rendered as one control unit not as caret + H. The caret indicates an escape character. You can typically type in escape sequences via ctrl v (eg ctrl-v + tab for the tab character instead of tab completion)

oars4• 4 years ago

Is the "ctrl + E" shortcut to jump to end of line in a terminal interpreted/executed by the terminal or bash?

ericbarrett2• 4 years ago

Bash, via the GNU Readline library.

oars4• 4 years ago

So following the examples in the article, inbetween the sent and recv, it's actually bash interpreting what the sent command is, not the terminal?

For example, the ls command example from the article:

sent: "l" recv: "l" sent: "s" recv: "s" sent: "\r" recv: "\r\n\x1b[?2004l\r" recv: "file\r\n" recv: "\x1b[?2004hbork@kiwi:/play$ "

The interpretation and processing of "\r" to return the final output is actually bash processing this, not the terminal?

gnubison6• 4 years ago

Note that, although bash handles this case, the terminal never interprets ^C or backspace or ^U or anything like that. That’s all in the tty driver — see OpenBSD’s termios(4) for information on how to configure it.

In general, “smart” programs like bash or vi or fzf or readline configure the termios state so that the tty driver doesn’t handle any keys. This gives them more control. When they exit, they restore the termios to the original state, so that you can still backspace in dumb programs like cat and grep.

So you might have a dance like this when you run vi:

- bash restores its saved termios, so that whatever program it’s running starts with a blank slate

- vi saves the original termios

- vi switches the termios into “raw mode” (simplification)

- you edit text …

- vi switches back to the state it saved

- vi exits

- bash saves the termios state

- bash switches to raw mode

megous• 4 years ago

This tells what happens when the virtual terminal receives some input. You can pass that input from a file and it would do the same.

There's also a whole lot that happens when you press a key to generate said input (and I don't even mean at the hardware level), which is perhaps much less known.

TacticalCoder• 4 years ago

And things turn really "fun" when you begin using modifiers which aren't supported by the USB keyboards specs, like the Hyper key. I've got my Linux / X configured to use Super and Hyper keys but things quickly turn weird. For example the Hyper key works totally fine from Emacs in GUI mode (and, over the years, I assigned a huge number of shortcuts to the Hyper key), since years and years, but I have never spent the time to make it work in the terminal. It's doable but requires some arcane magic. I'm not even talking about having the Hyper key working from a tty (like if I boot in non graphical mode) but simply having it work from, say, an xterm under X. Oh the fun.

So until I fix that no "emacs -nw" (emacs in terminal mode) for me as I rely way too much on my Hyper key.

aumerle2• 4 years ago

You dont need to do anything fun. Just convince emacs developers to support a modern terminal keyboard protocol. https://sw.kovidgoyal.net/kitty/keyboard-protocol/ and you get support for hyper out of the box.

comfypotato• 4 years ago

It's more than just the Hyper key (as I'm sure you're aware). The article mentions how C-S-anything is the same as C-anything. I've always wanted to set up a fully functional terminal Emacs (for playing through ssh on a tethered connection in a café) but it seems to be a fairly in-depth process to have an interface with all the things. I use QMK extensively in tandem with my Emacs configuration, and there is a ton of functionality to transpose to the terminal. Ultimately I think I would have to completely redo my configuration in both QMK/KMonad and Emacs to stick to the codes that are sent correctly.

Sending "F18 a" with QMK or KMonad (a random prefix-combo I picked for example purposes) instead of "H-whatever" (for whatever keystroke combo you have H-whatever bound) would work with a terminal in your case, but you'd have to change all those bindings and setup QMK/KMonad accordingly. That's altogether too much work.

easrng• 4 years ago

Another fun modifier is the Office key, which actually sends Ctrl+Alt+Shift+Win, so if you can manage to hold down all the keys Ctrl+Alt+Shift+Win+L will open LinkedIn on Windows laptops.

jeffbee• 4 years ago

This is why i can’t pass simple interviews. Whenever they ask this question I start talking about how the keyboard works, or, once, how my teletype model 33 works.

xwowsersx• 4 years ago

I really enjoyed reading this.

shadowofneptune• 4 years ago

Other fun with control characters: The article caught that Alt is ESC, but didn't catch that Ctrl+[ is too. Check it out in a program that uses control codes, like Emacs or Vi. I myself find it somewhat more ergonomic depending on the action.

gumby4• 4 years ago

She wrote that “everyone had one” but really it meant “everyone online at a given time” was using one. In a lot of facilities they were shared.

gumby4• 4 years ago

Look at the ascii chart by bit pattern and the characters will suddenly make sense. Here’s an old chart from the 60s that explains it:

https://programesecure.com/ascii-values-table-generator-in-c...

Unfortunately if you just see the character codes with the decimal, hex, and octal next to them this logic is obscured. Remember, it had to be implementable in (mechanical!) hardware.

zoomablemind• 4 years ago

...Look at the ascii chart by bit pattern and the characters will suddenly make sense.

Could you elaborate on what does the bit pattern reveal?

For example, I understand that Ctrl-C generates 0x30, which stands for ETX (end of transmission), but what is it there in 011?

aap_2• 4 years ago

Ctrl clears the highest two bits. E.g. @ (ascii 0x40) turns into ^@/NUL (ascii 0x00). Similarly C (0x43) -> ^C/ETX (0x03).

Older terminals (like the VT05) used bit-pairing also with shift. They just flipped some bits depending on the upper two bits. Compare column 2 with 3 in the linked graphic, and 4,5 with 6,7.

jart2• 4 years ago

I don't know if the chart helps, but

    #define CTRL(x) ((x) ^ 0100)
    // e.g. 'C' ^ 0100 is 3
    // e.g. '@' ^ 0100 is 0
    // e.g. 'M' ^ 0100 is 13 a.k.a. \r a.k.a. enter

I like this explanation better. It also explains what notation like ^C means. It's shorthand for 0100^C.

gumby4• 4 years ago

Just run your eye horizontally to see that toggling only a couple of bits move you between ^C, C, and c (back when ASCII was formalized few terminals supported lower case, and six bit character sets were common).

SheetPost2• 4 years ago

Here’s an old chart from the 60s that explains it

eh, the article linked does a very poor job of an explanation of ASCII

And no wonder, it's a deep subject - if you want answers the you want to watch this talk: https://m.youtube.com/watch?v=_mZBa3sqTrI

gumby4• 4 years ago

That site didn’t let me link straight to the chart. I’ve seen that chart for years (decades) but that was the only one I could quickly find on the web.

raggi• 4 years ago

There's a command for doing the diagnosis goterm was used for here, it's called script(1).

schoen• 4 years ago

I believe the reason cat gets interrupted when we press Ctrl+C is that the Linux kernel on the server side receives this \x03 character, recognizes that it means “interrupt”, and then sends a SIGINT to the process that owns the pseudoterminal’s process group. So it’s handled in the kernel and not in userspace.

Also interestingly, the use of \x03 for this purpose is a default, but it's not hardcoded. You can change it with the stty command.

For example, if you run

stty intr '^X'

then your interrupt character sequence will be Ctrl+X instead of Ctrl+C (!).

In order to make this change, the stty program actually has to call into the kernel with an ioctl call (TCSETS or a related ioctl, for "terminal control set settings").

You can learn more about this ioctl on Linux in the ioctl_tty(2) man page, and you can see the various settings that can be changed this way with

stty -a

(It's a little bit confusing which things are handled by the readline library and which things are handled by the kernel, as the kernel's ability to support some rudimentary line-editing features on an interactive terminal long predates libraries like readline. I think readline might actually disable kernel interpretation of some of these control characters when it starts accepting input, and then re-enable them when it stops, but I've never looked into that.)

formerly_proven• 4 years ago

I think readline might actually disable kernel interpretation of some of these control characters when it starts accepting input, and then re-enable them when it stops, but I've never looked into that.

That's the canonical mode bit (ICANON). Canonical mode means the kernel/terminal is line-buffered and line-editing is handled by it; non-canonical mode means user input is pushed to the application immediately.

tonnydourado• 4 years ago

For example, if you run
stty intr '^X'
then your interrupt character sequence will be Ctrl+X instead of Ctrl+C (!).

I have a feeling that if you actually do this in a real Linux system, a sysadmin will hunt you down and dismember you alive, but I might be wrong.

deadbunny• 4 years ago

Can confirm.

jwilk4• 4 years ago

Fun fact: it doesn't have to be Ctrl+something. If you feel paricularly evil today, you might try this:

  stty intr y

codesnik• 4 years ago

joke aside, it is a per-tty setting, right?

grymoire1• 4 years ago

Nope. As someone who administered a VAX with 50 simultaneous users, it's not a problem. It only affect that process and that user during that session. A logout resets everything. A more interesting problem was trying to read or write to a serial port that was hardwired to use a different baud rate.

You could type "stty 9600 /dev/tty4"; cat file >/dev/tty4" and it wouldn't work because when stty exited, the system would reset the terminal baud rate.

The proper way to do this (assuming you weren't the sysadmin and couldn't modify the default per-terminal baud rate), was to type the following

(stty 9600; cat file)>/dev/tty4

dylan604• 4 years ago

The timing of HN can be spooky sometimes. Earlier this week I was just wondering it I could switch the ctrl-c interrupt to a single key for easier keyboard smashing to stop an errant poorly thought out script. My choice would have been the spacebar as that is second nature muscle memory from all of my days in an edit bay where the spacebar acted as the All Stop and was engraved with the phrase Awww Shit! as that was the typical utterance from the editor just before using the key

dundarious• 4 years ago

Anyone interested in the machinations of all of this terminal stuff should look at antirez’ kilo, a terminal text editor in under 1000 lines of code: https://github.com/antirez/kilo

There is a nice tutorial that walks through how one might write it from scratch: https://viewsourcecode.org/snaptoken/kilo/

account42• 4 years ago

stty is useful when an interactive program died before it had a chance to restore the terminal mode so you end up with nothing displayed when you type - `stty sane` will fix that up.

noncoml• 4 years ago

2004h -> “Turn on bracketed paste mode”

2004l -> “Turn off bracketed paste mode”

https://en.m.wikipedia.org/wiki/ANSI_escape_code#CSIsection

durazabu2• 4 years ago

Same mechanism that we could use to disambiguate keys like ^H and backspace.

http://www.leonerd.org.uk/hacks/fixterms/

aumerle2• 4 years ago

That is obsolete and wrong in many ways: https://sw.kovidgoyal.net/kitty/keyboard-protocol/#bugs-in-f...

niels_bom• 4 years ago

Julia Evans is such a treasure. I’ve learned a tonne reading her stuff.

benreesman2• 4 years ago

I don’t know Julia Evans, certainly not her academic background.

But she blogs eloquently from the standpoint of someone learning things from first principles, in an accessible, actionable way. As a self-taught person this is near and dear to me. She’s like the 3Blue1Brown of systems programming, and there’s not much higher praise.

Striking the balance between rigor and practical applicability is tough, especially while assuming little prior knowledge.

Keep up the good work @jvns!

jbreckmckye2• 4 years ago

She was a Staff Engineer at Stripe. She wrote a great primer on SQL too.

https://jvns.ca/ is a treasure trove

benreesman2• 4 years ago

Stripe is an interesting place. It’s clearly a great business, but they’ve let that bleed over a bit into thinking their p95 people are Google p95, which is trivially silly.

It’s an awesome company, and they’ve got solid folks, but “Staff” at Stripe isn’t what makes Julia cool: Julia’s work with Recurse is way cooler.

I mean we’ve got Consul: https://stripe.com/blog/service-discovery-at-stripe

And then etcd: https://stripe.com/blog/operating-kubernetes

Both courtesy of Julia incidentally.

Chubby has worked since like 2003. We’re just talking a different level of ball game.

mongol• 4 years ago

Have anyone attempted to connect a serial terminal over bluetooth? At some point I thought that could be a neat idea because it would be "internet secure". I don't remember the circumstances exactly, but I never got it to work fully, there were some echoing of characters that I never figured out how to configure away.

Wildgoose• 4 years ago

The point of echoing back what was received is not because the client is "really dumb", it's actually good engineering practise. It directly confirms to the user what the machine believes it has received.

If this is incorrect, (whether caused by line noise or whatever), then the user knows this immediately. Unix has a lot of 2-character commands - you wouldn't want an innocent command accidentally mistranslated to "rm" and then press <Return>.

rmetzler• 4 years ago

Yes, state is only modified on the server, not in the dumb clients. Weakening this fundamental idea is what makes it hard to write modern single page applications.

grayfaced• 4 years ago

It also means the server can turn off echo for sensitive things like passwords.

gnubison6• 4 years ago

And, more importantly, for programs like vi and Emacs that want to handle rendering themselves.

jart2• 4 years ago

remote terminals are very old technology

TTYs or teletypewriters have been in use since the 19th century. I'd love to see a blog post that talks more about the early history.

pphysch2• 4 years ago

It's a valuable exercise to write a web shell service with Go standard library (rather than goterm). It shouldn't take more than 100-200 LOC and you will learn a lot about how SSH protocol, streaming IO, websockets work.

SheetPost2• 4 years ago

ah, it's also a great deal of a distraction if your goal was to see what gets sent when presing keys

pphysch2• 4 years ago

On the contrary, you don't have to do much more work than the author already did, and you get deep practical insights into how this very useful technology works beyond the marginally useful trivia of how terminals happen to internally represent keystrokes

jbreckmckye2• 4 years ago

There should be a body of these "koans", little program specs to implement to explore various technologies.

prosaic-hacker• 4 years ago

It is amazing how the understanding of what happens in our computers goes from : the air we breath to a mystery.

Although if you asked one of my 900 coworkers in 1981 who wrote flight simulation software in Fortran, how the VT100 terminals interacted with the computers they were connected to,only a few dozen would know.

One of my projects at the time was to create a library to support the creation of 'screen oriented' applications without knowing the escape sequences.

amelius• 4 years ago

For graphical interfaces we went from X11 to Wayland. Is anybody working on a replacement technology for text interfaces?

drainyard• 4 years ago

Something like Terminal Click? https://media.handmade-seattle.com/terminal-click/

atq2119• 4 years ago

Why? I don't think there is a clear and compelling case for doing this given the ecosystem issues. Meanwhile, there's slow but interesting innovation happening in this space e.g. around kitty.

kristopolous2• 4 years ago

I was imagining this would be an overview of the circuit closing upon the key press and the chips that translate it to the signal that traverses to the the USB connector and goes through a controller chip etc... From the mechanical closing of the circuit to the illumination at the display.

Maybe I should give that a go in some classic Tracy Kidder style. I certainly can't fill in all those steps. I'd have to do some learning myself

branchcontrol• 4 years ago

A slightly different exploration:

https://youtu.be/XUdxXON27xA

joezydeco• 4 years ago

It's actually a useful interview question in some positions, like embedded.

kristopolous2• 4 years ago

I've gotten it before and I had no idea <how> to answer.

This is characteristic of one of the common interview traps I fall in. I don't know who my audience is or what kind of answer they're looking for.

I can answer the software world version of that all day along with internationalization and the history. But hardware? No not really

lqet• 4 years ago

The client sends l and then immediately receives an l sent back. I guess the idea here is that the client is really dumb – it doesn’t know that when I type an l, I want an l to be echoed back to the screen. It has to be told explicitly by the server process to display it.

Of course the client may be really dumb! Here is a 1930s teletype used as a Linux terminal:

https://youtu.be/2XLZ4Z8LpEE?t=656

If you would not do it that way, you would run into all kinds of synchronicity problems. How could you be sure that, after rapidly making 50 keystrokes, every keystroke was received by the other end, in that order?

freedomben• 4 years ago

How could you be sure that, after rapidly making 50 keystrokes, every keystroke was received by the other end, in that order?

Exactly. I implemented a client that emulated a VT100 early in my career, and this is a real problem. There are various strategies you can use but by far the simplest and safest seemed to just be the echo and for the client to always display exactly what it receives[1].

There's nothing worse than typing out a command that you realize is wrong and potentially destructive, only to Ctrl+U it and have the client kill the line but the server didn't get the instruction, so when you press enter it runs the evil command. If the command doesn't echo anything you may not even know! I once accidented a space in the path I was deleting when (recklessly with -rf) trying to remove my ~/bin directory, like this:

    rm -rf ~ /bin

Good Lord that was a bad day. Thankfully I still had the installation disc to restore /bin, and a relatively recent backup of my home directory to restore that. I lost a few days of uncommitted code, but that felt like a trifle compared to what it could have been :-)

[1]: I love how mosh[2] handles this to get the best of both worlds. It will smartly show you what you typed, but sill underline it until it actually receives the echo from the server, so you can type and feel like there's no delay between bytes, but still be confident that client state matches the server state.

[2] https://mosh.org/

icedchai• 4 years ago

This was definitely a problem in the dialup modem days before error correction, where line noise might add extra characters, or modify what you were sending.

gnubison6• 4 years ago

More importantly, how do you know whether the keystrokes should be displayed at all? For example, take vi. When it starts up, it sets cc.c_lflag &= ~ECHO (basically like “stty -echo”) so that normal mode commands aren’t printed onscreen. The kernel knows whether ECHO is set, which is why it handles echoing.

smelbe• 4 years ago

And this, guys, is the reason why it was able to do fun things to other people's computers when their operating system did not inform the modem to cease receiving incoming bytes when the line was dropped and then picked back up again. It did not care where the bytes came from; it just accepted them...

jstimpfle• 4 years ago

I think the canonical resource on this is https://www.linusakesson.net/programming/tty/

pjmlp• 4 years ago

This is what happens on a classical UNIX terminal, other platforms have different workflows.

mega_dingus• 4 years ago

e.g. VMS would not echo back keystrokes until it was truly ready to accept input

No typeahead while a previous command is running, like you can do in UNIX et al

kps3• 4 years ago

And many mainframe (IBM et al) terminals worked more like HTML forms — typing modified only the local screen, with a key to send the entire current state to the host.

whalesalad• 4 years ago

As a kid my mom would take my sister and I to the big Pasadena Public Library which had these DEC terminals all over the place for using the card catalog to find books. I was fascinated by them and would often just sit and play hacker. Might even be a big reason for why I am a programmer today.

bryanrasmussen• 4 years ago

I might be really tempted to say "my terminal doesn't have any keys" if this came up in an interview.

throw0101a• 4 years ago

Nothing about scancodes and mapping the physical location of the key to the character depending on the keyboard layout (QWERTY/AZERTY/etc)?

* https://en.wikipedia.org/wiki/Scancode

* https://en.wikipedia.org/wiki/Keyboard_layout

sunyc• 4 years ago

Next step is to explain how ctrl+c work over ssh , it is my favorite system engineering interview question.

Joker_vD4• 4 years ago

The ssh client on your local end receives SIGINT and processes it by sending a special kind of packet over the ssh session to the other side; the sshd on the remote side receives this special packet and processes it by sending SIGINT to whatever command it has originally spawned.

IIRC telnet instead uses urgent TCP packets to indicate SIGINT.

gnubison6• 4 years ago

My guess: The ssh client on your local end receives a 0x03 byte directly from the terminal, because it has disabled ICANON and ISIG and whatnot, and forwards it to the remote connection. The remote sshd then feeds the 0x03 byte to the pseudoterminal it has setup, and then the (remote) kernel may or may not interpret that as SIGINT. For example, ISIG could be disabled (“stty -isig”) or it might be a different key (e.g. “stty intr '^X'”, as mentioned elsewhere in this thread).

Your setup fails to distinguish keyboard interrupts (intended for the remote machine) and real SIGINTs generated by kill(1). It also uses the local termios(4) settings instead of the remote ones.

tpoacher• 4 years ago

This also partly explains why a lot of keystrokes in nano do the same thing and cannot be bound separately (see attached file keystrokes.nanorc in https://savannah.gnu.org/bugs/index.php?61699 )

evernerdy• 4 years ago

Love this! The format for some reason made me flash back to using Telnet for IRC. lol

What happens when you press a key in your terminal?

Related Stories