Whence '\n'?

https://rodarmor.com/blog/whence-newline/

by lukastyrychtr • 2 years ago

338 128 2 years ago

$Preview of Whence '\n'?$ rodarmor.com

cpach2• 2 years ago

Previous discussion: https://news.ycombinator.com/item?id=41564527

nasso_dev• 2 years ago

This post was inspired by another post about exactly the same thing. I couldn't find it when I looked for it, so I wrote this. All credit to the original author for noticing how interesting this rabbit hole is.

I think the author may be thinking of Ken Thompson's Turing Award lecture "Reflections on Trusting Trust".

Karellen• 2 years ago

Although that presentation does point out that the technique is more generally used in quines. Given that there is a fair amount of research, papers and commentary on quines, it's possible that the author may have read something along those lines.

https://en.wikipedia.org/wiki/Quine_(computing)

ktm5j• 2 years ago

I totally missed that bit when the OP, but it definitely made me think of that paper so maybe.

yen2233• 2 years ago

I don't think so. I too recall seeing a post about this exact piece of trivia ('\n' in rust) years ago, but I couldn't find the source anymore.

tylerhou• 2 years ago

It might have been https://research.swtch.com/nih ?

yuchi• 2 years ago

Also have a read of this fabulous short web from 2009: https://www.teamten.com/lawrence/writings/coding-machines/

kijin• 2 years ago

The incorrect capitalization made me think that, perhaps, there's a scarcely known escape sequence \N that is different from \n. Maybe it matches any character that isn't a newline? Nope, just small caps in the original article.

paulddraper• 2 years ago

There is actually.

Many systems use \N in CSVs or similar as NULL, to distinguish from an empty string.

I figured this is what the article was about?

cpach2• 2 years ago

If you do view source, it’s actually \n, but it’s not displayed as such because of this CSS rule:

  .title {
    font-variant: small-caps;
  }

sedatk3• 2 years ago

So, the HN title is wrong.

deathanatos2• 2 years ago

Python has a \N escape sequence. It inserts a Unicode character by name. For example,

  '\N{PILE OF POO}'

is the Unicode string containing a single USV, the pile of poop emoji.

Much more self-documenting than doing it with a hex sequence with \u or \U.

binary1325• 2 years ago

That is in fact why I clicked this article. Oh well. Still a fun read. :)

archmaster• 2 years ago

if only this went into where the ocaml escape came from :)

diath• 2 years ago

It does, it links to this: https://github.com/ocaml/ocaml/blob/4d6ecfb5cf4a5da814784dee...

fiddlerwoaroof• 2 years ago

But this doesn’t really explain anything: ‘\010’ isn’t really any more primitive than ‘\x0a’: they’re just different representations of the same bit sequence

dist-epoch• 2 years ago

I remember a similar article for some C compiler, and it turned out the only place the value 0x10 appeared was in the compiler binary, because in the source code it had something like "\\n" -> "\n"

atoav• 2 years ago

One rule of programming I figured out pretty quick is: if there are two ways of doing it and there is a 50/50 chance of one being correct and the other one isn't, chances are you will get it wrong the first time.

chgs2• 2 years ago

The USB rule.

First time is the wrong way up

Second time is also the wrong way up

Third time works

fader• 2 years ago

It's because of the quantum properties of USB connectors. They have spin 1/2.

jancsika• 2 years ago

It's like the Two General's Problem embedded in a single connector.

You never really know it's right until you take it out and test the friction against the other orientation.

dtgriscom• 2 years ago

I boosted my USB plugged-in-successfuly-on-first-try rate when I imagined the offset block in the cable male USB connector as being heavy, so it should be below the centerline when plugged into a laptop's female USB connector. (Only works when the connector is horizontal, but better than nothing.)

dailykoder• 2 years ago

It's actually super easy and, atleast for me, was always intuitive. Most USB cables have their logo or something else engraved on the "top" with the air gap. And since the ports are mostly arranged the same way, there is rarely any problem. Maybe I am just too dumb to understand jokes, but it always confused me :(

crote3• 2 years ago

USB-C changed that to "It'll physically fit the first time, but good luck figuring out if it's going to work!"

ncruces• 2 years ago

I'm guessing the “other post” that inspired this might be: https://research.swtch.com/nih

dang2• 2 years ago

Discussed here:

Running the "Reflections on Trusting Trust" Compiler - https://news.ycombinator.com/item?id=38020792 - Oct 2023 (67 comments)

tzot• 2 years ago

I always thought, maybe because of C, that \0??? is an octal escape; so in my mind \012 is \x0a or 0x0a, and \010 is 0x08.

So I find this quite confusing; maybe OCaml does not have octal escapes but decimal ones, and \09 is the Tab character. I haven't checked.

dpassens• 2 years ago

It is indeed a decimal escape: https://ocaml.org/manual/5.2/lex.html#char-literal

fanf22• 2 years ago

Yeah backslash-decimal character escapes are really rare, the only string syntaxes I know of that have them are in O’Caml, Lua, and DNS

binary1325• 2 years ago

Is O’Caml an Irish fork of OCaml? :)

syncsynchalt2• 2 years ago

There's some truth in that direction, but it's not related to backslash escapes (which are symbolic/mnemonic, \n is "[Ne]wline", \r is "carriage [R]eturn", \t is "[T]ab", and so on).

Instead, consider the convention of control characters, such as ^C (interrupt), ^G (bell), or ^M (carriage return). Those are characters in the C0 control set, where ^C is \0x3, ^G is \0x7, or ^M is \0xD. You're seeing a bit of cleverness that goes back to pre-Unix days: to represent the invisible C0 characters in ASCII, a terminal prepends the "^" character and prints the character AND-0x40, shifting it into a visible range.

You may want to pull up an ASCII table such as https://www.asciitable.com to follow along. Each control character (first column) is mapped to the ^character two columns over, on that table.

That's why \0 is represented with the odd choice of ^@, the escape key becomes ^[, and other hard-to-remember equivalents. These weren't choices made by Unix authors, they're artifacts of ASCII numbering.

sudobash1• 2 years ago

Minor nitpick: the terminal prints the character OR'ed with 0x40 (not AND'ed).

wruza3• 2 years ago

Better visible here: https://www.commfront.com/pages/ascii-chart

gjvc• 2 years ago

this is a nothingburger of an article

coolio1232• 2 years ago

I thought this was going to be about '\N' but there's only '\n' here.

dang2• 2 years ago

It's in the html doc title but the article doesn't deliver.

ynfnehf• 2 years ago

First place I read about this idea (specifically newlines, not in general trusting trust) was day 42 in https://www.sigbus.info/how-i-wrote-a-self-hosting-c-compile...

"For example, my compiler interprets "\n" (a sequence of backslash and character "n") in a string literal as "\n" (a newline character in this case). If you think about this, you would find this a little bit weird, because it does not have information as to the actual ASCII character code for "\n". The information about the character code is not present in the source code but passed on from a compiler compiling the compiler. Newline characters of my compiler can be traced back to GCC which compiled mine."

moefh• 2 years ago

I was hoping GCC would do the same, leaving the decision about the value of '\n' to GCC's compiler, but apparently it hardcodes the numeric values for escapes[1], with options for ASCII or EBCDIC systems.

[1] https://github.com/gcc-mirror/gcc/blob/8a4a967a77cb937a2df45...

codetrotter• 2 years ago

But these numeric values are also ASCII representation of numbers, rather than being the actual byte that is written to the output. Maybe there is hope still. Where do the byte values for those numbers come from when the compiler writes output?

happytoexplain3• 2 years ago

This is over my head. Why did we need to take a trip to discover why \n is encoded as a byte with the value 10? Isn't that expected? The author and HN comments don't say, so I feel stupid.

kibwen• 2 years ago

The point is to ask "who" encoded that byte as the value of 10. If you're writing a parser and you parse a newline as the escape sequence `\n`, then where did the value 10 come from? If you instead parse a newline as the integer literal `10`, then where does the actual binary value 1010 come from?

The ultimate point of this exercise is to alter your perception of what a compiler is (in the same way as the famous Reflections On Trusting Trust presentation).

Which is to say: your compiler is not something that outputs your program; your compiler is also input to your program. And as a program itself, your compiler's compiler was an input to your compiler, which makes it transitively an input to your program, and the same is true of your compiler's compiler's compiler, and your compiler's compiler's compiler's compiler, and your compiler's compiler's compiler's compiler's compiler, and...

dotancohen• 2 years ago

All right Stan, don't belabour the point.

mikl• 2 years ago

The interesting point is how the value of 10 is not defined in Rust’s source code, but passed down as “word of mouth” from compiler to compiler.

analog31• 2 years ago

The earliest reference on my shelf is Kernighan and Ritchie, 1978. Like you say, it's probably just been passed on from generation to generation. And it's easier to say "just like C" than to make up a new convention. Python uses the same convention.

happytoexplain3• 2 years ago

Ohh, a lot of spoken and unspoken things in the article made me think this was supposed to be a bug/quirk investigation. This makes much more sense.

yen2233• 2 years ago

If you had to rebuild the rust compiler from scratch, and all you had was rustc's source code, there's nothing in the source code to tell you what '\n' actually maps to.

It's an interesting real-world example of the Ken Thompson hack.

crote3• 2 years ago

The thing is, why 10? Why not 9 or 11? The code says "if you see 'string of newline character', output 'newline character'". How does the compiler know what a newline character is? Its code in turn just says "if you see 'string of newline character', treat it as 'newline character'"...

As a human I can just Google "C string escape codes", but that table is nowhere to be found inside the compiler. If C 2025 is going to define Start of Heading as \h, is `'h' => cooked.push('\h')` going to magically start working? How could it possibly know?

Clearly at some point someone must've manually programmed a `'n' => 10` mapping, but where is it!?

TacticalCoder• 2 years ago

The thing is, why 10? Why not 9 or 11?

Check the first Unicode codepoints, 10 is defined there:

000A is LINE FEED (LF) = new line (NL), end of line (EOL)

It was already 10 in ASCII too (and the first 128 codepoints of Unicode are mostly the same as ASCII [I think there are a few tiny differences in some control characters]).

So to answer your question: it's neither 9 nor 11 because '\n' stands for "new line" and not for "character tabulation" nor for "line tabulation" (which 9 and 11 respectively stands for).

Clearly at some point someone must've manually programmed a `'n' => 10` mapping

I don't disagree with that.

kreco• 2 years ago

The thing is, why 10? Why not 9 or 11? The code says "if you see 'string of newline character', output 'newline character'". How does the compiler know what a newline character is? Its code in turn just says "if you see 'string of newline character', treat it as 'newline character'"...

You can look into the ascii table then.

10 is the line feed. '\n' is there to help human because it makes more sense than '10'.

Nobody is asking why 'a' is equal to '61'.

From the codebase, you know that '\n' is a char. A char is a value between 0 and 255, if you explicitly convert '\n' to int then you happen to find the ascii value and you are good to go and there is no need to pretend there is any poetry in this.

Requoting you:

"if you see 'string of newline character', output 'newline character'"

It simply becomes "if a you see 'the arbitrary symbol for the new line', output 'the corresponding ascii value'".

I read the quote as "if you see 'a', output 'a' in ascii code." which is not mysterious in any kind of way.

IMO, the article does not make sense at all because it pretends to wonder where an hexadecimal value is coming from, but with any other kind of symbol it would be the exact same article, you enter 'a' and find some weird hexadecimal value and you won't be able to correctly trace it from the source code.

It would make sense it 'a' would display 'a' but '\n' would display some hexadecimal garbage in a text editor.

amelius4• 2 years ago

Why backslash?

o11c• 2 years ago

Because backslash is a modern invention with no prior meaning in text. It was invented to allow writing the mathematical "and" and "or" symbols as /\ and \/.

dTal• 2 years ago

Hm. According to Wiki, "As of November 2022, efforts to identify either the origin of this character or its purpose before the 1960s have not been successful."

While your rationale was used to argue for its inclusion in ASCII, as an origin story however it is very unlikely, as (according to wiki again): "The earliest known reference found to date is a 1937 maintenance manual from the Teletype Corporation with a photograph showing the keyboard of its Kleinschmidt keyboard perforator WPE-3 using the Wheatstone system."

The Kleinschmidt keyboard perforator was used for sending telegraphs, and is not well equipped with mathematical symbols, or indeed any symbols at all besides forward slash, backslash, question mark, and equals sign. Not even period!

sedatk3• 2 years ago

"Why character X for Y?" has usually a universal answer: character X wasn't used commonly for the given scope at the time, was available widely, and felt sensible enough for Y.

amelius4• 2 years ago

Since you mentioned Y, why was the Y combinator named Y?

Were A...X already taken?

amelius4• 2 years ago

A more interesting question: what would our code look like if ASCII (or strings in general) didn't have escape codes?

wongarsu• 2 years ago

In PHP you see a lot of print('Hello World' . PHP_EOL); where the dot is string concatenation (underrated choice imho) and PHP_EOL is a predefined constant that maps to \n or \r\n depending on platform. You could easily extend that to have global constants for all non-printable ascii characters.

jiehong• 2 years ago

The font you use could choose to display something for control characters, so they would have a visible shape on top of having a meaning.

Perhaps like [0] (Unicode notation).

[0] https://rootr.net/im/ASCII/ASCII.gif

zokier• 2 years ago

It would depend more on what we are intending to do, are we controlling a terminal, or are we writing to a file (with specific format).

Terminal control is fairly easy answer, there would be some other API to control cursor position, so the code would need to call some function to move the cursor to next line.

For files, it would depend on what the format is. So we might be writing just `<p>hello world</p>` instead of `hello world\n`. In fact I find it bit weird that we are using teletype (and telegraph etc) control protocol (what ASCII mostly is) as our "general purpose text" format; it doesn't make much sense to me.

rswail2• 2 years ago

Except that terminals didn't exist when ASCII was created, printers existed, either with a "carriage" that carried the actual print head (like IBM Selectric typewriters, or the original ASR33 teletypes) or "chain printers" that printed an entire line at a time with a set of rods for each character position that had the entire character set on them.

ivanjermakov• 2 years ago

This is actually a valid problem when writing quines[1]. You need to escape string delimiters in a way without using its literal.

This is what `chr(39)` is for in the following Python quine:

    a = 'a = {}{}{}; print(a.format(chr(39), a, chr(39)))'; print(a.format(chr(39), a, chr(39)))

[1]: https://en.wikipedia.org/wiki/Quine_(computing)

eru3• 2 years ago

You don't need these gymnastics of knowing ASCII in Python quines:

    a = '''a = {}
    print(a.format(repr(a)))'''
    print(a.format(repr(a)))

(OK, it's not a quine yet, because the output has slightly different formatting. But the formatting very quickly reaches a fixed point, if you keep feeding it into a Python interpreter.)

The actual quine is the slightly less readable:

    a = 'a = {}\nprint(a.format(repr(a)))'
    print(a.format(repr(a)))

https://github.com/matthiasgoergens/Quine/blob/c53f187a4b403... has a C quine that doesn't rely on ASCII values either.

fsckboy3• 2 years ago

what would our code look like if ASCII didn't have escape codes?

ASCII is the layer that doesn't have escape codes (although it does have a single code for ESC), ascii is just a set of mappings from 7-bit numbers to/from mostly-printable characters

ezequiel-garzon• 2 years ago

At least a full fourth of the ASCII code points, 0x00-0x1F, are not printable. A bit more, as we should add del (0x7F) and, according to some though more controversially, space (0x20).

amelius4• 2 years ago

What would you call the lower 32 or so codes then? Control codes?

rswail2• 2 years ago

If ASCII didn't have control sequences then teletypes wouldn't have worked.

How do you make the carriage (a literal thing on a chain being driven back and forth) "return" to the start of the line without a "carriage return" code?

How would you make the paper feed go up one line without a "line feed" code?

Same for ringing the bell, tabs, backspace etc.

A "new line" on a teletype was actually two characters, a CR and a LF.

Unfortunately, using one of the other sequences (eg RS, for "Record Separator") would have saved billions of CPU cycles and misinterpreted text files dealing with the CRLF, CR, LF, LFCR sequences to mean "new line".

binary1325• 2 years ago

Cool, but actually it was just 0x0A all along! The symbolic representation was always just an alias. It didn’t actually go through 81 generations of rustc to get back to “where it really came from”, as you’d be able to see if you could see the real binary behind the symbols. Yes I am just being a bit silly for fun, but at the same time, I know I personally often commit the error of imagining that my representations and abstractions are the real thing, and forgetting that they’re really just headstuff.

Etherlord87• 2 years ago

The point of the article, that a lot of people in the comments seem to be missing, is that this alias is not defined in the SOURCE CODE of the foo. And not in the rust compiler. And Rust compiler is compiled... Using Rust compiler! So if you follow the source code, there's no definition, anywhere - that is, unless you have a knowledge (that isn't there in the source code, not in the latest revision at least), that the Rust compiler wasn't always compiled in Rust compiler (duh!) and you know where it was compiled. So the definition of this alias is an information that is somewhat "parasitic", secondary, embedded implicitly, like non-DNA information inherited from your parents, known as "Epigenetic mechanisms" (not memes).

binary1325• 2 years ago

Yeah, but my point is, it’s 0A in the compiler’s binary. ‘\n’ is just its name. If you could somehow see inline where the name maps to in the binary, you’d be able to go look at it.

i4k• 2 years ago

This is fascinating and terrifying.

phibz• 2 years ago

Backslash escape codes are a convention. They're so pervasive that we sometimes forget this. It could just as easily be some other character that is special and treated as an escape token.

gnulinux• 2 years ago

This is a fascinating post. It reads to me like some kind of cross between literate-programming and poetry. It's really trying to explain the idea that when you run `just foo` the very 0x0A byte comes from possibly hundreds of cycles of code generation. Back in the day, someone encoded this information into OCaml compiler -- somehow -- and years later here in my computer 0x0A information is stored due to this history.

But the way in which this phenomena is explained is via actual code. The code itself is besides the point of course, it's not like anyone will ever run or compile this specific code, but it's put there for humans to follow the discussion.

pmarreck2• 2 years ago

wait, Rust was bootstrapped via OCaml?

odo1242• 2 years ago

Yep

Also, the original Rust developers were OCaml devs, Rust borrowed design features (like matching and additive types) from OCaml, the syntax for lifetimes in Rust is the syntax for polymorphism in OCaml, and they borrowed semantics such as latent typing.

pmarreck2• 2 years ago

That is very interesting!

defrost• 2 years ago

Interesting that 10 hours in there's no thread hits for EBCDIC.

All theories being bandied about should account for the fact that early C compilers appeared on non-ASCII systems that did not map \n "line feed" to decimal 10.

https://en.wikipedia.org/wiki/EBCDIC

As an added wrinkle EBCDIC had both an explicit NextLine and an explicit LineFeed character.

For added fun:

    The gaps between letters made simple code that worked in ASCII fail on EBCDIC. For example for (c = 'A'; c <= 'Z'; ++c) putchar(c); would print the alphabet from A to Z if ASCII is used, but print 41 characters (including a number of unassigned ones) in EBCDIC.

    Sorting EBCDIC put lowercase letters before uppercase letters and letters before numbers, exactly the opposite of ASCII.

The only guarentee in the C standard re: chracter encoding was that the digits '0'-'9' mapped in contiguous ascending order.

In theory* simple C programs (that printed 10 lines of "Hello World") should have tha same source that compiled on either ASCII or EBCDIC systems and produced the same output.

* many pitfalls aside

skissane• 2 years ago

As an added wrinkle EBCDIC had both an explicit NextLine and an explicit LineFeed character.

Despite EBCDIC having a newline/next line character (NEL), it is rarely encountered in many EBCDIC systems. Early on, most EBCDIC systems (e.g. MVS, VM/CMS, OS/400, DOS/VSE) did not store text as byte stream files, but instead record-oriented files – storing lines as fixed-length or variable-length records. With fixed-length records, you'd declare a record length when creating the file (80 or 132 were the most common choices); every line in the file had to be of that length, shorter lines would be padded (normally with the EBCDIC space character, which is 0x40 not 0x20), longer lines would either be truncated or a continuation character would be used. With variable-length records, each record was prefixed with a record descriptor word (RDW) which gave its length (and a couple of spare bytes that theoretically could be used for additional metadata). However, in practice the use of variable-length records for text files (including program source code) was rather rare, fixed-length records were the norm.

So even though NEL exists, it wasn't normally used in files on disk. Essentially, newline characters such as NEL are "in-band signalling "for line/record boundaries, but record-oriented filesystems used "out-of-band signalling" instead. I'm not sure exactly how stdio was implemented in the runtime libraries of EBCDIC C compilers – I assume \n did map to NEL internally, but then the stdio layer treated it as a record separator, and then wrote each record using a separate system call, padding as necessary.

Later on, most of these operating systems gained POSIX compatibility subsystems, at which point they gained byte stream files as exist on mainstream systems. IBM systems generally support tagging files with a code page, so the files can be a mix of EBCDIC and ASCII, and the OS will perform translation between them in the IO layer (so an application which uses EBCDIC at runtime can read an ASCII file as EBCDIC, without having to manually call any character encoding conversion APIs, or be explicitly told whether the file to be read is EBCDIC or ASCII). Newer applications make increasing use of the POSIX-based filesystems, but older applications still mostly store data (even text files and program source code) in the classic record-oriented file systems.

From what I understand, the most common place EBCDIC NEL would be encountered in the wild was EBCDIC line mode terminal connections (hard copy terminals such as IBM 2741 and IBM 3767).

titwank2• 2 years ago

I wondered if clang has the same property, but it's explicitly coded as 10 (in lib/Lex/LiteralSupport.cpp):

    /// ProcessCharEscape - Parse a standard C escape sequence, which can occur in
    /// either a character or a string literal.
    static unsigned ProcessCharEscape(const char *ThisTokBegin,
                                      const char *&ThisTokBuf,
                                      const char *ThisTokEnd, bool &HadError,
                                      FullSourceLoc Loc, unsigned CharWidth,
                                      DiagnosticsEngine *Diags,
                                      const LangOptions &Features) {
      const char *EscapeBegin = ThisTokBuf;
      // Skip the '\' char.
      ++ThisTokBuf;
      // We know that this character can't be off the end of the buffer, because
      // that would have been \", which would not have been the end of string.
      unsigned ResultChar = *ThisTokBuf++;
      switch (ResultChar) {
    ...
      case 'n':
        ResultChar = 10;
        break;
    ...

titwank2• 2 years ago

Similarly in GCC it is hardcoded, with a choice of ASCII or EBCDIC (in gcc/libcpp/charset.cc):

    /* Convert an escape sequence (pointed to by FROM) to its value on
       the target, and to the execution character set.  Do not scan past
       LIMIT.  Write the converted value into TBUF, if TBUF is non-NULL.
       Returns an advanced pointer.  Handles all relevant diagnostics.
       If LOC_READER is non-NULL, then RANGES must be non-NULL: location
       information is read from *LOC_READER, and *RANGES is updated
       accordingly.  */
    static const uchar *
    convert_escape (cpp_reader *pfile, const uchar *from, const uchar *limit,
                    struct _cpp_strbuf *tbuf, struct cset_converter cvt,
                    cpp_string_location_reader *loc_reader,
                    cpp_substring_ranges *ranges, bool uneval)
    {
      /\* Values of \a \b \e \f \n \r \t \v respectively.  */
    #if HOST_CHARSET == HOST_CHARSET_ASCII
      static const uchar charconsts[] = {  7,  8, 27, 12, 10, 13,  9, 11 };
    #elif HOST_CHARSET == HOST_CHARSET_EBCDIC
      static const uchar charconsts[] = { 47, 22, 39, 12, 21, 13,  5, 11 };
    #else
    #error "unknown host character set"
    #endif

      uchar c;
    
      /* Record the location of the backslash.  */
      source_range char_range;
      if (loc_reader)
        char_range = loc_reader->get_next ();
    
      c = *from;
      switch (c)
        {
    ...
        case 'n': c = charconsts[4];  break;
    ...

johnisgood2• 2 years ago

If so, this should answer the question where the compiler knows it from?

ProofHouse• 2 years ago

A lot LOT better content than his stupid wanko manko perv post

adamrezich• 2 years ago

This is very interesting. I don't know if it was the intent, but Jon Blow doesn't plan to self-host the Jai compiler anytime soon, and I wonder if the rationale is at least in part due to things like this. It's interesting how he doesn't see self-hosting as any sort of immediate goal, quite unlike Rust and Zig.

Whence '\n'?

Related Stories