Decoded: GNU coreutils (2019)

http://maizure.org/projects/decoded-gnu-coreutils/

by goranmoomin • 5 years ago

288 38 5 years ago

maizure.org

tyingq4• 5 years ago

Really lovely work. I'm curious if the png flowcharts are generated from data, or hand drawn.

Edit: Also, some easter egg looking thing at the bottom right of the page:

  ##*#**##****#*#**/\##*###*****#**#*#*#**#******#**#*#*####*#*##*

</div>

Edit: Fixed asterisks, I think.

jedimastert• 5 years ago

  ##*#**##****#*#**/\##*###*****#**#*#*#**#******#**#*#*####*#*##*

HN's formatter is having a time with that many asterisks

I don't recognize the format, can someone help me out?

tyingq4• 5 years ago

It's 64 characters. I'd guess binary if that /\ wasn't in the middle.

mshockwave• 5 years ago

Very interesting way to visualize some of the most important cornerstones in *nix systems

ojnabieoot2• 5 years ago

Very nice work and much easier than trawling through the repository.

Some ignorant and probably cliched musing: when I look at small utilities like these I am always struck by a seeming distinction between best practices for little C programs versus best practices for large C applications (the author of the post touches on this ad well).

In particular, the explicit flow (including goto) and “pedantic” style is actually quite appropriate for something < 1000 lines and where the expected behavior is extremely well understood. In cases like pwd, mkdir, etc, trying to abstract too much is arguably a mistake for maintainability and understanding.

I say all this as an immutable functional-first dev who hasn’t done much native code :) And I think the various type-safe / memory-safe / etc versions of these tools are worth developing. But there’s something to be said about well-optimized native code that clearly “does what it says on the box” in a way that’s accessible to anyone who understands basic Linux programming - even if they can only contextually read C code.

(My only real gripe is typographic / linting related, mostly due to being a whippersnapper).

setpatchaddress• 5 years ago

Could not disagree more about goto. Small programs always turn into larger ones. And what you have at the end if you're not from the beginning using practices appropriate for larger programs is spaghetti code.

I'm not criticizing it in context -- a lot of this code dates back to the mid 80's if I'm not mistaken. But always write new code using scalable idioms.

overboard2• 5 years ago

If this program has remained small for 40 years, then maybe not all small programs turn into larger ones.

monocasa3• 5 years ago

I'd like to see the use of goto.

There's two 'allowed' uses in C that are common and represent good code even today. goto error cleanup stubs, and goto in virtual machine dispatch loops.

The size of the codebase doesn't really matter for those cases; they're largely considered the idiomatic way to go about the problems they're trying to solve.

ojnabieoot2• 5 years ago

I agree with you in general. But I think in this specific case it’s a bit more complicated: the downsides aren’t as bad as they normally would be, and the use of primitive flow constructs arguably has an advantage in this domain:

POSIX and similarly stuffy requirements (even if “soft”) means that this code is fairly static. While there is some bloat in the pragmas, etc., these applications are necessarily slow to change and I think it’s reasonable to say that they won’t suffer from feature bloat anytime soon. So the normal software risk considerations are a bit different here. Further, any changes to the code will be fiercely reviewed, and the individual programs are small enough that increases in complexity will be quickly spotted. Relatedly, these programs are small enough that, if a refactor to more structured code were necessary, the work would be quite feasible. So while the risks of goto are real in any C program, in practice I think they’re quite minimal here.

And I do think you’re missing an advantage. These are core userspace functions that perform safety- and security-critical kernel interactions. So I definitively agree there is a strong argument to use safe code, modern abstractions, and so on. This is especially true for modern PCs that really can afford to spend a few extra cycles creating a folder.

But a modern code construct, correctly applied, is only as safe as the compiler. This is not guaranteed! A common “gotcha” with buggy C compilers is inappropriately pruning instructions because the compiler optimizes away a loop or else statement. It is hardly a frequent issue but similar bugs have shown up in recent gcc/clang releases. And in particular core developers who are working on operating systems are more likely to be using shaky C compilers.

Using gotos and ugly global state has the distinct advantage that generated assembly tends to have less “surprises.” If there is a bug in the compiler it will be less well-hidden; if there is a bug in the program then there is less mental work between analyzing the C and analyzing the disassembly for debugging.

Again, in general I think you’re correct and that my argument is ultimately more of a judgment call.

EDIT: I didn’t really want to address any structural advantages of goto for, e.g. exception handling via breaking loops earlier, etc. I am not a domain expert enough to comment appropriately but it does seem there are cases where properly abstracted cleanup code in C is more spaghettified than a goto: https://lkml.org/lkml/2003/1/12/203

kiwidrew• 5 years ago

This is in keeping with the style of the original Unix utilities.

Having a handful of global variables reduces the amount of stuff being passed around from function to function; utilities don't need to worry too much about free()ing dynamic allocations, since that gets cleaned up on exit anyways; none of the code has to be re-entrant, because each invocation of the utility is running in its own process.

ufo• 5 years ago

The biggest takeaway for me is that I learned about the existence of some utilities that I had never known were there. Specially "factor" and "tsort".

vram222• 5 years ago

There are many cool / useful less-known utilities in GNU / Linux.

Check man7.org for good, though brief info on many of them.

I had explored many of them a while ago.

Maintained by Michael Kerrisk, author of The Linux Programming Interface, a kind of reference bible for Linux APIs and system calls.

Edit: many of which are used in making such utilities.

MaxBarraclough• 5 years ago

Don't forget recutils.

https://www.gnu.org/software/recutils/manual/A-Little-Exampl...

https://en.wikipedia.org/wiki/Recfiles

rustyminnow• 5 years ago

Here's the list of coreutils pages for anybody else interested: https://man7.org/linux/man-pages/dir_by_project.html#coreuti...

rwmj• 5 years ago

There's also "moreutils"[1] which is a set of useful additional tools. "errno" is indispensable if you're a Linux programmer.

[1] https://joeyh.name/code/moreutils/

rank0• 5 years ago

That's pretty neat! But I feel some of those tools can just be implemented with coreutils, pipes, and redirects...

cillian64• 5 years ago

I use sponge from moreutils all the time. You know how you always want to ‘sort <file >file’ or something but that doesn’t work because file gets truncated before it gets read? With sponge you can do ‘sort <file | sponge file’

gautamcgoel• 5 years ago

I'm blown away by this project. What a great way to learn about the coreutils, and also see how C is written in the real world! I'm curious how the author made the diagrams explaining each utility - did he use Inkscape?

mraza007• 5 years ago

OMG I’m so surprised I was going to post a question on HN yesterday that how can I learn about GNU Coreutils and today I wake up see this

What a coincidence!!!

Truly an amazing resource on GNU coreutils

gbajson• 5 years ago

I have just spent 5 minutes trying to find any useful use cases for 'vdir'. Does anyone of have idea why other 'ls' has been created?

JonathonW• 5 years ago

Someone years ago wanted a shortcut for `ls -lb`, didn't want to change aliases on every machine they use, and had access to the coreutils source?

In GNU coreutils, it's not really "another" ls; it's the same source code with a preprocessor macro set differently (ls, dir, and vdir all share the same source).

dang• 5 years ago

Discussed at the time:

Decoded: GNU Coreutils - https://news.ycombinator.com/item?id=20328650 - July 2019 (55 comments)

psychoslave• 5 years ago

"How is GNU `yes` so fast?" was already discussed on this topic: https://news.ycombinator.com/item?id=14542938

bogomipz• 5 years ago

What a fantastic post! If the author is reading - please post more of these Linux "decoded" series. You have a great visual style and the content is wonderfully succinct. I highly recommend reading the "Decoded: The 'top' utility (procps)" post as well:

http://maizure.org/projects/decoded-top-procps/index.html

Decoded: GNU coreutils (2019)

Related Stories