pixelbeat

How the latest coreutils leveraged terminal hyperlinks to directly link to the full online documentation

Phoronix didn't link the full NEWS: https://savannah.gnu.org/news/?id=10816

Some extra notes...

Accurate (cgroup aware) nproc is important, as detailed in: https://danluu.com/cgroup-throttling/

The new base58 encoding performs well being around 500 times faster than arbitrary int implementation internal to cpython 3.13 for example

GNU coreutils can be built as a single binary with ./configure --enable-single-binary. One can install this variant on Fedora for example with the coreutils-single package, and this is used in some container images.

FWIW GNU coreutils' shuf uses reservoir sampling for larger inputs to give bounded memory operation

42/2

For similar functionality have a look at crudini, which is a utility designed for interacting with ini files from shell scripts:

https://github.com/pixelb/crudini

Nice article. See also: https://www.pixelbeat.org/programming/stdio_buffering/

It's also worth mentioning a recent improvement we made (in coreutils 8.28) to the operation of the `tail | grep` example in the article. tail now notices if the pipe goes away, so one could wait for something to appear in a log, like:

    tail -f /log/file | grep -q match
    then_do_something

You might find my "symlinks, hardlinks and reflinks explained" article useful: https://www.pixelbeat.org/docs/unix_links.html

The vi vs emacs thing permeates a lot of tools. I find it useful to map all tools to be consistent with one or the other. Personally I use vi. Unfortunately setting vi mode for readline has a few caveats, but I was able to work around all of them with the settings in:

https://www.pixelbeat.org/settings/.inputrc

One might be wondering why 'test' and '[' are separate binaries at all. This is to give maximum flexibility wrt copying binaries etc. Note one can build coreutils like busybox as a single binary, by just `./configure --enable-single-binary` at build time. This is already available as an option on Fedora at least through the coreutils-single (1.2MB) package.

Sounds like a case of: https://www.pixelbeat.org/programming/sigpipe_handling.html

uutils uses the coreutils test suite, so it makes sense to add a test case for that, which uutils will eventually get to. I'll do that now.

yes(1) is the standard unix way of generating repeated data. It's good to do this as quickly as possible. I really don't understand why so many get annoyed with this code. 130 lines isn't that complicated in the scheme of things.

Nice. This handles the mangled example I discussed at:

http://www.pixelbeat.org/docs/unicode_utils/

GNU seq uses this trick:

slow path:

    seq -f '%.1f' inf | pv > /dev/null
     ...[  12MiB/s]

    seq inf | pv > /dev/null
     ...[ 491MiB/s]

The GNU variant was discussed recently at: https://news.ycombinator.com/item?id=14542938

The commit that sped up GNU yes has a summary of the perf measurements: https://github.com/coreutils/coreutils/commit/3521722

yes can be used to generate arbitrary repeated data for testing or whatever, so it is useful to be fast

Very good point. I added a whole section to the article, implementing the counting lines example with `make -j`, which performs just as well as `xargs -P`

That really is a bash bug IMHO I've discussed this with upstream bash at http://lists.gnu.org/archive/html/bug-bash/2015-02/msg00052....

I've some general notes on SIGPIPE mishandling at http://www.pixelbeat.org/programming/sigpipe_handling.html

Yes this is true. Note BSD supports this better with Ctrl-T to generate SIGINFO, which one can send to any command even if not supported, in which case it's ignored. Using kill on linux, and having that kill processes by default is decidedly more awkward.

It's also worth noting the separate "progress" project which can be used to give the progress of running file based utilities.

We generally have pushed back on adding progress to each of the coreutils for these reasons, but the low overhead of implementation and high overlap with existing options was deemed enough to warrant adding this to dd

Yes "D" is not for disk/drive/device/...

It comes from the DD (data definition) statement of OS/360 JCL, and hence why dd has the unusual option syntax compared to other unix utils

BTW if you are using dd to write usb drives etc. it's useful to bypass the Linux VM as much as possible to avoid systems stalls, especially with slow devices. You can do that with O_DIRECT. Also dd recently got a progress option, so...

    dd bs=2M if=disk.img of=/dev/sda... status=progress iflag=direct oflag=direct

http://www.pixelbeat.org/docs/coreutils-gotchas.html#dd

This machine isn't available yet :) But since Lenovo is the Red Hat corp laptop vendor, support is generally excellent. I've used Fedora 23+ on the Third gen X1 carbon with zero issues.

Yes ignoring .gitignore contents is a very useful feature. That could be added quite easily to findrepo though would probably have to be an opt as it is often useful to search intermediate build files etc. that aren't checked in. Whereas `git grep` handles the other use case of only searching checked in files.

Interesting info wrt efficient unicode processing for \w and -i.

cheers

Thanks for the detailed comparisons and writeup.

I find this simple wrapper around grep(1) very fast and useful:

http://www.pixelbeat.org/scripts/findrepo

Note the #status anchor is best removed from the url

Yes some of these useful functions should be bound by default IMHO. Here are my settings to bind ctrl-{left,right} and {up,down} to more useful defaults. I.E. jump as the shell tokenizes the line, and search history for what's already typed respectively

mosh is fantastic for me when working remotely over vpn. I don't have to worry about reconnections, disconnections, packet lag (due to local echo feature). mosh in combination with screen (or tmux) gives scrollback support + other features

I use vim to get colorized pages and drill down to other referenced man pages. Search for MANPAGER in:

http://www.pixelbeat.org/settings/.bashrc

FSF collects money for infrastructure etc.

Generally the GNU project maintainers are not paid by the FSF. Mostly they're paid through the likes of Red Hat etc., with varying overlap between day job and maintainership

Yes exactly. For example GNU coreutils auto enables -Werror for devs when running from a git repo, whereas tarball releases do not add that option. See:

http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=blob;f=confi...

Note mv tries reflink by default (when moving files across BTRFS subvols) since doesn't need a separate copy of the data. Now as storage systems evolve there are less guarantees that one does get multiple copies, with deduplicating at lower layers etc., and therefore cp may change at some stage to reflinking by default, especially as clone_file_range() moves to the VFS level. Actual data redundancy would then achieved at a higher level with separate file systems, devices, data centers, ... where arguably it needs to happen now anyway.

The approach that http://www.pixelbeat.org/fslint takes is to md5 and sha1 the files to avoid false positives.

The core of the duplicate detection is the shell script here:

https://github.com/pixelb/fslint/blob/ffcd3b85/fslint/findup...