pixelbeat__

A direct link to the Ubuntu post

https://discourse.ubuntu.com/t/an-update-on-rust-coreutils/8...

Some of this elegance discussed from a programmatic point of view

https://www.pixelbeat.org/docs/utf8_programming.html

LOL (I used to work for Meta, so appreciate the facetious understatement)

It seems dotslash would complement uv well

https://dotslash-cli.com/docs/

DotSlash to get the interpreter for your platform, and uv to get the dependencies.

Perfect for corporate setups with custom mirrors etc.

GRUB is mentioned but not detailed.

Here are some details: https://www.pixelbeat.org/docs/disk/

https://github.com/coreutils/coreutils/commit/14d24f7a5

That bring GNU date(1) line coverage from 79.8% to 87.1%

I see you use flags to determine if a file needs syncing. When we used fiemap within GNU cp we required FIEMAP_FLAG_SYNC to get robust operation.

(We have since removed the fiemap code from cp, and replaced it with LSEEK_DATA, LSEEK_HOLE)

The benchmark is against unibyte text. You would get more accurate results by doing `export LC_ALL=C` in your benchmark script

Note the terminal -> HTML conversion used to serve wttr.in is based on https://github.com/pixelb/scripts/blob/master/scripts/ansi2h...

A similar tool for ini files https://www.pixelbeat.org/programs/crudini/

I wrote about ASCII and UTF-8 elegance at:

https://www.pixelbeat.org/docs/utf8_programming.html

rxzec

The commit with more details on that perf change is: https://github.com/coreutils/coreutils/commit/fcfba90d0d27a1...

A summary of other changes just released in GNU coreutils 9.5 are:

- mv accepts --exchange to swap files

- env accepts --argv0 to override command name

- od supports half precision floating point formats

- timeout fixes various races

- chmod -R avoids symlink replacement attack

For all the release details see: https://lists.gnu.org/archive/html/coreutils-announce/2024-0...

(Hi Giuseppe!)

Right, the standard centos system binaries are used to provision services and manage hardware.

The internal services linking against the runtime libs you mention, are actually linking against about 2000 built from source packages, and are essentially a separate distro (with a distro in this sense being an ABI compatible set of libs running on a kernel)

Use ldd on any service binary to see that it's linked against a separate distro. (It's best to use the ldd from the corresponding platform).

There are actually three distros on each host. Current runtime platform, previous runtime platform, centos platform.

It's worth noting that this discusses the centos based provisioning and hardware management platform.

The actual distribution used to _run_ all Meta backend services is completely separate and built from source (it does share the (non centos) kernel). This is done for flexibility, performance, service isolation reasons

Reminds me of my website from 2006 (not entirely reproduced) at https://web.archive.org/web/20060208131821/http://www.pixelb...

Definitely not

All the main platforms have realpath(1). It was added to GNU Linux over 10 years ago now, to aid portability, and be a more natural command to access this functionality than readlink(1)

Yes, macos takes its utils from FreeBSD.

An interesting factoid is that FreeBSD/macos sort(1) was using GNU code until recently, since this is quite tricky to implement. Eventually it was reimplemented for GPL avoidance reasons.

We do consider macos though, and ensure all tests pass on macos for each release

czkawka is basically a rust port of my python FSlint tool, which I no longer have time to maintain: https://www.pixelbeat.org/fslint/

Padding is only required if concatenating / streaming encoded data. I.e. when there are padding chars _within_ the encoded stream.

Padding chars at the end (of stream / file / string) can be inferred from the length already processed, and thus are not strictly necessary.

Note how padding is treated is quite subtle, and has resulted in interesting variations in handling as discussed at: https://eprint.iacr.org/2022/361.pdf

We generally give credit if at all possible. Even if a patch is extensively adjusted, we'll commit under the original author's name

I optimized yes because:

* The fast version is still simple enough

* The general functionality being provided is to output arbitrary data repeatedly, and it can be useful to do this as fast as possible

The gnulib manywarnings module is useful to add as many warnings as appropriate in a portable manner

https://www.gnu.org/software/gnulib/manual/html_node/manywar...

That's used in dev mode for various projects like GNU coreutils etc.

When distros etc. are building though these warnings are not used. It's worth stating as I've seen this issue many times, that -Werror at least should not be used by default for third parties building your code

There have been various attempts to improve on strcpy() etc. The current thinking is that strscpy() is the best general interface to this functionality. See the "improving ... strcpy" section at

https://www.pixelbeat.org/programming/gcc/string_buffers.htm...

String handling in C has many gotchas indeed. Here are some of my notes on the subtleties:

https://www.pixelbeat.org/programming/gcc/string_buffers.htm...

Oh I will reference this from https://www.pixelbeat.org/programming/shell_script_mistakes.... where I also advise against this common archaic idiom

This technique is useful for running things that normally run quickly, but sometimes need more time.

The technique is used extensively in the GNU coreutils test suite, as detailed in the "performance" section at https://www.pixelbeat.org/docs/coreutils-testing.html

You're not considering the scale.

We have faster iteration than upstream distros. More flexible. More tuned. I won't give exact details but 1% CPU gives extremely significant monetary savings, and there are at least 15% savings from static linking, PGO, LTO, more appropriate `-march`, more appropriate CPU security sharing considerations, ...

Billions of dollars per year, in essentially electricity and required systems savings (considering the scale of serving 3 billion users a day).

Also devs get access to the latest compilers, language levels, and libs, completely independent of distros, who have a more general compat issue to contend with. Considering there are about 30k tech in Meta this value also multiplys up.