We are stuck with egrep and fgrep (unless you like beating people)

https://utcc.utoronto.ca/~cks/space/blog/unix/EgrepFgrepStuckWith

by GrumpySloth • 4 years ago

310 341 4 years ago

utcc.utoronto.ca

kevincox2• 4 years ago

What problem is this warning trying to solve? Are these two symlinks too much maintenance burden? Or is the check in the code hurting the code quality? Is the extra check at startup ruining performance?

I'm usually in favour of having one way to do things but in this case, with this much legacy it just doesn't seem worth it.

roenxi• 4 years ago

This is free software. It is legitimate to consider convenience for the maintainers. If they don't want to maintain two symlinks then they are empowered to make that call.

If anyone thinks it is a big enough problem that they want to fork the software they can, or the distros can maintain their own symlinks. But I think in this case the simple answer is if the maintainer doesn't want it in the source tarball then it isn't going to be there and that is more than sufficient a justification.

Complaining is probably more reasonable than asking for a justification here.

raverbashing2• 4 years ago

It is legitimate to consider convenience for the maintainers.

Yes, but the convenience of... creating 2 symlinks and adding the flag based on argv[0]?

Sorry, I don't buy it. Wanting to die on such a tiny hill has GNU written all over it though

Ensorceled5• 4 years ago

I don't know when society in general just said "Fuck It" to the idea of stewardship; but we now have it ingrained that people in positions of trust and power (whether volunteered, elected or appointed) are not morally or ethically beholden or responsible to the communities they have taken it upon themselves to represent.

At least we used to pay lip service to that ideal.

colonwqbang• 4 years ago

It can't be that much work to maintain a symlink. The idea that it could be less work to remove a standard feature that has been part of Unix for several decades, has no connection with reality.

There is no reasonable explanation for this decision except that somebody thought that having both egrep and grep -E was "ugly" according to their own personal sensibilities.

_jal2• 4 years ago

It is legitimate to consider convenience for the maintainers

In which case, time to fork it.

I don't fucking care who you are, you do not break grep and keep a privileged position on my machine.

fleddr• 4 years ago

What 'maintenance' goes into symlinks that already exist? The only thing one needs to do is nothing at all. It is in fact more 'maintenance' to delete them.

MobiusHorizons2• 4 years ago

What exactly do you imagine the maintenance being? egrep and fgrep are already shell scripts, and those scripts haven't been edited since 2015. If not needing to change something for 7 years counts as a maintenance burden, sign me up.

tgv2• 4 years ago

Unfortunately, this seems symptomatic of GNU's attitude: we've come up with something silly and it's going to screw some people over badly in the least agreeable moment, but we're too stubborn and proud to admit it. Take it, or buy a Windows 11 license.

barrkel4• 4 years ago

The problem is people being wrong, and the upside is feeling good about punishing them. IMO.

saalweachter2• 4 years ago

At a guess, it isn't actually the symlinks, it's the argument parsing they're trying to simplify.

Old programs have stupidly complex argument parsing. If you can pass your arguments as "cmd foo bar path", "cmd -f bar path", "cmd -fbar path", "cmd path --foo=bar" or "fcmd bar path" and "cmdf path bar", it can be really convenient for users who can structure their commands in the way that makes the most sense for them.

But it can be really frustrating to maintainers who are maintaining -- and testing! -- a thousand lines of bespoke argument parsing for a ten line function.

It's really, really tempting to define a simple syntax for argument parsing, turn it into a library, and reduce your pile of shell commands to a few lines of argument configuration and a function call.

masklinn• 4 years ago

At a guess, it isn't actually the symlinks, it's the argument parsing they're trying to simplify.

There’s no parsing. GNU’s egrep and fgrep are trivial shell scripts:

    #!/bin/sh
    exec grep -E "$@"

Forge36• 4 years ago

Why aren't these split into separate packages? If a distro wants to drop them, they can still be installed. If some distro wants to include them with a warning that's also fair game.

forgotpwd164• 4 years ago

Why aren't these split into separate packages?

Because they're the same thing. What all is about are the `xgrep` commands being symlinks to `grep`. Though I guess you can have packages that just add the symlink.

If some distro wants to include them with a warning that's also fair game.

Some distros already do what is recommended in release notes. Rather being symlinks they're wrapper scripts. E.g. in Nix the `fgrep` and `egrep` are just `exec ${nixpkgs.gnugrep}/bin/grep -F "$@"` and `-E` respectively.

ksherlock• 4 years ago

GNU egrep and fgrep are wrapper scripts (and have been for 10 years for more). Wrapper scripts that now warn you not to use them.

https://git.savannah.gnu.org/cgit/grep.git/tree/src/egrep.sh

thepawn1• 4 years ago

alias fgrep='grep -F'

VWWHFSfQ• 4 years ago

the issue is that programs rely on the existence of fgrep in its PATH. your shell alias doesn't fix that.

jonas-w• 4 years ago

It is a bit hacky but you could put "grep -F" in a script file in your path.

emptyparadise• 4 years ago

    #!/bin/sh
    exec grep -E "$@"

xpressvideoz• 4 years ago

The egrep and fgrep commands have been deprecated since 2007.

Isn't 15 years more than enough time to handle the deprecation?

angry_octet• 4 years ago

No. Make install a grep compatibility library if necessary, but don't change output syntax.

loloquwowndueo2• 4 years ago

Sure now that I’ve finished rebuilding all my dependencies to use 64-but time_t and migrated all my services to IPv6 I can finally take some time to finish the great *grep deprecation :)

I’m just joking :)

ht85• 4 years ago

To be fair, the pain of migrating time_t is nothing next to what I will feel trying not to type `fgrep` every time.

Beltalowda9• 4 years ago

I betcha most people didn't even know they were considered "deprecated".

ozim• 4 years ago

There is so much outdated info on the internet in various forms that it is hard even to realize what is "proper modern way" of doing things unless you really are into Linux \ config stuff.

I am casual user so I can get around system but modern ways always surprise me when I finally find out about it.

kibwen• 4 years ago

And now they do, so perhaps it's working as intended. :P

gw99• 4 years ago

Yes. This is the first I've heard of it and I've got a ton of scripts that are probably going to break.

LinuxBender• 4 years ago

I learned about the deprecation from ShellCheck [1] that warns if using egrep instead of `grep -E`. That tool deprogrammed many of my bad habits. I had never seen any discussions about it otherwise.

[1] https://www.shellcheck.net/

dspillett• 4 years ago

I for one didn't know they were properly deprecated.

Though I've not used either since uni over two decades ago¹ on a somewhat off-standard Unix a few of the machines ran, so until this thread I can't say I was remembering that they existed at all.

----

[1] using grep -E when needed since²

[2] and presumably grep -F too though I don't remember ever actually doing that

vultour• 4 years ago

Most people shouldn’t have been using this in the first place. If you’ve been taught to use these in the last ~10 years then someone, somewhere, has failed horribly.

ajross4• 4 years ago

Not when the command in question has been a standard part of all Unix shell environments since 1977, no. Is that a serious question?

https://medium.com/@rualthanzauva/grep-was-a-private-command...

What actually happened here is that POSIX skipped it. It never entered a standard, even though it was (literally!) in every OS and available to everyone. But no one cared that it wasn't in some arbitrary standard, because it was always there. For half a century!

I dare say that "fgrep" and "egrep" have more active users (both interactive and scripted) than "awk" or "ed" or "od" or "bc", all of which are still around.

netfl0• 4 years ago

Please put it back.

ufo2• 4 years ago

The blog makes two main points: 1) adding new error messages causes compatibility problems; 2) some people are used to typing "egrep".

On the first point the author only gives hypothetical examples. I feel the argument might have been more compelling if we could see some concrete examples of things that break with GNU Grep 3.8.

As for the second point, I find it less convincing than the first one. If it's just the muscle memory then an alias would be an acceptable workaround. And I doubt that "everyone in the world" would want such an alias, as the author suggests.

airza• 4 years ago

It seems pretty simple, piping bash commands into other bash commands and other text stream juggling is a pretty typical use of these commands and so changing what stream is output can change the behavior of consumers of the output of these functions.

I haven’t done anything with fgrep and egrep before but piping grep into another grep for more complex classes of text search is something i use a lot.

tuetuopay2• 4 years ago

It's more than likely the warning will be printed to stderr, not out, so there will be no impact on the actual work done.

mannykannot10• 4 years ago

With regard to the first point, the examples may be hypothetical, but they are also very plausible.

When a change has little or no objective benefit, I feel the burden of demonstrating that it is harmless falls on those making the change.

As has been pointed out elsewhere, this is free software and the maintainers are free to do whatever they like. That does not stop others having an opinion about it, especially when it is in the form of constructive criticism.

ufo2• 4 years ago

Sure, but it would still be nice to have at least one such example. Looking at the rest of the discussion thread here on HN as of now it's still only hypotheticals.

ajross4• 4 years ago

On the first point the author only gives hypothetical examples

I have scripts everywhere, some of them 20 years old or more, that use fgrep. For years and years it was a "best practice" thing if you were checking for a fixed string (so that you didn't accidentally match on a "." or whatever by forgetting it was a regex).

teddyh12• 4 years ago

You have two options: The first option is to simply replace "fgrep" with "grep -F" everywhere in all your scripts, which is correct but is more work than your other option, which is to add your own "fgrep" script somewhere in your path.

Any of these options seem reasonable to me.

cestith6• 4 years ago

The shell happens to be a programming tool and not just an interactive command interpreter. There are whole books on how to write shell scripts portably across various Unix-y platforms. Many of the examples in those books will now throw warnings on systems using the GNU tools.

bitofhope3• 4 years ago

If a book on writing portable shell scripts across *nix platforms depends on commands not specified by POSIX, I don't think those books are doing their job very well.

javier_e062• 4 years ago

scripts will be break. Strange outputs will appear. Devops will rip the gowns and douse ashes on their heads. Patches will arrive and a years later someone will make a joke about fgrep like the jokes about ed, the GNU ed line editor.

rgoulter3• 4 years ago

Hopefully Devops will be able to make use of nice tooling like linters (e.g. shellcheck for bash) which can hope to catch such problems.

TacticalCoder• 4 years ago

Does the warning go to stdout or somewhere else?

Beltalowda9• 4 years ago

stderr

ur-whale2• 4 years ago

stderr

For now.

liveoneggs• 4 years ago

I use egrep and fgrep regularly. I guess I am just getting too old for GNU.

throwaway67743• 4 years ago

Funny, I've never ever used egrep or fgrep in the 20 odd years I've been operating systems, always preferred explicit flags to aliases or argv[0] evil.

chriswarbo2• 4 years ago

I get the annoyance, for both the GNU maintainers and distro maintainers/sysadmins. I'm not too wedded to either outcome, but my take is that we should avoid global mutable state:

- If you're relying on random globals (like the path /usr/bin/egrep) to (a) exist and (b) behave in a certain way, then don't mutate them. Stick with known-good versions, and treat updates like any other code change (review, test, etc.). This is the usual case for legacy systems. In other words, don't blindly run `apt-get -y upgrade` on systems which are meant to be stable.

- Alternatively, don't use globals: install grep in its own directory (e.g. via the `--prefix` argument of `./configure; or using a sledgehammer like chroot), and treat that location as a capability (as per capability-security). Programs/scripts which need that grep must be given its location explicitly (either for direct calls, or by having it prepended to their $PATH). If you want to use an updated version elsewhere, just install that to a different location; no need to touch the existing setup.

(Shout-out to Nix for doing this via ./configure and $PATH. Runner-up prizes for chroots/jails/zones/containers/VMs/etc.)

AshamedCaptain3• 4 years ago

Both of your suggestions are basically equivalent to "think before you upgrade" and/or "don't upgrade". This is the second elephant in the room with all these distros that "don't use globals" and/or statically link everything (or do an analogue to that, like nix). There's very little benefit for desktops. So the upgrade to dependency X breaks component Y, and you are forced not to update X, or at least, Y's copy of X. Great. What do you do now? Swim away from upstream? Stay on outdated components? The situation is as unatenable long-term as it is on a regular distro...

The first elephant in the room is that generally you _do_ want most dependencies to be global state. Is a situation where every program is using its own version of the graphics toolkit, with different theming issues or even different themes altogether, really ideal for a desktop situation? What about libevent -- so that your mouse's wheel has one acceleration curve in some programs and some other speed in some other programs ? Or what about ibus, where generally having different client versions running simultaneously means your entire input system stops working ?

Even grep is likely something that you'd prefer to be global state, lest the grep exec() by a Python interpreter have different features than the one launched by your main shell.

eptcyka• 4 years ago

Yes, but Nix just solves this nicely enough.

rgoulter3• 4 years ago

The situation is as unatenable long-term as it is on a regular distro.

I may be misremembering, but isn't this pretty much the default user experience on rolling distributions like Arch Linux? -- You consult the wiki to see what changes you need to make are (or maybe see what breaks after an update), make the change, and get on with your desktop experience.

thomascgalvin2• 4 years ago

If you're relying on random globals (like the path /usr/bin/egrep) to (a) exist and (b) behave in a certain way, then don't mutate them.

It's a bit old now, but one of the principles of Twelve Factor Applications is to vendor-in all of your dependencies.[1]

A twelve-factor app never relies on implicit existence of system-wide packages. It declares all dependencies, completely and exactly, via a dependency declaration manifest. Furthermore, it uses a dependency isolation tool during execution to ensure that no implicit dependencies “leak in” from the surrounding system. The full and explicit dependency specification is applied uniformly to both production and development.

This scenario, with `fgrep` and `egrep` releasing a potentially breaking change, is exactly why this principle exists. If your software depends on "whatever fgrep happens to be lying around at the moment", your application might break the next time you build and deploy a new image. If you're pinned to a specific version, however, you're protected.

[1] https://12factor.net/dependencies

mcv• 4 years ago

You're protected, but it also means you can never upgrade anything. An application can never know what future versions of a dependency it can work with (because a new version might contain a breaking change), so it will always require versions that existed at the time of its own creation.

But those versions might contain vulnerabilities, and the new version might fix those while being fully compatible in every other way, in which case you really do want the upgrade.

Those dependencies should really declare when they change existing behaviour of the previous versions, and only then should your application refuse the upgrade.

No idea if there is a system that works that way; I can imagine it could get hideously complex.

sillysaurusx• 4 years ago

(I want to love Nix, but it’s so foreign. I do hope it becomes more of a standard. It’s just hard to learn and hard to use until you learn.)

rgoulter3• 4 years ago

I'd prefer to qualify that a bit:

When you can get Nix to do what you want, it's wonderful.

When you hit a barrier (e.g. a program you want isn't packaged), that's when Nix is very hard to use, and requires learning.

ilc• 4 years ago

I went over to Nix cold turkey for my new work laptop. Took about a week or two, to get it where I wanted it. The real issue I hit is that Nix is so alien to working with "normal" unix stuff... like stuff that assumes /usr/bin/grep exists etc, I had issues, with the existing code-base I was hired to work on.

So, I went back to Fedora. But I'd goto Nix again in a heartbeat. It is a great system, and if you start a company with it day 1... there will be a bit of heartburn as you start, but I can't see regretting it.

deeblering4• 4 years ago

Or...

And I'm going out on a limb here...

Don't support user hostile maintainers who force breaking changes to a decades old API just because they felt like it.

usefulcat• 4 years ago

This reminds me of the last time daylight savings time was changed (in the US). A bunch of hassle and software breakage.. for what? Just to get back to the same place we already were (working software)? What a waste.

ziml772• 4 years ago

I think it's a bit silly to get rid of such established aliases. Surely they aren't a real maintenance burden.

Though I can't say I entirely hate it, for the (admittedly absurd) reason that egrep = grep -E, fgrep = grep -F, but pgrep != grep -P! It's an awkward incongruity that's easy to get tripped up on occasionally.

blacklight• 4 years ago

While I hated the decision of adding warnings without much notice (which, in the case of such widely used CLI tools, is the equivalent of a breaking change), I also found an easy solution that would prevent my scripts from spitting out lots of unneeded warnings.

alias egrep='grep -E' alias fgrep='grep -F'

Now GNU developers can keep doing whatever they're doing, and I can keep doing whatever I used to do.

misnome• 4 years ago

Your scripts expand aliases?

koprulusector7• 4 years ago

    shopt -s expand_aliases

acdha4• 4 years ago

While I hated the decision of adding warnings without much notice (which, in the case of such widely used CLI tools, is the equivalent of a breaking change)

15 years isn't “much notice”? I had already stopped using those back then because, as noted in the article, they weren't standardized and so you had to work about portability across Unix installations.

It's also worth noting that this is only a breaking change if you are using the non-standard names in a context where you are trapping output. For the vast majority of people using a shell script which doesn't use the common name, they will at some point upgrade, see the warning, spend 30 seconds making the change, and never think about it again. If you're that sensitive to the extra work, presumably you also do some testing before installing new upstream releases.

EDIT: it was actually 17 years ago that the warning was added about egrep/fgrep:

https://git.savannah.gnu.org/cgit/grep.git/commit/?id=0b4859...

ourmandave• 4 years ago

Tangentially, I'm still searching for a grep for Windows you can run from file explorer.

I've resorted to right-clicking a command window and using findstr. =(

I'm sure there's also a powershell equiv but I haven't taken the time to walk the PS object tree to find it.

richard_todd3• 4 years ago

powershell has `sls` which expands to `Select-String`

koprulusector7• 4 years ago

I’ve been using grep -E for a long time, I thought I remember it being from a warning from egrep or something.. can’t really remember for sure, though.

Either way, I don’t see what the big deal is, just add an

    alias egrep='grep -E'

If you’re worried about your non-interactive shell scripts,

    shopt -s expand_aliases; alias egrep='grep -E'

and you can move on with your life.

EDIT: I must’ve been warned by shellcheck

EDIT2: here’s another one liner with one caveat being updates (also needs to be run as root)

    # unalias grep && unalias egrep && cp "$(command -v grep)" "$(dirname "$(command -v grep)")/egrep"

EDIT3: just realized for most this will only ever come up in scripts because many distros already add alias egrep='grep -E' to your shell aliases (~/.bash_aliases, ~/.zshrc, etc). Thus you may only need the shopt -s expand_aliases.

3pt141592• 4 years ago

The problem is that we don't just live on our own machines. We hop around to different boxes and just being able to type what you are used to matters. My dotfiles and other tools don't go or even work everywhere.

We shouldn't change the oldest parts of our OSes without a really good reason.

acdha4• 4 years ago

I find this reasoning interesting since it's why I _stopped_ using egrep/fgrep around the turn of the century: because those aren't standard, it wasn't uncommon to find that you depended on some behaviour which wasn't available in the version installed on some random server but it worked when you used grep.

ComputerGuru• 4 years ago

It’s a matter of principle. If you wrote a script yesterday (or a year ago) and didn’t include that line, you need to re-release it today. That’s stupid.

Also, you’re being very sanctimonious about this but what if you didn’t see this HN story. Would you know to do that for the next script you distribute?

koprulusector7• 4 years ago

Frankly I can count on two hands the number of times I’ve used egrep in my life (same for grep -E). As long as I can remember I’ve relied on sed, awk, or find -regex for filtering output with regex. I only say this because I can’t relate to those that are upset about the change.

That said, to answer your questions directly: hopefully you aren’t auto updating anywhere in production or important, so this won’t effect anything until the box has grep 3.8. Ubuntu 20.04 uses grep 3.4, for example.

And the warning prints to stderr, so honestly, I'm having a difficult time seeing this being an actual problem for more than 0.1% of users.

I’m not unsympathetic to those that it adversely affects, but I genuinely haven’t seen anyone point out any severe consequences of the change. I’m not saying it’s impossible, if someone has a real example (sorry, saying you might get a page on the weekend for a warning printed to stderr is a stretch), I’m all ears.

kazinator2• 4 years ago

fgrep and egrep have been indicated as obsolescent by the Single Unix Specification and POSIX since the late 1980's.

The man page for GNU grep says:

  In addition, the variant programs egrep, fgrep and rgrep are  the  same
  as  grep -E,  grep -F,  and  grep -r, respectively.  These variants are
  deprecated, but are provided for backward compatibility.

"Deprecated" should be read as "maybe ain't gonna be around one day". "Use interactively, or in your personal setup, but don't put into shipping code".

So if you wrote a script yesterday with these things, you're just one of those people who don't look into specs or man pages.

The scripts of coders who don't read documentation are going to tend not to be portable or reliable anyway. That doesn't mean we should break things left and right to trip people up, but in this case the writing has been on the wall for a very long time.

tyingq• 4 years ago

I don't think aliases help in one common case listed in the article. That is, scripts run out of cron, that don't read initialization files. And that case will email on unexpected stderr writes.

lizardactivist• 4 years ago

Happy to have left the mess that is grep, awk, bash, zsh, fish, etc. behind and just use Python for scripting.

42e6e8c8-f7b8-4• 4 years ago

I like hand saws and I like circular saws/Skil-saws. I use them for different things.

salawat2• 4 years ago

Python has it's own problems. Not the least of which being it being hundreds of times slower than coreutils.

I'll take learning Shell and a handful of coreutils, over praying the sysadmin has successfully navigated/allowed me to navigate all the Python footguns.

Also, good luck doing anything embedded where space is at a premium. Not all base systems have/need Python. Everything needs a shell, however.

dahart2• 4 years ago

Hehe, wasn’t the Python 2 to 3 incompatibility debacle a thousand times worse than [fe]grep? Python2 is still lingering in places even after Python 2 was removed.

dncornholio• 4 years ago

Unnecessary potentially breaking change. I'll rest my case.

cestith6• 4 years ago

" there's a difference between avoiding fossilization and the kind of minimal, mathematical purity that we see GNU Grep trying to impose here. Unix has long since passed the point where it had that sort of minimalism in the standard commands. Modern Unix has all sorts of duplications and flourishes that aren't strictly necessary, and for good reasons."

Well, really, the type of purity many of the original Unix hands desired was that each tool did one thing well, not that one tool did everything well. Some folks would say that putting the -E and -F flags in GNU grep in the first place instead of using egrep and fgrep was the wrong direction. Now GNU is wanting to further consolidate the hydra.

jagtesh• 4 years ago

What’s wrong with removing the commands and adding shell aliases instead? That sounds perfectly reasonable to me.

eg. In bash, they can be expressed as:

alias fgrep=grep -F

alias egrep=egrep -E

This sounds like a push for purity - similar to what happened in Python 3 with the move from print “xyz” (special keyword) to print(“xyz”) (standard function).

The new function requires three additional keystrokes every time it is used.

zorked• 4 years ago

It breaks my script that calls /usr/bin/fgrep

henearkr7• 4 years ago

They (the maintainer) should make the behavior of `grep` depending on the name of the executable.

Then one could simply do a `ln -s grep egrep` and be done with it.

barrkel4• 4 years ago

That's exactly what it does already. The maintainers are objecting to the existence of the links.

henearkr7• 4 years ago

Yep. Almost (it uses a wrapper script instead of a symlink), but it ends up being such a silly situation all the same...

Thanks for pointing it out though, I was not aware!

Pxtl• 4 years ago

I do wish that there was a standard pipeline for warnings, so stuff like this didn't have to go into error (bad) or output (worse). Powershell has such a feature but of course since it's not at the OS/Posix level then support is spotty.

bitofhope3• 4 years ago

You can technically pipe to streams other than &1 and &2, so you could use

  printf "WARNING: %s" "$warn_msg" >&3

But of course, nonstandard.

int_19h3• 4 years ago

Why is it bad for it to go stderr? It's supposed to be used for all kinds of app meta-output, not just errors. If you want to actually check if the command failed, that's what exit codes are for.

I did run into some Node.js code that assumed that anything printed out to stderr is a fatal error - but that's just people making wrong assumptions, not using the interface as intended.

kwhitefoot• 4 years ago

Sounds like a non-story to me. fgrep and egrep have been deprecated for nearly fifteen years and anyone who wants the old behaviour has only to create a couple of simple script files.

Any distro that wants to can easily maintain such things.

drewg1232• 4 years ago

I got this obnoxious whine from egrep on FreeBSD after my latest update. This post made me look, and I had gnu grep install as a dependency for something. Once I removed it, sanity is back and the built-in BSD egrep doesn't whine.

On FreeBSD, they are all the same:

$ ls -li /usr/bin/egrep /usr/bin/fgrep /usr/bin/grep /usr/bin/rgrep

281835 -r-xr-xr-x 4 root wheel 30736 Sep 26 14:37 /usr/bin/egrep

281835 -r-xr-xr-x 4 root wheel 30736 Sep 26 14:37 /usr/bin/fgrep

281835 -r-xr-xr-x 4 root wheel 30736 Sep 26 14:37 /usr/bin/grep

281835 -r-xr-xr-x 4 root wheel 30736 Sep 26 14:37 /usr/bin/rgrep

Izkata• 4 years ago

I didn't check where they came from, but on Ubuntu 18.04: egrep, fgrep, and rgrep are shell scripts that call grep with -E, -F, and -r

drewg1232• 4 years ago

On FreeBSD, it checks the first character of the program name and modifies behavior based on that. See https://github.com/freebsd/freebsd-src/blob/main/usr.bin/gre...

koprulusector7• 4 years ago

Thanks for sharing. I feel less silly for suggesting this now (as root):

    unalias grep && unalias egrep && cp "$(command -v grep)" "$(dirname "$(command -v grep)")/egrep

froh2• 4 years ago

why no hard link?

kevans91• 4 years ago

FWIW, I have no interest in making a similar change to bsdgrep and I can't imagine anyone else would be compelled to bother, either. I just don't see the value in removing these historical names that makes the hassle worth it.

Poiesis• 4 years ago

As a software developer, I have long-since ingrained the behavior of looking up the documentation before I assume something works in a particular way. Perhaps doubly so when shell scripting, where I have to verify if I can rely on a particular package being present, or the minimum version of something I can expect, or what the output is guaranteed to be.

When I was considering using fgrep, I looked it up; lo and behold, it's not part of the POSIX standard. So I used the option instead, and have since.

I'm a little puzzled why people have problems with these little things when it feels like it's pretty common for tools to change given enough time (for example, ifconfig and netstat to ip and ss). It's just essentially an API that gradually refines over time.

barrkel4• 4 years ago

It's not that common for tools that have been around for decades, and don't have system dependencies (like your network configuration examples do) to change.

egrep and fgrep are used on ad hoc command lines daily, and option alternatives are less convenient to use. I personally use 'egrep -o' and 'fgrep -f <(...)' all the time. I already find it tedious to have to type 'sed -r' and am half-considering wrapping it in an 'esed' variant.

Eliminating egrep and fgrep is change for the sake of change. Removing a couple of hard links from a few bin directories would be the total positive achievement, at the cost of years of work removing the utilities from scripts - or more likely, adding the links back, or variants thereof, like shell scripts which add -E / -F as required.

im3w1l2• 4 years ago

Yet another reason why shellscripts are bad huh.

grnmamba• 4 years ago

How does the article lead to this conclusion?

im3w1l2• 4 years ago

People are upset that changing literally anything on the system like adding a deprecation warning to stderr will break their fragile shellscript setup.

kristjank• 4 years ago

GNU Grep manpage says:

7th Edition Unix had commands egrep and fgrep that were the counterparts of the modern ‘grep -E’ and ‘grep -F’. Although breaking up grep into three programs was perhaps useful on the small computers of the 1970s, egrep and fgrep were not standardized by POSIX and are no longer needed. In the current GNU implementation, egrep and fgrep issue a warning and then act like their modern counterparts; eventually, they are planned to be removed entirely.

Considering how drastically GNU departs from the bare POSIX interface in some of their tools, I find it strangely pedantic how the maintainers hold up the standard in this specific case. Considering the Linux mantra of "Never Break Userspace", I can't find a good reason to drop the binaries (or the script equivalents of calling grep -E/-F) besides uprooting a historical choice that never hurt anybody (with little benefit and a plethora of potential consequences) in the name of better-late-than-never correctness.

javier_e062• 4 years ago

GNU: GNU is Not Unix. Or Linux for that matter.

froh2• 4 years ago

somehow the opengroup removed egrep and fgrep from POSIX in 2018 https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xcu...

chroot, usefull for poor man's containment of 'stuff' also went out.

and they replaced tar with pax --- which uses a tar file format

one of the things I love in Unix is the slow pace of evolution in systems tools. relevant know how doesn't rot as fast as in other environments.

radford-neal3• 4 years ago

I use fgrep all the time. Why should I spend time converting scripts, and retraining my fingers, and forever after taking longer to type grep -F? This is a breaking change that has no upside, only downside. It is idiocy.

Just to elaborate, there must be many thousands (hundreds of thousands? millions?) of shell scripts out there that use fgrep. For no reason at all (whatever maintenance burden fgrep represents must be far outweighed by the time wasted debating this issue), these scripts will no longer work. There is no guarantee that defining an alias somewhere is going to fix all your usages of fgrep. And even if it did, the cost for every single installation that is affected of someone realizing there is a problem, finding the source of the problem, figuring out that an alias will fix it, making sure that that alias gets preserved after updates, and testing that everything now works will exceed the benefit to the maintainers of grep (which is actually negative, as explained above). So looked at from a community-wide perspective, the cost is many thousands of times greater than any possible (but actually non-existent) benefit.

And as the post mentions, there are also all the books, and all the stackoverflow answers, that use fgrep, and which will now have non-working code.

As for the argument that this is open-source software maintained by volunteers, who can therefore do whatever they wish - if you are a maintainer of widely-used open-source software, and decide that you personally are not interested in working in a manner that benefits the community of users, then you should resign and let someone else take over.

barrkel4• 4 years ago

There's a certain type of person who gets pleasure from following rules exactly, particularly when other people don't follow those rules. A kind of smug moral superiority.

If they get into a position of power, they can make life worse for other people, without much regard for the actual cost of breaking such rules.

Removing egrep and fgrep is desperately petty stuff.

PaulHoule2• 4 years ago

It’s a show of how free software partisans are often out of touch.

The cost of egrep and fgrep is just two symbolic links. Some people though would rather talk about ‘libre’ and ‘free as in speech’ and ‘free as in beer’ than talk at all about the user experience. And they wonder why most people just run windows or macOS, but they’ll never understand.

gnfargbl• 4 years ago

If you look at the POSIX standard that removes fgrep & egrep [1], then you can see that most of the utilities being removed there really are quite obscure things.

In light of that, then another (perhaps more charitable) way to look at it is that the maintainers are keen to clean up stuff that they perceive as old cruft, and perhaps don't realise quite how ingrained these particular pieces of "cruft" are into the ecosystem.

(On the other hand: the same POSIX link specifies the removal of cpio and, uh, tar. I'm not sure I see that last one flying.)

[1] https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xcu...

MichaelCollins• 4 years ago

The worst is when the rule they cite isn't a real rule at all, like Lennart Poettering's defense of systemd breaking on usernames starting with a digit, which is completely valid as far as the Linux kernel is concerned but he decided such names were ackshully against the rules and therefore systemd wasn't broken, bug report closed not-a-bug.

cryptonector8• 4 years ago

We've seen a lot of this in recent years.

paulmd2• 4 years ago

There's a certain type of person who gets pleasure from following rules exactly, particularly when other people don't follow those rules. A kind of smug moral superiority.

HN commenters in a nutshell.

Not you in particular, but oh my god, the rules-lawyering and lecturing on this site is unbearable at times.

Cthulhu_• 4 years ago

I mean they could just make it an alias that calls the appropriate grep function, and mention fgrep/egrep are just aliases in all the documentation. Tools like ShellCheck could warn on use of fgrep/egrep in scripts in favor of grep if it wants things to be pure.

But the cost of keeping an alias around for decades is negligible I think. It's cheaper than breaking every existing usage in any case.

pfortuny• 4 years ago

This is exactly why Torvalds is so radical on not regressing: it is costly for him (I am sure there are many things hr would have done differently now) but ensures the users’ security.

thomastjeffery2• 4 years ago

I find it amusing that someone who uses fgrep all the time - and is presumably familiar with alias and sed - would be worried about the inconvenience of updating their shell scripts...

I think the larger issue is that people generally don't do dependency management for their scripts. Scripts are programs, and they are dependent on the shell environments they are written for.

radford-neal3• 4 years ago

I think you're not grasping the point that the cost of using 'alias' or updating scripts, though not huge, is paid by every single user, and often over and over again, every time they switch to using a different system. Against this cost for every user, one weighs the benefits of this change - which are zero. Indeed, the benefits are negative, since the maintainers must surely be wasting more time debating the change than they will ever save from it.

Of course, scripts are dependent on their environment. That's why changing the environment for no reason is such a bad idea. Nobody manages their shell scripts to account for the possibility that 'true' might cease to exist some day...

hinkley2• 4 years ago

decide that you personally are not interested in working in a manner that benefits the community of users, then you should resign and let someone else take over.

As walled gardens begin to reassert themselves, I think this advice above will be the big argument in open source over the next ten years and needs to be written on every vertical surface. Either we will have social contracts in open source, or we will have commercial contracts. There will be diminishing examples of “both” and people will continue clutching their pearls about co-opting until something changes.

“Be grateful you get anything” is good advice for people struggling to find their own peace of mind, but when it comes up in an argument that’s just abuse.

kazinator2• 4 years ago

If your reason for using fgrep is that grep -F is more characters longer, the logical thing would be to have your own function or alias which is called just f or fg.

I stopped using egrep and fgrep in scripts ~25 years ago when I found out they weren't in the standard.

grep -F is only useful when the pattern contains regex characters that should be treated literally, and since the default regex language of grep is BRE, those characters are few.

If only one special character occurs you break even on character count if you use a backslash escape:

  fgrep 'a*c'
  grep 'a\*c'

nickcw• 4 years ago

If you want modern regexp syntax then forget egrep/grep -E and use grep -P for Perl Compatible Regular Expressions. PCRE is the most commonly used regex syntax for modern programming languages, eg Python, Go (mostly) etc and grep -P will save lots of annoyance if you use one of these!

tolciho• 4 years ago

The -P is not for portability.

  $ grep -P foo
  grep: unknown option -- P

xani_• 4 years ago

    grep -P is only true way

gjvc2• 4 years ago

   grep --perl-regexp --only-matching pattern some/where/file/name

aka

   grep -Po pattern some/where/file/name

jmclnx• 4 years ago

Here is what I gave for 'fgrep' on Slackware 15:

cat /bin/egrep

#!/bin/sh

exec grep -E "$@"

Is that really hard to maintain :) I was expecting to see a link, but instead it is a shell script GNU is asking is to create. I do not know why GNU says that is hard to maintain going forward. BTW, this is grep v3.7

waffletower• 4 years ago

It is depressing, even sickening how common such exercises in tunnel-vision vigilantism occur among software engineers. The same lack of understanding of downstream impacts is found in library maintainers that use auto-updating transitive dependencies, for example.

bitofhope3• 4 years ago

I'm definitely not in favor of fossilizing Unix, but there's a difference between avoiding fossilization and the kind of minimal, mathematical purity that we see GNU Grep trying to impose here.

I can't really see the difference. Adding a deprecation warning to stderr in a non-POSIX command 15 years after deprecation notice is among the smallest possible changes to [GNU's Not] Unix I can think of. Even then, the change is trivially silenced by deleting a single line in a shell script or two[1] and seems that some distros already do that for you[2].

Yes, this will break something for someone[3] and I might well be that someone. You truly have my sympathies. But if you want to run a system for a decade and a half without ever needing changes, stick to POSIX. You can't have your fossils and eat them too.

[1] https://git.savannah.gnu.org/cgit/grep.git/commit/src/egrep.... [2] https://github.com/void-linux/void-packages/pull/39340 [3] https://xkcd.com/1172/

acdha4• 4 years ago

But if you want to run a system for a decade and a half without ever needing changes, stick to POSIX. You can't have your fossils and eat them too.

… and test before making major system upgrades. For most users this won't be arriving until the next major distribution release and it seems unlikely that this will be the only visible change in such a move. If it's that big a deal, you should have a test environment, change management, dedicated admins, etc.

aftbit• 4 years ago

On my system, egrep and fgrep are _already_ shell scripts, just changed to add the stupid warning:

    $ cat =egrep =fgrep
    #!/bin/sh
    cmd=${0##*/}
    echo "$cmd: warning: $cmd is obsolescent; using grep -E" >&2
    exec grep -E "$@"
    #!/bin/sh
    cmd=${0##*/}
    echo "$cmd: warning: $cmd is obsolescent; using grep -F" >&2
    exec grep -F "$@"

Looking at src/egrep.sh in the git repo[1], I see that before this warning was added in 2021, the last change was back in 2015. In fact, since the git repo history began in 1998, there have been a total of 4 changes to this file. Clearly the maintenance burden of these scripts is not sufficient to require the maintainers to drop them. My guess... this is being done out of some weird sense of purity over practicality. Bad call by Paul Eggert IMO[2].

1: https://savannah.gnu.org/git/?group=grep

2: commit a9515624709865d480e3142fd959bccd1c9372d1 added the warnings

yjftsjthsd-h• 4 years ago

cat =egrep =fgrep

What is that? I can't seem to find it by searching for " =" in `man bash`

professoretc• 4 years ago

In ZSH `=executable` expands to the full path of `executable`.

pwagland• 4 years ago

This is a zsh thing, it's shorthand for:

cat "$(which egrep)" "$(which fgrep)"

sprremix• 4 years ago

It's a zsh thing.

14.7.3 ‘=’ expansion

If a word begins with an unquoted ‘=’ and the EQUALS option is set, the remainder of the word is taken as the name of a command. If a command exists by that name, the word is replaced by the full pathname of the command.

From: https://zsh.sourceforge.io/Doc/Release/Expansion.html#g_t_00...

kccqzy2• 4 years ago

If you have interacted with Paul Eggert in real life (I have; he's a professor at UCLA), you would've known that he is a man that values purity over practicality. Not surprising at all.

Phelinofist• 4 years ago

Is it the same guy that maintains tzdb? https://lwn.net/Articles/870478/

mrybczyn• 4 years ago

Tool devs should get to decide on functionality - and then it's up to the distro maintainers to make aliases and organize well into their distros. Here it seems that the former are encroaching on the latter, for historical reasons?

By the way:

    ~$ bgrep
    Command 'bgrep' not found, but there are 20 similar ones.

There's not many [a-z]grep's left in the alphabet, reserve yours now :-)

robertlagrant2• 4 years ago

(I'm definitely not in favor of fossilizing Unix, but there's a difference between avoiding fossilization and the kind of minimal, mathematical purity that we see GNU Grep trying to impose here.

I'm not sure this can be reconciled with "printing a deprecation warning is a breaking change".

L1fescape• 4 years ago

GNU Grep: "Am I so out of touch? No, it's the users who are wrong"

hnlmorg11• 4 years ago

There seems to be a lot of confusion in this discussion about what the change is and how egrep and fgrep work. These are not symlinks like some have suggested but rather shell scripts. You can see the exact commit diff here:

https://git.savannah.gnu.org/gitweb/?p=grep.git;a=blobdiff;f...

I remember around ~10 years ago being told "you should never use `egrep` because it is slower than `grep -E`." precisely because the former requires extra forks() compared to the latter. However I'd counter that advice saying "if you're concerned about the performance of forks() then why are you running your code as a shell script to begin with?"... and I stand by that assessment now. In fact it would probably take me longer to type `-E` than it would for any modern system to run fork() (and I'm sure as hell not planning on using `grep` inside any hot loops!)

I think what will likely happen here is that distro maintainers will either remove that warning themselves or take ownership of the `egrep` and `fgrep` shell scripts. I'd be surprised if that warning made its way to the mainstream distros. I also wouldn't be surprised if the GNU grep maintainers did a u-turn on this change -- though it has already been committed to git for more than a year now and this is the first I've heard people complain about the change.

tomn2• 4 years ago

the overhead isn't even an extra fork, as it uses exec

hnlmorg11• 4 years ago

Good point. Though invoking $SHELL and parsing the script (as short as it is) can't be cheap either. Academically speaking of course; I'm not trying to justify the change on performance grounds.

txutxu• 4 years ago

On most context involving "using grep", shell scripts, interactive shell, etc, it's nothing.

81 Vs 128 system calls on my laptop (just printing --version)

    $ strace -fc grep -E --version 2>&1 | tail -1
    100.00    0.000000           0        81         3 total
    $ strace -fc egrep --version 2>&1 | tail -1
    100.00    0.000000           0       128         6 total

Measured with "time" 0m0.002s Vs 0m0.003s, always, testing 4 times each option.

Most situations where it could be being used, probably are surrounded by much bigger optimizations to work on.

tannhaeuser2• 4 years ago

When it comes to avoiding forks, the question is why do egrep, fgrep have to be shell scripts rather than interpret the invocation name (argv[0]) for switching behavior. Also, I find the original POSIX spec to lump everything into a single grep binary questionable since egrep, fgrep developed as extension and restriction, resp. of grep under a classic regexp discourse and it isn't clear at all that automaton construction for powerful egrep support has to live in the same binary as fgrep which isn't using automaton construction at all.

hnlmorg11• 4 years ago

When it comes to avoiding forks, the question is why do egrep, fgrep have to be shell scripts rather than interpret the invocation name (argv[0]) for switching behavior.

I don't know. I'd have taken the argv[0] approach personally. And it's the approach some other grep implementations have taken too (eg https://github.com/freebsd/freebsd-src/blob/main/usr.bin/gre...).

The argv[0] approach (to me) feels like a right kind of compromise.

ratmice• 4 years ago

The gnu coding standards has this to say: https://www.gnu.org/prep/standards/standards.html#index-beha...

Please don’t make the behavior of a utility depend on the name used to invoke it. It is useful sometimes to make a link to a utility with a different name, and that should not change what it does.

The next section provides some reasoning:

Providing valid information in argv[0] is a convention, not guaranteed. Well-behaved programs that launch other programs, such as shells, follow the convention; your code should follow it too, when launching other programs. But it is always possible to launch the program and give a nonsensical value in argv[0].

jomar• 4 years ago

I too was surprised to see these were shell scripts. I was expecting the grep/fgrep/egrep names to be hard links to the same executable that would check `argv[0]`, as the BSD implementation does.

More interesting than the commit diff is the brief discussion on the bug report from when this all happened a year ago, as referenced in the commit message: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=49996

Sigh. Time to retrain the old fingers again...

hnlmorg11• 4 years ago

Thanks for sharing that. It was an interesting read. Particularly this comment:

The irony... that one of our own tests used fgrep!

If ever there was an argument for fgrep becoming a pseudo-standard that should remain, that would have been it.

gatonegro• 4 years ago

I'd be surprised if that warning made its way to the mainstream distros.

This is the first time I'm hearing about the warning, but I thought I'd run `egrep` and `fgrep` on my Arch system. Sure enough, both result in warnings: `egrep: warning: egrep/fgrep is obsolescent; using grep -E/-F`

So it's made its way into Arch, at least. Though, like I said, this is the first time I'm hearing about this.

thayne• 4 years ago

I use arch, and I had an alias for grep to expand to `egrep --color=always` in my interactive shell. I noticed real fast. Fortunately it's an easy fix to `grep -E --color=always`.

sidenote: I wish that grep would allow running `grep -E -F`, and have the `-F` flag override the `-E` flag, rather than giving an error about conflicting options, so that I could have an alias like this to make extended regex the default, but allow me to change it with a flag.

chaxor2• 4 years ago

"if concerned with performance why are you running code as a shell script"

Rewrite grep in python before running - got it. :P

hnlmorg11• 4 years ago

That's not what I suggested. I was saying if you're writing performance sensitive code with a hot loop calling `egrep` then a smarter approach might be to use a language better tuned for performance which supports regex libraries.

Shell scripts have their place and can out perform most languages if you're writing simple logic that can run across large datasets in parallel, such as

  cat very-large-file | grep "do not want" | sed -r 's/foo/bar/' > very-large-file-refactored

(please excuse the useless use of `cat`, it's there to illustrate the direction of data flow)

But in those types of scenarios the cost of $SHELL interpretation and fork() is massively outweighed by the savings of stream processing.

So my point was: if you're writing a function which $SHELL interpretation and/or fork() create enough of a performance impact where you're looking to optimize how you exec `grep -E`, then maybe it's time to investigate whether $SHELL is the right language to write your function in.

vlovich1232• 4 years ago

though it has already been committed to git for more than a year now and this is the first I've heard people complain about the change.

This reasoning is flawed. I only noticed it a few months back on Arch. So between the time it takes the project to make a new release and it hitting Linux distros (Debian is slower and Ubuntu even moreso), don’t expect a flood of complaints to be time correlated with the change.

hnlmorg11• 4 years ago

This reasoning is flawed.

I wasn't making any reasoning. I was just making the observation that this change has been staged for a while now and that I'm surprised there hasn't been more noise about it before now.

don’t expect a flood of complaints to be time correlated with the change.

I expect complaints to be correlated with whenever a blog/tweet/whatever moaning about the change happens to trend. I don't think it has much to do with when distros pick up the change because, as I also said, I expect distros to backport the old behavior. So I think the timing of any backlash is entirely dependent on the mood of the internet hive mind.

brundolf2• 4 years ago

Why wouldn't they just convert them to aliases instead of adding the warnings?

hnlmorg11• 4 years ago

aliases aren't a global state. They're shell and profile specific (as there are plenty of instances where a profile isn't loaded).

With `egrep` defined as a shell script, any process (even callers who are not shells) will have the same behavior.

That doesn't mean the shell script is the best solution though. Personally I'd rather they be a symlink to `grep` and have grep check the name it's been invoked as, and if it's "egrep" (or "e" if you're just checking the first byte) then implicitly enable the -E flag. This is how FreeBSD works and it feels like a more elegant solution in my opinion.

kps3• 4 years ago

I remember around ~10 years ago being told "you should never use `egrep` because it is slower than `grep -E`."

I remember around 30 years ago reading that egrep was faster (the UNIX versions were entirely separate implementations, egrep of course being newer) and have used it ever since.

lamontcg2• 4 years ago

exactly the same here.

although i'm using ripgrep more now, proving that you can teach an old dog new tricks.

fnordpiglet• 4 years ago

I am not sure I agree with the logic. On an absolute instruction count scale grep will often dwarf the instructions necessary for the higher level logic captured in the shell script. Insert appeal to andahls law here. Writing in some more optimal language often requires considerably more work, and if the marginal benefit of reducing a few forks is minuscule compared to grep performance, why would you do this?

hnlmorg11• 4 years ago

It depends on where your hot path is. If you’re writing the kind of performance critical code where the different between ‘egrep’ and ‘grep -E’ matters, then you shouldn’t be writing it in $SHELL to begin with. Not just because of the expense fork()ing, but because Bash is interpreted, because each exec requires loading new code into your L1 cache, and so on. It’s just never going to be as quick as something written in most other languages with a mature regex library.

I’m not saying this as a criticism against shell scripting (I’m an author of an alternative $SHELL and shell scripting language, so do have a love for shell scripting). I’m saying it as a criticism against people who try to optimise against the wrong things.

I guess in a way we are “violently agreeing” (as some say) because your point about the engineering effort mattering more than micro-optimisations is an example of the right kind of optimisations. Which is what I’m also, albeit badly, trying to describe.

mike_hock5• 4 years ago

What's the problem? Just make egrep and fgrep shell aliases in the default /etc/profile (if you're a distro maintainer).

Also, where's the extra fork?

    #!/bin/sh
    exec grep -E "$@"

cryptonector8• 4 years ago

The maintainers' fears about using argv[0] to select behavior are unfounded.

Anyone exec()'ing grep with an alternate argv[0] can just use (or not use) the -E/-F options to get the behavior they want.

It's very simple. There is just no good reason for egrep or fgrep to be shell scripts or wrapper programs of any kind. Even if the were a good reason for having a wrapper program, the wrapper program could be a very tiny C (or even assembly) program that just exec()s grep with -E or -F added, thus avoiding any additional fork() calls (though there would still be one additional exec() call).

But, really, argv[0] dispatching is plenty good enough.

This warning, however, is much too annoying.

Plus, argv[0] dispatching has worked for every other Unix and Unix-like OS of note for 4+ decades! What's so special about GNU in this regard? The answer is clearly: nothing.

m463• 4 years ago

wow. Didn't realize they are shell scripts on linux.

on (an older) macos system, they are identical executables

gjvc2• 4 years ago

see also:

    which

being deprecated, replaced by:

   command -v

indigodaddy• 4 years ago

If they change this, when might we see it bubble into RHEL? 10?

dahart2• 4 years ago

The article and very few comments acknowledge that this is being done to facilitate removal of these binaries. Assuming the binaries are going away, is it better to add a warning, or to surprise people be just removing them?

In general, there’s a good, legitimate, hard question here about how to handle removing things from our software. It needs to be done, probably more often than we do it, and it’s hard enough for software writers and maintainers to bring themselves to remove things. Are there better strategies? What more can we do to make removing features less painful, beyond publishing the deprecation schedule, adding a warning in advance, and then removing them after the schedule and the warning have been out for a while?

kevincox2• 4 years ago

I don't think most of the comments are worrying about the difference between warning or removal. Most of the comments are about what is wrong with egrep continuing to work perfectly well as it has for a decade without warning or being removed.

kentonv2• 4 years ago

Some context: egrep and fgrap are implemented as wrapper scripts that look like this:

    #!/bin/sh
    exec grep -E "$@"

28 bytes.

Now consider the cost/benefit between two options:

1. Maintain these wrapper scripts forever.

2. Update all scripts in the world that use these to use the flags instead, eventually removing the wrappers.

What is the cost of option 1? It's essentially zero. No changes means no engineering work. Some hard drive space is wasted, but TBH you can probably buy a single hard drive with enough capacity to store all copies of these files that will ever exist. Perhaps some people's sense of cleanliness will be offended by the continued existence of non-standard commands but it's hard to demonstrate any actual cost from that.

What is the cost of option 2? Many thousands of hours of engineering labor, just to update scripts. But that's just the beginning: There will likely be production outages when the script are finally removed, since inevitably some uses will be missed. That will cost money. It will also lead to a general increase in fear of software updates, leading even more people to run outdated installations that are vulnerable to security flaws. Security breaches will cost more money.

It seems clear this change will cost millions of dollars, whereas keeping things as they were would have trivial cost. Therefore, this change should not have been made, and should be reverted.

Reasoning like this is why widely-used platforms simply never remove "deprecated" APIs:

* Java still contains APIs deprecated 25 years ago in version 1.1.

* Linux's ABI is famously stable, with Linus verbally abusing anyone who even suggests a backwards-incompatible change. (I don't approve of the abuse part.)

* The web platform is backwards-compatible even with objectively bad JavaScript design decisions made by Netscape in the 90's.

* C libraries still include `gets()` even though any use of it implies a security vulnerability. (Though we hope any remaining users are not running in security-sensitive use cases...)

* The Windows API is full of trash but you can still run executables built in the 90's.

"Deprecated" was never supposed to mean "will go away". It was always supposed to mean only "there's a better way to do this".

jeroenhd• 4 years ago

If the GNU folks want to remove the wrapper scripts, they can. It's their software and they write the best practices for it.

Distro maintainers can still ship these scripts. They can add aliases to the user profile to do the same thing without having to launch a sh instance. Maybe they'll add a second package, grep-utils, that just contains the wrapper scripts so you can choose between GNU's position and the "I don't like change" position.

Microsoft has deprecated and removed tons of stuff and so did browsers. Vista crashed so often because Microsoft did away with their entire driver model and vendors wrote quick wrappers around their old, crappy drivers and shipped those. Many remaining browser features are reasons why web development sucks so much ("quirks mode" for one) and Java not breaking compatibility has given it many problems that dotnet solved by breaking compat once. Java also has removed several deprecated packages and moved them to libraries instead (like Nashorn).

Deprecated means "if we see a reason to remove this, we may remove this in the future". It's no guarantee for removal but it's no guarantee for being kept around either.

kentonv2• 4 years ago

If the GNU folks want to remove the wrapper scripts, they can. It's their software and they write the best practices for it.

I'm not making any argument about the maintainer's rights. I am making an argument about whether it was a good technical decision.

[people can work around it]

That doesn't make it a good idea, though.

Microsoft has deprecated and removed tons of stuff and so did browsers.

Only when there was a real benefit deemed greater than the cost.

In this case there is essentially no benefit whatsoever to the change, except some abstract notion of cleanliness or pedantic spec compliance.

cies• 4 years ago

I'm on ripgrep (Rust, very fast). Thanks BurntSushi!

adren67• 4 years ago

I completely agree that rg/ripgrep is a way better alternative to most grep implementations including GNU grep not only in terms of feature

check https://beyondgrep.com/feature-comparison/ for a detailed comparison

but also in terms of speed, including the fact that ripgrep allows greping by default in compress files, many different encoding and more!

This software is really a must-have for anyone who spends some time on the CLI

Communitivity• 4 years ago

I also use ripgrep, and swear by it. It has many of the features previously only found in ag or git grep (parallelism, respect .ignore files, search hidden files, etc).

It's old, but here'd a feature comparison of ag, git grep, ripgrep and others:

https://beyondgrep.com/feature-comparison/

chrismarlow9• 4 years ago

I started to get really passionately angry about this because I am an fgrep user. Then I realized I can avoid my blood pressure spiking by just running a one liner and making a few edits. The muscle memory might take a while to fix but I'll be better for it and avoid my day starting off with anger about trivial things.

sudo grep -rF 'fgrep' / 2>fgrep.rip.stderr.log | tee fgrep.rip.log

Goodbye old friend...

superkuh• 4 years ago

Old software remains stable and usable. Updates break software, remove features, and introduce new exploits. I'll stick with the old reliable.

p4bl0• 4 years ago

I see the problem for scripts, but for people it's just a matter of setting an alias command, isn't it?

rrwo• 4 years ago

I've started to wean myself off of `fgrep`, but only because I am using `zgrep -F` as I'm often searching through logs, which may or may not be gzipped.

But this is a stupid change.

bravetraveler• 4 years ago

Just, why? Change for the sake of being noticed? Not getting enough attention at home?

Countless tools have figured out how to argv[0] or whatever - the path of the binary changing the behavior

chubot• 4 years ago

BTW I ONLY use egrep and fgrep !!!

- egrep means [0-9]+ works like in Perl/Python/JS/PCRE/every language, not [0-9]\+ like GNU grep.

- egrep syntax is consistent with bash [[ $x =~ $pat ]], awk, and sed –regexp-extended (GNU extension)

- These are POSIX extended regular expressions. Awk uses them too.

- `fgrep` is useful when I want to search for source code containing operators, without worrying about escaping

This style makes regular expressions and grep a lot easier to remember! I want to remember 2 regex syntaxes (shell and every language), not 3 (grep, awk, every language) !

This change should be reverted; there is no point to needless breakage

Again you wouldn’t remove grep –long-flags because it’s not POSIX

blep_• 4 years ago

You may also be interested in `grep -P`, where `P` is short for `Please just use PCRE like everything else in this century`. I don't think any of the others support that though.

teddyh12• 4 years ago

The GNU tools and programs are, theoretically, created for the GNU system, which has evidently chosen to (eventually) not provide fgrep and egrep, as they are not part of the POSIX standard. This is the GNU project’s choice to do. Other operating system projects, like Debian GNU/Linux, who use the GNU tools to provide their operating system, might choose otherwise and (separately) provide the fgrep and egrep tools. This would then be their choice (and one that I personally expect them to make).

I.e. if you don’t run straight GNU as your OS, don’t complain about this; instead object if your operating system which you actually use or depend on choose to break behavior and interfaces which your programs rely on.

richard_todd3• 4 years ago

It's interesting to think about the way these things evolved. Imagine if, over time, the `compress` utility got -gz, -bz2, -lzma, etc. flags, and `gzip`/`bzip2` were all converted to deprecated shell scripts. When is the right time to consolidate variants under one program (and deprecate the variants), and when is it better to let small utilities keep doing one thing well(TM)?

I see people talking about removing the compatibility scripts being driven by a sense of purity, but wasn't it that same sense of purity driving someone to collapse `fgrep` and `egrep` into `grep` in the first place? The sense that these are all just variants of the same goal, and thus should be flags to the same program? Why bother combining them if not to ultimately remove the "extra" programs one day?

I'm not sure what the right answer is. On one hand, I like the idea of a smaller namespace of programs with a larger set of well-documented switches. The alternate universe where `compress` covers the widely-used variants of compression and the variant utilities fell away over time sounds kind of nice. Or imagine if early UNIX had arrived at a "plugin" model, where top-level programs like grep should have pluggable regex engines which can be provided by independent projects? The culture we have, of tiny independent projects, will always make consolidation and deprecation messy events.

mirker2• 4 years ago

As someone who learned about regular expressions before extensively using grep, I found grep to be quite unintuitive, since extended grep is what I am conceptually thinking about from a “theory” point if view. One difference between grep and compression is that grep is bounded by regular languages, but compression is more free form. It’s conceivable to therefore view grep as mature enough of a technology to be finished once and for all, but compression will continue branching into many disparate algorithms.

richard_todd3• 4 years ago

Consider the popularity of ffmpeg and imagemagick. Putting the mainstream algorithms together under one interface seems to be an appealing model to a lot of people, even though new algorithms are constantly being developed in those areas.

Personally, for audio encoding, I'm much happier just installing ffmpeg than I would be gathering and learning a bunch of flac/ogg/opus/mp3/etc encoders.

MatthiasPortzel• 4 years ago

The original UNIX design philosophy was very much one-command for one-thing.

=> http://harmful.cat-v.org/cat-v/ UNIX Style, or cat -v Considered Harmful

But that argument is an argument against pretty much all command-line flags. Rob Pike argues `ls` shouldn’t have an option to split the output into columns, you should pipe ls into a column-splitting program.

So the pure version of that philosophy is long-gone. It makes this current effort seem rather arbitrary and meaningless.

xt00• 4 years ago

I guess they backed out the “which” change before this so you wouldn’t see something silly like:

which egrep

And it would print

You should use command -v

egrep is being deprecated

/usr/..

Like jeez..

manv12• 4 years ago

This only affects linux. All the old AIX/SunOS/Solaris/HP-UX boxes will never get this update, so luckily it should have 0 effect on older platforms.

That makes me feel better, that all the old crap that I've forgotten about that might be running in production somewhere will be fine.

That said, this is a ridiculous move. The only practical effect of this change is to potentially break millions of scripts, all for an ideal that, to be frank, nobody gives a shit about.

radford-neal3• 4 years ago

Beyond the inadvisability of making a breaking change for no reason, it's worth noting that deprecating fgrep is actually positively undesirable. Use of fgrep should be encouraged.

The reason is that many times people want to match a literal string. This is best done with, for example, "fgrep [a] <file". Note that this is not the same as "grep [a] <file", since '[' has special meaning in regular expressions. Of course, you can write "grep \\[a] <file", but not everyone has the set of special characters used for regular expressions at the top of their mind.

Of course, one could get in the habit of using grep -F when intending the pattern to be just a literal string. Or one could write one's own fgrep shell file. But both of these options require more effort than just using fgrep. One aim of good design should be to make it easy to do things in the reliable way. That way it's more likely to be done.

teddyh12• 4 years ago

If we are talking about interactive use, you could always use an alias.

But if we are talking about scripts, you should at least use "fgrep -- [a] < file". And if you’re adding an option anyway, you might as well use "grep -F -- [a] < file". Personally, I prefer using options, specifically long options, in scripts; meaning "grep --fixed-strings --regexp=[a] < file".

If you don’t do this, the script will fail spectacularly the day when the string happens to start with a hyphen (-).

romeoblade• 4 years ago

I thought I was the only one who perfected long options! Are you me?

I've had coworkers call me out (not rude, just "hey you know you just use -l... instead of --longopt") on calls because I always use long options when available. I use the hyphen explanation all the time as I've ran into it a few times.

I also prefer CLI applications that are designed to use the "=" for arguments with long options. Applications which don't use "=" or respect it, irk me because the it's ambiguous... "Is that argument an argument or sub command" when looking through history.

cryptonector8• 4 years ago

What's special about GNU Grep 3.8 is that its version of egrep and fgrep now print an extra message when you run them. Specifically, these messages 'warn' (ie nag) you to stop using them and use 'grep -E' and 'grep -F' instead.

Can we please stop with the make-work nonsense??

Please stop making us change scripts just because you've decided that some command shouldn't have existed that does exist. Stop it. Just stop.

See `which` deperecation. Please, no more.

Asooka• 4 years ago

It's probably time for the Linux world to distance itself from the trash fire that is GNU, as they keep breaking backwards compatibility. This isn't 1970, it's 2022, people! You do not just break the world for no good reason.

GNU has served its purpose, it's time to stop relying on them. It would be a lot easier to convince the BSD people to support the few GNU specific flags in user land utilities and to add GNU libc compatibility, than it would be to convince the GNU maintainers to write good software.

throw10920• 4 years ago

Hmmm, perhaps "a program's output is part of its interface" wasn't such a good idea after all...

If programs communicated with each other using structured interfaces and not "plain text" (already a misnomer), this would be a non-issue.

ngvrnd• 4 years ago

I like beating people in concept but not in practice.

shmerl• 4 years ago

ripgrep (rg) is an option. I switched to it for all my grepping needs.

manv12• 4 years ago

It's ironic that GNU is using POSIX as a justification for a decision.

From RMS:

Following a standard is important to the extent it serves users. We do not treat a standard as an authority, but rather as a guide that may be useful to follow. Thus, we talk about following standards rather than "complying" with them. See the section Non-GNU Standards in the GNU Coding Standards.

We strive to be compatible with standards on most issues because, on most issues, that serves users best. But there are occasional exceptions.

For instance, POSIX specifies that some utilities measure disk space in units of 512 bytes. I asked the committee to change this to 1K, but it refused, saying that a bureaucratic rule compelled the choice of 512. I don't recall much attempt to argue that users would be pleased with that decision.

Since GNU's second priority, after users' freedom, is users' convenience, we made GNU programs measure disk space in blocks of 1K by default.

https://opensource.com/article/19/7/what-posix-richard-stall...

markhahn• 4 years ago

preach!

kcexn• 4 years ago

I'm so confused by this change. fgrep and egrep are already just shell scripts that are almost just like aliasing grep -E and grep -F. Why is the GNU team suddenly so worried about these shell scripts?

If the maintenance of these scripts is so hard. They could just deprecate them entirely. It's pretty unlikely that they would stop working or become insecure anytime soon. And sysadmins would just source in the aliases into their shell environments anyway....

c7b• 4 years ago

typing 'grep -F' is two more characters, one of them shifted

Adding in more special characters is also hostile to users who aren't based in a single English-speaking country and regularly have to deal with different keyboard layouts.

forty• 4 years ago

While we are discussing common command aliases, could we decide to make ll (= ls -l) a standard one? I have a hard time using a shell without it, and it's annoying to create the alias on all the machines I ssh to ^^

dschuetz• 4 years ago

Seriously, what's the deal? Just use aliases instead, if you're fond of using egrep or fgrep yourself. Refactoring and re-deploying scripts should not be a big issue in any environment either these days. Admin/Devs are lazy, yes. But THAT is the least of their problems maintaining shell scripts.

Jenda_• 4 years ago

I first thought about different warning that GNU grep >= 3.8 emits: about "stray \" (it's also in the changelog - https://lists.gnu.org/archive/html/info-gnu/2022-09/msg00001...). I have personally seen this one several times already and I only have grep 3.8 on my desktop, as I use Debian stable everywhere else. But this warning may prevent hidden bugs in the future.

nullc• 4 years ago

Just be glad they haven't handed over maintenance to the gnome project.

kps3• 4 years ago

Soon grep will be a systemd module.

ykonstant2• 4 years ago

Surely you mean systemd-grepd -z nuts

arrakeen• 4 years ago

debian appears to be just removing the warnings:

    grep (3.8-2) unstable; urgency=low
      This Debian grep release removes the deprecation warning about egrep and
      fgrep. These alternative programs will be still shipped by Debian. Although,
      for portability reasons, users are encouraged to use grep with the concerned
      options instead of those alternative programs.

We are stuck with egrep and fgrep (unless you like beating people)

Related Stories