The Shell Hater's Handbook (2010) [video]

http://confreaks.tv/videos/gogaruco2010-the-shell-hater-s-handbook

by mmphosis • 9 years ago

90 45 9 years ago

confreaks.tv

the_cat_kittles• 9 years ago

the resource that he made has been moved to http://shellhaters.org/

miobrien• 9 years ago

Funny presenter.

unixhero• 9 years ago

So... He was ... Bashing bash?,

I for one like my shells dirty with hidden corners and black magic. Name anywhere where you do not have bash available; embedded, grub, rescue boot shells. I don't do embedded. Never will.

But on a serious note, does anyone know a good resource where I can pick up a better la guage and never have to bash script again? I love writing it. It I am sooo restricted by it, yet keep solving problems with it and also making good money like last night. Maybe Python is the way?

teddyh• 9 years ago

Maybe xonsh would be for you? (http://xon.sh/)

wazoox• 9 years ago

Python and Perl are both reasonable substitutes. Perl is closer in spirit (black magic, direct call of external programs with backticks or qx(), almost direct translation of sed and awk calls).

sethrin3• 9 years ago

I'm familiar enough with Bash to have written an opinionated but more-or-less comprehensive guide to it. I think Bash should be avoided as much as possible. Bash is great for things that you would have had to type out anyway, but using it to write programs is foolish. My rule of thumb has become, "if it's more than 30 lines long, or if it uses more than two variables, it should be written in another language". That said, I'm in the process of learning how to do that myself.

Python is definitely a capable language for scripting, but I would also point out that Homebrew is built on Ruby and uses that to great effect. You might also check out this article on scripting with Ruby:

http://radek.io/2015/07/13/ruby-scripting/

And if anyone else has any tips on this subject please do pass them along.

jwhite8• 9 years ago

I appreciate your point, but I think the choice to write all or parts of an application in shell depends a lot on what you need to achieve. Writing 100 lines of shell to glue parts of your application together may be the best way to ship something today, even if you plan to replace that with another language in 3 or 6 months time.

Would you care to give a link to your guide? I'd be interested to read it.

mannykannot• 9 years ago

Bash (and other shells) have their purpose, and that purpose is running processes, and process-, filesystem- and general-system administration. Pipes, especially, are so much easier in a shell than any other way that I have seen (I just noticed ninjin's comment about Julia - I will have to give it a look.)

For this sort of thing, I will use a combination of Bash, Awk, Perl and Python, with Bash as the foundation, but often using at least one of the others. For example, if a step in a pipeline can easily be solved with regular expressions and/or simple arrays and hash tables, I will pick Perl, but if it needs more complicated data or functions, I will pick Python.

thomastjeffery3• 9 years ago

Ruby is definitely better syntactically for scripting then Python.

ninjin• 9 years ago

While Python does the trick just fine, I find how Julia handles "shelling out" and IO with the shell to be very pleasing. I am probably still a bit ignorant in terms of what is out there, so I am reading the comments with great interest.

https://julialang.org/blog/2012/03/shelling-out-sucks

https://julialang.org/blog/2013/04/put-this-in-your-pipe

cturner• 9 years ago

I've been down exactly the rabbit hole you're speculating about, and my firm conclusion is: python is not a shell replacement. [update: some other posters here are referring to xonsh. Maybe they succeeded where I failed, and python can be done shellable. Will be looking into this.]

* Python does strange things with IO buffering. Just when you think you've nailed all the edge-cases of this, some more come along. Its decisions are good for a general-purpose programming language, and awful for a shell.

* Python is awkward for some close-to-the metal operations such as interaction with unix sessions and process groups.

* Python forking is wacky. There's magic in there, and there's some stuff that gets inherited under the cover. If you fork and don't immediately execv, you are probably in for Interesting Times.

When I looked at the code for Ansible, I felt a bit like I'd encountered a rival alien civilisation who had tested and reacted to the same laws of the universe, and reached identical conclusions. Ansible is written in Python. But it generates bash scripts, and then bash does dispatch.

Given the power of general-purpose languages we have on unix, it's remarkable how stunted shell evolution is. Which is to say, there is an exciting opportunity here.

A shell built as a Forth dialect could be a game-changer. It would have a certain number of powerful builtins. It may even evolve higher-level programming concepts. I am interested in further discussion towards this, and maybe a new github project, particularly a collaboration with someone with high fluency in Forth. I'm in London: beers.

The art of a good shell would merge (1) fluid interaction; (2) effective wrapping of the unix API; (3) high-enough-level programming.

jabl• 9 years ago

Apart from shells having wacky syntax and semantics, due to shell pipelines being plain text, you spend a lot of time writing flaky mini parsers with stuff like sed/cut/awk/perl one-liners/ to get the data you want.

In an alternative universe it would be nice if all these tools would consume and emit s-exprs. Or maybe I'm just wishing for the return of the Lisp machines, where AFAIK "the shell" was the Lisp REPL (never used one myself so I could be all wrong, of course).

Kids these days would of course use json instead, at least until the serious (tm) enterprise people would swoop in and replace it all with xml and ShellReplFactoryInstanceFactoryFactoryBean.

daxelrod• 9 years ago

Are you aware of PowerShell? Pipelines act on full .Net objects.

vram223• 9 years ago

ShellReplFactoryInstanceFactoryFactoryBean

You forgot to include Adapter, Builder, Strategy and a few other words in that.

vram223• 9 years ago

Kids these days would of course use json instead

JSON? How last week. The cool(est of the cool) kids are already moving on to TOML and its ilk.

smitherfield• 9 years ago

Ruby is my goto language for replacing shell scripts and even the occasional one-liner. I find it much more suited to that task than Python (more shell-like features, better regex/text processing, more flexible program structure), and much better-designed than Perl.

jwilk2• 9 years ago

So where's the actual handbook?

jwhite8• 9 years ago

I spent many years avoiding shell, in large part because all the arbitrary-looking syntax and features (like quoting rules) the speaker highlights make it frustrating and difficult to master, and I felt like it was a treacherous enemy to be overcome through desperate struggle, rather than a tool and ally to rely on. I know some people thrive on mastering complex and intricate rules, but that's not my forte. I do better with overarching principles and a unified set of orthogonal constructs, such as C (ok, debatable, but compare to C++), Lisp, Python or Haskell.

It took a lot of pushing from a mentor for me to finally click with the overarching principles of shell and see how to master it and become productive with it. He really forced me into it as well, I wasn't a willing pupil for quite some time, but he kept persisting and finally I started listening to him. I'm so glad I did. I've never read any book or tutorial or guide that teaches what he taught me either, despite searching several times. Also, I don't think I could have learned to master shell if I hadn't learned functional programming first. The day I realised how amenable shell is to higher-order thinking was a real epiphany.

These days shell is a really important tool in my toolkit, and I can see just how much some of my colleagues are held back by not knowing it.

Anyway, I really enjoyed the video, it was a nice exposition.

nsomaru• 9 years ago

Care to document some of those insights for those of us in a similar position?

jwhite8• 9 years ago

Here are a couple of thoughts:

- I used to spend a lot of time stuffing things into variables and then trying to operate on those, which is how you do things in imperative languages. Shell is better thought of in dataflow terms, i.e. pipelines. So I started using pipelines more and more.

- The corollary is that you start thinking more about data (passing between pipeline stages) and less about control flow.

- Compare an if statement in C, for example, to an if statement in a functional language. In C an if statement has no value per se, but in functional languages if "statements" are actually expressions that have a value:

f :: A -> String f x = if isFoo x then "foo" else "other"

In shell an if statement is just another command, so you can do things with its "value", i.e. its stdout:

    if do_test
    then
        cat file1.txt
    else
        cat file2.txt
    fi | grep pattern

Because I was so used to thinking of if statements as just control flow, I never used to think of piping them to another command like that.

Also if you think carefully, that last pipe to "grep pattern" has the flavour of partial evaluation in functional programming.

- I tend to think more in terms of building up an execution environment these days, in the same way that lexical scope works in regular programming languages. In shell, of course, that means command invocations, sub-shells, etc., because the shell doesn't really have good lexical scoping. The simplest example I can think of is executing some commands in another working directory with a sub-shell:

    (
        cd $dir
        run_test > "${f}.log"
    )

You could compare it to RAII in C++ or with- style macros like WITH-OPEN-FILE in Common Lisp.

Those are just some examples.

EDIT: Also, have a read of the links in chubot's post: https://news.ycombinator.com/item?id=14012453 . That's the sort of mentality that will change your approach to shell programming.

coldtea2• 9 years ago

These days shell is a really important tool in my toolkit, and I can see just how much some of my colleagues are held back by not knowing it.

Perhaps, but the shell is still a badly accrued (more than designed) set of options.

jwhite8• 9 years ago

Yes, totally agree. But it offers something different to say Python, as the video points out. Some things are more compact in shell than in Python, because it's not the same sort of programming language as Python. And I reach for Python as well, when it's the right tool.

falcolas• 9 years ago

Well, on the plus side of that, accrued means they're actually useful, as opposed to "designer through they would be useful".

I wouldn't mind a better shell, but I'm not going to throw out my current shell for the lack of a better shell.

yjftsjthsd-h• 9 years ago

I'm confused. You had a good time with C but trouble with shell? I was strongly under the impression that 90% of Unix shells were strongly inspired by C semantics. Return codes, quoting, strings in general, etc. You maybe could even try csh if that was better.

jwhite8• 9 years ago

Every language has its warts, but C is a pretty small language when it comes down to it. Compare that to just the number of different ways you can expand a parameter in bash:

https://www.gnu.org/software/bash/manual/html_node/Shell-Par...

That's the sort of thing that turned me off. I still tend to avoid those nooks and crannies, although they are sometimes useful. I'm not an expert.

I'm not sure about Unix shells being inspired by C semantics. I actually started out on csh or tcsh, I forget which, two decades ago. I'm not sure that making a shell language "C-like" is a useful goal, and those two shells both fell out of favour because they have many flaws. I switched to bash quite quickly.

chubot• 9 years ago

I agree on both counts. My first experience with shell was the fact that 'foo=bar' works for assignment but 'foo = bar' doesn't. Surely that must be a mistake! Why would anyone design such a silly language?

I got past all of that, and I also see people struggling to accomplish things that can be done in 30 seconds with shell. My main language is Python, and for certain tasks it just can't compete with shell. Believe it or not I think I once wrote a Python script to do what "find | sed -i" does.

Now I'm back in the "who would design such a silly language" mindset, and I am designing and implementing a new shell. You might like some of my blog posts:

"Pipelines Support Vectorized, Point-Free, and Imperative Style"

http://www.oilshell.org/blog/2017/01/15.html

"Shell Has a Forth-like Quality" http://www.oilshell.org/blog/2017/01/13.html (on HN previously)

jwhite8• 9 years ago

Thanks for those links, I think I will enjoy them. I was actually thinking about Forth as well when I was writing those comments about functional programming. I'll be interested to read your take on it.

ams6110• 9 years ago

I'm the opposite, I avoid GUIs. I use pretty much only the shell, and text-oriented programs. I use a graphical desktop just to manage my terminal windows. The only other application I use daily is a web browser.

I guess maybe because I started with computers before there were GUIs that is what felt normal.

thomastjeffery3• 9 years ago

I guess maybe because I started with computers before there were GUIs that is what felt normal.

For me, probably not. I did use DOS before Windows as a kid, but that was long before really learning to use a shell. Familiarity with DOS simply made me familiar enough with the idea to be unafraid of it.

When I learned to manage files, handle software, etc. I learned to do it with a GUI in Windows. When I started to use Linux, I was vaguely aware of the usefulness of a shell, and took the opportunity to really learn. To this day, I rarely even have a file manager installed, in favor of a shell environment.

jwhite8• 9 years ago

Ha! I should have been more precise in my language. I was avoiding learning how to write more substantial shell scripts and programs, not the command line per se.

I also got my start on pre-GUI computers (a VIC-20, specifically).

jwilk2• 9 years ago

Don't use "[ ... -a ... ]". It's not portable and has unclear semantics.

nlawalker• 9 years ago

This[1] Stack Overflow question sums up everything I dislike about shell scripting. Not only does it highlight the finnicky string-oriented nature of it, but the answers...

Multiple contradictory answers, disagreement, sub-sub-sub explanations, historical context, caveats, workarounds, hypothetical scenarios, portability, personal taste, different versions and implementations all show up. One answerer felt the need to include a footnote.

It's like the opposite of Python's "there's one right way to do it": There are 50 right ways to do it, all of which are wrong in some case or another, potentially depending on which way the wind is blowing.

All in regards to checking for an empty string.

[1] http://unix.stackexchange.com/questions/136628/bash-script-x...

unknown2374• 9 years ago

Not to start a war, but Python's "there's one right way to do it" is nothing more than an ambitious claim with little ground. Maybe it applies for doing something brainless like printing a string, but the claim completely falls apart when trying to do something as simple as list indexing/slicing or even iteration. I'm not condemning Python in anyway, I am a fan of the latter mention features, but that "zen" of Python is not applicable.

wwweston2• 9 years ago

Python's "there's one right way to do it" is nothing more than an ambitious claim with little ground.

Probably better described as an ideal, not always lived up to, but serves as a guiding principle...

arca_vorago• 9 years ago

Inevitably when shell programming is mentioned, particularly bash, I hear quite a few devs/programmers talk about how "shell is bad and you shouldn't use it" maybe with an added "for anything not a one-off or over 20 lines"...

I completely disagree. As the sysadmin who has been the guy actually managing devs/programmers messes in prod, shell is the one thing that just works, is easily readable, is easily documented, is fairly easy to write, and generally keeps me from pulling my hair out or become violent with the devs.

Take it from the guy who actually has to wrangle all your strange python/perl/go/node... bullshit, shell is something you shouldn't be bashing.

wwweston2• 9 years ago

shell is something you shouldn't be bashing.

But apparently, bash is something you should be s(h)elling.

m-j-fox• 9 years ago

https://youtu.be/olH-9b3VJfs

vram223• 9 years ago

Reading this thread and the discussion of the pros and cons of the shell, reminded me of this:

The Bentley-Knuth problem and solutions:

https://jugad2.blogspot.in/2012/07/the-bentley-knuth-problem...

It is about an interesting programming problem to do with text processing - involving finding word frequencies (apparently initially posed by Jon Bentley to Donald Knuth).

The above post (on my blog) has two solutions by me, in Python and shell, and links to the original discussion on another blog, where I first read about the problem - which was an interesting read, with many comments:

More shell, less egg:

http://www.leancrew.com/all-this/2011/12/more-shell-less-egg...

The Shell Hater's Handbook (2010) [video]

Related Stories