How to write idempotent Bash scripts

https://arslan.io/2019/07/03/how-to-write-idempotent-bash-scripts/

by kiyanwang • 7 years ago

655 187 7 years ago

argd6782• 7 years ago

The problem with the idempotent trend in configuration management is that it’s all based on not tracking or knowing what the current state of something is. So reasoning about how these systems work is fundamentally impossible. It would be better to focus on systems where by we can always know the state and improve the tooling there.

platistocrates• 7 years ago

I don't see any conflict here. Idempotency is not a silver bullet and cannot fully replace state tracking, but it adds simplicity where appropriate.

LfLxfxxLxfxx• 7 years ago

On the contrary. With idempotence you know the state after an action. rm -f will always delete the file (if possible). ln -sfn will always work, even for a directory.

With the default behaviour of rm, ln -s, etc you know neither the state before, nor after.

posix_me_less5• 7 years ago

The option -f is making the script more heavy handed and less likely to fail. But this is not necessary for the script to be idempotent and sometimes is not desirable. You can deal with errors in a more safer way and still be idempotent.

iforgotpassword2• 7 years ago

It's a pragmatic approach to this problem. Usually tracking state means that you write some version number or similar to a file, but than if for some reason that goes out of sync with the actual state, for example because you put it in /etc and someone accidentally restores an older version of it, hilarity ensues, because it's guaranteed the devs never expected this to happen and don't handle it in any way.

For the case of a setup script you'd need to update your state file after every step and in turn when running it again see what the file says and resume at whatever point it says. But then you risk running into above problem.

I try to write idempotent scripts whenever possible, combined with general sanity checks specific to whatever environment the script expects.

argd6782• 7 years ago

The reason it’s pragmatic is due to how systems are designed to begin with, it’s not what you want.

We know what the state is right after the command for an undetermined amount of time, but we have no knowledge of what’s changing it. Why not cover that? There’s no reason not to other than it hasn’t been worked on enough. Consider a database schema migration, there’s a reason those are not idempotent, you know each state. Seems quite solvable.

aey2• 7 years ago

Why wouldn’t you use make? Side-effect tracking and cleanup is the job of a build tool.

da_chicken2• 7 years ago

Why would you use a build tool when you're not writing a build script?

throw0101a• 7 years ago

Instead of "built tool" think "state machine". (What is the state of the .o file compared to the .c file? Newer or older?)

bdcravens• 7 years ago

It's not necessarily a build tool - it's commonly used for that, but it's a useful DSL. rake in the Ruby world is used for all sorts of tasks for web apps and more, even though it was originally designed as the Ruby version of make. Look at npm - the NODE package manager, but it quickly grew to be used for more than server-side applications. A title is a description, not a constraint.

andreareina• 7 years ago

How do you get make to track something other than the existence and age of a file?

aey2• 7 years ago

If it’s IO, then the inputs and outputs can be tracked as files. If it’s not IO, and just a pure computation it can be reproduced on demand.

twoodfin• 7 years ago

   something_else && touch something_else_done

zimbatm2• 7 years ago

make only knows how to track side-effects on files though.

A better system needs to be able to track directories creation, package installation, filesystems creations and mounting, service restarts and so on.

Zenst• 7 years ago

Many people who write scripts do not factor the environment they are run in and miss including traps to handle events like being terminated mid-way thru and cleaning up to a sanitized state. After all, creating a file, you can't presume their is enough free space, but people do in scripts all the time.

Then you have scripts that you want run once and if run again not have any more effect. FOr example a script to be run by ops on production servers, what happens if they accidently run it twice! An easy way to cover that is to have the script create a file, which you check at the start of the script and if present you exit. For this you can use the date/time, PID and script name for a unique file name, create in a /tmp directory that you cleanup weekly via sculker or whatever frequency you require. Handy way to handle scripts that you want run successfully once and never to be run again.

cjvirtucio• 7 years ago

I have not been doing DevOps for very long, but a useful trick I've found is to create a staging directory, perform my work there, diff the result with the existing state, and copy if a difference exists. I've found it very easy to clean up to a sanitized state in this way; the existing state is safe, since all I have to do is just destroy my work.

prepend• 7 years ago

If possible, I’ll try to make a symlink just point to the staging dir after tests complete. This isn’t always possible, but makes it easy to roll back to the old prod folder if anything goes awry.

nerdponx• 7 years ago

What do you use to compute the diff? Just the diff command itself on individual files?

fwip4• 7 years ago

You might find rsync cleaner to use than a diff/cp loop.

Another approach is to create a unique directory (named by hash or date) with the results, and once it's good, symlink the real directory to it. (Full disclosure, I stole this approach from nextflow.io, a great workflow system aimed at bioinformatics).

    ln -sfn "build-cache/2019-07-07-21:23:04" "build"

pjscott• 7 years ago

Apple's new-ish APFS filesystem includes something for this pattern: the renamex_np(2) system call with the RENAME_SWAP flag. This atomically swaps two files, and can be used to atomically swap your staging directory with the existing directory. I'm not sure if there's command line support yet, though.

grosswait• 7 years ago

Assuming umask is one that bites any dev that has never worked with hardened systems.

amelius3• 7 years ago

Just have a database table, and update it with a special key. When the transaction commits, you know the script has executed correctly, and the key is written as a result. When the script doesn't complete, the commit is not performed, and you can run the script again. No need for "idempotent" scripts.

Trying to do this with a filesystem instead of a real database, and things get messy.

dredmorbius5• 7 years ago

Without proper locks, this sets you up for race conditions.

zbentley2• 7 years ago

You still have to make that messy choice between at-most-once and at-least-once execution of each transactionally guarded section.

pdkl952• 7 years ago

traps to handle events like being terminated mid-way thru and cleaning up to a sanitized state

Traps are underused. They solve a lot of error-handling and cleanup problems really easily!

Here[1] is a simple example of using a trap to cleanup a tempfile, that safely handles pipeline errors and unexpected exit/return. The same basic idea works for many kinds of error handling, cleanup, or similar work.

If you ever need to cleanup something at the end of a function

    foo() {
        setup_command
        # ...
        cleanup_command
    }

it using a trap might be as simple as moving the cleanup command to trap at the earliest possible time in the function the command can run:

    foo() {
        setup_command
        trap "cleanup_command" RETURN
        # ...
    }

[1] https://gist.github.com/pdkl95/61fc242e7961cc2584a787ed1760c...

ddebernardy• 7 years ago

Intriguing. I always trap before setting up, on the basis that a script could get interrupted before setup is completed. Is there any general agreement on what the correct order should be?

kahlonel• 7 years ago

An easy way to cover that is to have the script create a file, which you check at the start of the script and if present you exit.

This does sound a good solution on paper, but "checking at the start" and "creating a file" are two different steps (i.e. not atomic) and will cause trouble eventually if your system has the tendency to run the script twice. A better solution is to use the `flock` command before creating the flag file. For example,

  exec 999>/var/lock/my.lock
  flock -n 999 || exit 1
  if [ ! -f flag_file ]; then
    echo "script not ran before, running"
    touch flag_file
  else
    echo " already ran, exiting"
    exit 1
  fi
  # do stuff here

dredmorbius5• 7 years ago

Within a shell script context, 'mktemp' and 'mv' are almost always what you want, possibly 'ln -s' for libraries.

The first safely creates a temporary file. The second (with noclobber set or -n argument) will atomically rename a file, but not overwrite existing contents.

Linux uses symlinks to system library files in order to allow in-place atomic replavement without affecting running processes with open filehandles to earlier versions. MS Window's lack of this feature is what makes (or made, I'm very out of date) malware removal and system updates so painful.

mnutt5• 7 years ago

It’s good to know which command line tools are idempotent, but I think once you start running automated scripts repeatedly you’re better off with something like Chef where you declaratively define the end state and let chef figure out how to converge upon that state. Taking the “create a directory” example, you would say “I want to end up in a state where this directory is created and has these permissions” and chef will determine if the directory already exists.

There are of some downsides to that approach which stem from the fact that you can’t possibly declare every possible thing. For instance, if you were to run the above declaration and then later remove the declaration, chef wouldn’t know to delete the directory; at that point chef wouldn’t know anything about the directory at all. So ideally you can build infrastructure from scratch each time (Docker, etc) rather than having to converge an existing machine.

itchyouch• 7 years ago

Chef does have an action on their primitives, so it is possible in the directory example to set the action to :delete instead of only implying :create

mnutt5• 7 years ago

You’re right, it’s totally functional, the (minor) challenge I see with it is that it breaks the illusion that you’re describing the state of the system, and feels more like describing a set of changes you want to make.

pas8• 7 years ago

Chef isn't magic, it just uses a lot of cookbooks which are just running things to check what the state is.

Usually those cookbooks will want to have things in a particular way, which might not be the way you want.

Chef/Ansible/Puppet all have the problem of having many layers of overhead for the same thing.

This makes them slow, hard to debug, hard to read and even write (in my opinion).

Sure, bash is hell, but the go from bash to something like Rust, instead of Chef/Ansible.

Sure, again, this might sound rather unconventional, but you get safety, speed and convenience of a full fledged programming language, single binary output, etc. And you need to test cookbooks/playbooks too, so why spend enormous effort on cobbling together scripts and high-level abstractions instead of writing just what you need with the assumptions you really have - instead of what a general cookbook/playbook has to have (which is a vast difference).

empath752• 7 years ago

Writing a shell script in rust would be obnoxiously difficult for no real benefit.

eddieroger• 7 years ago

They didn't claim Chef was magic, and yeah it's just a bunch of cookbooks, but that doesn't change their point that it's declarative and useful for describing an end-state. Chef is hard and slow if you rely solely on others' work without understanding what it's doing, which is functionally equivalent to magic. But if you know how to tune things, keep your cookbooks down to only required ones, and a few other tricks you'll learn from usage, it can be fast as anything. I've seen one particular setup meant to keep a server up to date and configured to have HAProxy serve over 100+ endpoints run in under a minute. It's possible, just a different approach.

lukeschlather• 7 years ago

Rust doesn't get you anything resembling what you want here.

Chef/Ansible/Salt could be described as rudimentary type systems for operating system states, that enable you to convert between different types/states.

It's conceivable you could build such a system in Rust, but Rust itself has no primitives that would make it easy. It certainly isn't a good starting point when you want to set up a database server.

mnutt5• 7 years ago

Yeah, I find out-of-the-box cookbooks about as helpful as out-of-the-box docker images: useful as a starting place but to leverage chef you really need to write your own. Chef resources/providers are nicely composable and the primitives are pretty easy to rewrite if you want, my main point is that chef gets you to think in terms of convergence instead of brittle imperative changes.

You could rewrite your bash scripts in rust but while it would probably be safer/more correct than bash, the whole paradigm of writing imperative scripts to manipulate existing machines is flawed, whichever language you write it in. And executor performance is almost never the bottleneck in my experience.

ajxs2• 7 years ago

Many of the comments in this thread seem to be sucked into a false dichotomy. There are better options than _either_ Rust or Bash for writing complicated shell scripts. Perl/Python are more suited to the task than either. The challenge of deciding what degree of complexity warrants the investment of more complicated language is a design decision left to the developer to figure out. Any thought given to designing more resilient software is a good thing, but I can't help thinking that expecting any real degree of safety and security from bash is expecting too much. I don't work in devOps mind you, but whenever I have to write scripts that do anything more dangerous than creating files I leave bash on the shelf.

xorcist• 7 years ago

Absolutely this. When the intent is to get the system into a desired state, reach for Puppet/Salt/Ansible.

If you are doing sysop^Wdevops work it is the single most important thing you can learn, once you have a good grasp of your shell and your editor.

The difference between a giant git archive of shell scripts that various people have modified over the years and state changes described in a configuation management language is the difference between fixing things in the small and reasoning about integrated systems. It's something that needs to be experienced to be appreciated.

Reasoning about system state as an integrated whole is just as relevant when you are shipping applications as containers, if not more so. It's not uncommon to start using something like Kubernetes without first being able to describe global state and the result is just as messy as before, if not worse. Something like Helm is impossible to understand unless you have complete control over your configuration.

cpitman• 7 years ago

100% Agree. I originally got into Puppet after spending several months with a client that wanted me to write bash scripts to push out a major middleware deployment, switchover, and uninstall to 10,000+ stores. There was no standard for the hardware in the stores, and their configuration was all over the place because 100's of teams also used (non-idempotent) shell scripts to push changes.

I learned a lot about catching all the different bad things that can happen, adding prerequisite checks, rolling back changes automatically, etc. In the end, a tool like Puppet/Chef/Ansible would have taken a fraction of the time to develop scripts for.

Containers have definitely lessened the need for mutable configuration management, but they are still useful in the 90% of environments that are still working in a past decade.

audience_mem• 7 years ago

Is it not a bit of a large jump from "shell script" to "chef"? There is some inbetween.

I understand a shell script is not suitable for orchestrating a datacenter, but sometimes it is suitable for installing a couple of things.

pnutjam2• 7 years ago

Ansible is good for small or large installs.

NelsonMinar• 7 years ago

I got hung up on the claim "Touch is by default idempotent. ... A second call won’t have any effects". The whole point of touch is to have an effect every single time you run it. It updates the file atime and mtime. That may seem harmless in your application, but it's definitely not no effect. Also promiscuous touching is the source of a bunch of bogus last modified dates in source code bundles.

stephenr• 7 years ago

I had the same thought about `ln -sf` - removing and recreating the symlink aren't no-effect.

_0w8t• 7 years ago

To avoid that I use readlink in my scripts to read the previous value of the link and call ln -s only when the value is not what is expected.

kousikk2• 7 years ago

Last modified timestamps in source code are a bad idea and they make builds non-hermetic. Better to fix up those scenarios in code than work around them.

Irrespective you have a good point that touch does have side effects.

rgoulter• 7 years ago

FWIW, I don't think "hermetic" is in widespread usage.

My understanding is that "hermetic builds" refer to the kind of thing that tools like Guix, Nix, etc. go for. Something like "Declare all the inputs and dependencies to get a reproducible output".

RijilV• 7 years ago

I hate to nitpick, but from the man page of touch:

touch -- change file access and modification times

If one were to argue side effects of touch, it would be that non-existent files are created. The purpose of touch is to update access and modification times.

dymk8• 7 years ago

Within the scope of the article, which is about modifying the structure and logical contents of the filesystem (a-la `chef converge` - the author never once mentions FS metadata), it's idempotent.

farslan2• 7 years ago

Op here. Yeah you’re right. I’ve mentioned it though in the context of file contents. Just updated to make it clear.

riazrizvi3• 7 years ago

I've heard Idempotency confused with Consistency. Idempotency is where a function f can be applied (as a composition) multiple times and it gives the same result, so Idempotency: f(f(x)) = f(x). Whereas Consistency: f(x) = f(x) = f(x). An example of an idempotent function f is RaiseToThePowerZero(x) on x > 0. We usually take consistency for granted, so it's perhaps better to think of an example of a non-consistent function f which would be RandomNumberGenerator().

JoeAltmaier2• 7 years ago

Better examples: Write("filename", position, data) is idempotent; Append(handle, data) is not.

mehrdadn2• 7 years ago

as a convolution

Sorry, what does idempotency have to do with convolution?

stefco_• 7 years ago

If you count the filesystem stats on a file as part of the system state, it is certainly not idempotent. You need to be aware of that effect on system state before you can decide whether to ignore it.

Idempotency requires that the function map from a space back onto itself. Touch is only roughly idempotent in that you can define a statistic S on the overall system state discarding filesystem stats such that S(f(f(x))) = S(f(x)).

mehrdadn2• 7 years ago

Just a note... even reading a file updates the access time, so if you reason this way then no program touching any files in the worktree will ever be idempotent... which seems to lose the utility of the concept.

intc• 7 years ago

This is an easy one. Touch is by default idempotent. This means you can call it multiple times without any issues. A second call won’t have any effects

Every call to touch will alter the modification time of the file (example.txt in this case).

jrumbut• 7 years ago

It will change the access time too. Almost every suggestion in the post has this problem which is unfortunate since access/modify/create time are used in a lot of scripts to see which files were or were not processed by prior runs of a script (whether this is a good idea or not).

It is very difficult to write truly idempotent bash scripts (consider log files). Creating sort of idempotent or at least rerunnable scripts would be nice but I think even that would take more care than this.

theamk2• 7 years ago

Do people actually use access time for anything?

I occasionally do full-text searches on system files, and this resets access times on all files. Thus, I cannot imagine using a script which cares about it.

mongol• 7 years ago

This is useful and I think it would be good if more tools had idempotency options available. Just today I needed to run a script that added a zypper repo in OpenSuse. It fails the second time because the repo is already added and there is no option to avoid it. Also the error code is not distinct so it is not easy to script around. Ideally every tool should have a mode "accomplish this result" in addition to "do this".

michaelmcmillan2• 7 years ago

Let's try to avoid using a hammer on anything that resembles a nail: https://github.com/valvesoftware/steam-for-linux/issues/3671

dymk8• 7 years ago

How is that related to the article aside from "both involve bash"?

pmarreck3• 7 years ago

It's because rm -rf "$STEAMROOT/"* will evaluate as rm -rf "/"* if $STEAMROOT for some reason is blank or unset, which is what happened here. (As someone in that discussion mentioned, a little more Bash knowledge might have suggested using rm -rf "${STEAMROOT:?}/"* instead, to force it to (at least) error if it is empty or unset.)

canadev• 7 years ago

personally, I feel like rm -f is dangerous, and never would recommend using it.

zokier• 7 years ago

The first example ("idiom") is bit tricky, because the title is "Creating an empty file". An alternative would for example be

    echo -n > example.txt

Both fit the description of "Creating an empty file". Is it idempotent, or more or less so than `touch example.txt`? If there is already an existing file, then touching it obviously will not empty it, so the end state is not "empty file exists" like one would expect from operation "Creating an empty file"

Ironically article calls "This is an easy one"

dylan6042• 7 years ago

If a file has a line ending in it, is it truly empty?

fwip4• 7 years ago

The -n flag keeps echo from emitting a line ending.

gabrielblack3• 7 years ago

"echo -n " is useless, is sufficient:

example.txt

without prepending the echo command. This will create an empty file or it will truncate an existent file. The touch command could be preferable to preserve the content, if needed.

gabrielblack3• 7 years ago

P.S.

> example.txt

this one (with a double > ) could replace touch in this application case: it's less readable but more efficient.

zamadatix• 7 years ago

"The -f flag removes the target destination before creating the symbolic link, hence it’ll always succeed."

Removing something and adding it back seems to be, by definition, not idempotent. What if something tries to access that file in between? What about the filesystem timestamps (same issues as with the "touch" claim)?

grzm2• 7 years ago

Idempotency focuses on the result, rather than implementation. If the intent of the script is to ensure the file is there when the script is finished, then it’s idempotent. If the time stamps matter for the intent of the script, then, yes, what you pointed out would be an issue.

Godel_unicode2• 7 years ago

This is incorrect. Idempotency is about not needing to worry about running something more than once. If you introduce race conditions with follow-on runs, you're not idempotent.

ggm• 7 years ago

The chmod command can alter the flagset on the link not the target, if you select the right options.

How do you guarantee the replacement symlink had the same chmod flags and grp flags as the original? (chmod -h for reference*)

I don't think these commands are idempotent in the wide. They satisfy only the narrowest "yea, most normal-ish cases are sort of maybe ok" but since we lost atomicity, there are significant windows of time in the filesystem where inodes are in flux, and the new thing has a different inode to the old thing, if you delete and replace it. The disk is different.

dlgtho• 7 years ago

True. In such case you need to create a different symlink within the same filesystem and then use "mv" to atomically overwrite the current symlink.

ln -sfn target new && mv -Tf new old

zimbatm2• 7 years ago

All the examples are more-or-less idempotent. For example the `blkid /dev/sda1 || mkfs.ext4 /dev/sda1` doesn't guarantee what type of filesystem the partition will have. `touch file` changes the mtime of the file. `ln -sf` will fail if the target is a directory and is not atomic.

It would be nice to have a toolset of commands that are all idempotent to make that type of task easier.

bonyt• 7 years ago

Don’t forget mktmp for making temporary files or directories, instead of putting files in a set place in /tmp.

https://linux.die.net/man/1/mktemp

arendtio5• 7 years ago

Well, sadly mktemp is not part of POSIX. So while it is available on a wide range of systems, there are also different implementations with different options. So while I like the job it does, I am also pretty disappointed how 'unportable' scripts become as soon as they use mktemp :-/

achamayou2• 7 years ago

And anyway you really want mkstemp ;)

slrz2• 7 years ago

You can invoke mktemp(1) in ways that work across most contemporary systems. At least if you don't care that the resulting names are going to look a bit weird (have lots of XXXXXX in them due to name prefix vs. template differences among implementations).

posix_me_less5• 7 years ago

It is sad to read these comments by people believing POSIX is something relevant today. My advice is to give up on false hope that POSIX will solve portability problems. POSIX ideas of shell are 30 years old and refusing to use anything newer alone won't make your script work correctly on all systems. If feasible, install bash, which has become the standard de facto. Or just write your script to test for the shell version and then launch appropriate code path.

gabrielblack3• 7 years ago

mktemp couldn't make a script idempotent. I remember a script that failed because it needed to create temporary executables in /tmp. To apply some hardening policies, someone remounted /tmp in "noexec" mode and ... kaboom. So if you use that, better if you specify TMPDIR with proper checking.

united893• 7 years ago

This is generally terrible advice. A better option is to 'set -e' and ensure that the bash script exits when there's a failure.

Bash scripts can't be idempotent because they operate in an external environment that can't ever be. The better option is to just be extra safe.

See:

http://redsymbol.net/articles/unofficial-bash-strict-mode/

dymk8• 7 years ago

That's orthogonal to the article - in fact, the article hints that you'd take this advice when your bash script is failing midway, such as it might with `set -e`.

The article is about writing bash scripts that don't care if you've ran them once, or 10 times.

unixhero3• 7 years ago

Precisely.

farslan2• 7 years ago

Op here. Exactly this. I’ve seen many times where people had scripts and had to modify it (such as commenting code) when something has failed in the middle of the script.

news_hacker• 7 years ago

Seems like all the pedants came out for this article. In good-faith I understand the principles OP is suggesting and have first-hand experience of their usefulness. But it's easier to nit-pick I guess.

rhizome• 7 years ago

The article is about writing bash scripts that don't care if you've ran them once, or 10 times.

Is the idea that participants don't know what "idempotent" means in a thread with that word in the title?

xhgdvjky• 7 years ago

I agree. using -f is a poor excuse for not handling errors properly. imagine doing that in any other language.... oof

godelski2• 7 years ago

I'm not sure why people see errors as a bad thing. Errors are good. They stop you from doing stupid things. But maybe this article is good for telling people why you should handle errors well or when you write code to make sure it fails gracefully. Because if you don't people are going to start telling others to use the force flag when removing (almost always a terrible idea).

Learn to handle errors, don't learn to brute force your way through them.

jmts• 7 years ago

My understanding of 'set -e' is to ensure that errors are handled appropriately, not to brute force around them. If an error occurs that is not handled, this is treated as a problem and the script exits to ensure that one failure doesn't cause additional unforeseen problems as a result of the script running to completion.

empath752• 7 years ago

This is surely a good collection of flags to know, but I sort of object to the premise of the article.

There is a good reason that many of these flags are not default behavior, and it’s that they can be quite destructive.

If you’re writing a script, as the beginning scenario says, that has errors in it, and you expect to have more, do you really want to tell all the commands to power through and delete and overwrite stuff if it encounters an unexpected state? Telling people to just always use these flags and not only when they’re quite sure what their script is doing, is probably playing with fire.

This may make the script idempotent in repeated runs, but the first run might not be what you expect at all.

iforgotpassword2• 7 years ago

If you’re writing a script, as the beginning scenario says, that has errors in it, and you expect to have more, do you really want to tell all the commands to power through and delete and overwrite stuff if it encounters an unexpected state?

Where are they saying this? These flags are about making sure the script wouldn't error out right away if you run it again, eg how a regular mkdir for a path that already exists would not exit 0 and thus end script execution (given set -e is active, which the article seems to imply). Also it is not so much about purposely writing a script that contains errors, because that would just mean the error is encountered every time, but more about transient errors like running out of disk space, a curl call failing because of network hiccups etc. Everything your script did up until that failure shouldn't prevent a second call from succeeding.

mooneater• 7 years ago

Great article, but a common scenario is missing: I want to run a long running script, but not if another copy of it is already running.

dymk8• 7 years ago

https://linux.die.net/man/1/flock would probably help

fwip4• 7 years ago

A word of caution on flock: it may fail when used on some network filesystems, and it is tricky to make sure you've got it right. If you're locking on-disk it's pretty straight-forward, though.

sirn• 7 years ago

flock(1) is not POSIX, though. mkdir(1) can be used if you absolutely want a POSIX way to manage locks. For example:

    if ! mkdir .lock; then
        printf >&2 "Already running?\\n"
        exit 1
    fi

Some network file system implementation do not guarantee atomic mkdir, so you still need an extra caution with this method.

unixhero3• 7 years ago

Great article. Very helpful.

gist2• 7 years ago

A good way to remember idempotent is to think of the 'delete' or 'mark as read' function for gmail.

You can delete an email or mark it as read even if it's been done before (on another open screen as an example).

bklaasen• 7 years ago

Actually I think the "summon lift" button is a better example of idempotence. It doesn't matter how many times you mash the button, the lift is coming ASAP, no quicker. Hitting the button when it's lit doesn't cancel summoning the lift.

gist2• 7 years ago

By 'lift' you mean 'elevator'? When I first read this I thought you meant 'summon lyft' the ride service but then realized what you meant!

Interestingly in the 'old days' of elevator operators hitting it multiple times would be either a 'hurry up' for the operator or an annoyance that maybe made it come slower!

finchisko• 7 years ago

I think the problem should be solved within the OS using sandboxing, like in iOS (or to some degree Android too) and macOS and Windows store applications. For example, when you uninstall an app, there are no leftovers in the system. You don't need to check, if file exists or not ...

To some degree, this has been also solved with containers. It's much easier to create new container image, then create proper idempotent script. And you can create create image from running system too. So you can just "bash" commands and don't care about the state of the system. When done, you just create final image and you can be sure, that state of the system in the container would same as you intended.

I'm talking mostly about install scripts. Of course there are valid use cases for idempotent bash scripts.

rb808• 7 years ago

I think Python is getting so popular & ubiquitous now I can't see any reason to write bash scripts any more except for perhaps very simple ones that are just a few lines. Other people's bash scripts are regularly just too difficult and complex to maintain.

sametmax• 7 years ago

Besides, EAFP is actually a good way to make things idempotent AND free of race conditions.

Godel_unicode2• 7 years ago

Python startup time is just not in the same league as bash though, if you tried to replace bash across the board you'd end up with lots of unnecessary lag.

achamayou2• 7 years ago

Is that really true with limited imports though, and/or with precompiled bytecode?

I was surprised to find that Ubuntu’s handler for unknown commands that tries to suggest what you could apt install is written in Python. I’ve never noticed it being slow.

pmarreck3• 7 years ago

While not disagreeing, I would not agree that "everyone should go to Python" since I would personally prefer (for example) Ruby or Elixir, which can work just as well as scripting languages and are at least as readable, while also not triggering too much "VM startup" time.

Of course, others would prefer, say, Clojure, or Kotlin (although the JVM startup time is the worst of all), but therein lies the issue... We're stuck with Bash scripting because everyone (more or less) knows/understands it to some extent, it's installed everywhere, and it has essentially no startup-time cost.

pletnes2• 7 years ago

If you’re into minimal docker images, bash or similar shell might be the only script interpreter available.

Python might not exist on your windows machine.

Etc.

antpls• 7 years ago

Surely we could all agree on a minimal subset of useful Python to write scripts : loops, arrays, lists, some strings functions, etc. The resulting Python interpreter would probably be tiny.

For example, there is https://docs.bazel.build/versions/master/skylark/language.ht... which is a subset of Python.

nyuszika7h• 7 years ago

Python might not exist on your windows machine.

Bash might not exist either.

jdgiese• 7 years ago

I agree, in general, but I do think it is worth pointing out that bash is very good at what it was made for---i.e., starting processes, redirecting their output, etc.

Compare:

  { a; b && c; } ≫ log.txt

With the same written in Python:

  from subprocess import check_call, CalledProcessError
  
  f = open("log.txt", "a+")
  
  try:
      check_call(["a"], stdout=f)
  except CalledProcessError as e:
      pass
  
  try:
      check_call(["b"], stdout=f)
      check_call(["c"], stdout=f)
  except CalledProcessError as e:
      exit(e.returncode)
  
  exit(0)

(Note: I took these examples from an article I wrote a while back about programming languages https://innolitics.com/articles/programming-languages/ )

1337shadow• 7 years ago

That's why Xonsh exists !

tjoff• 7 years ago

Is there any hope for a more modern ubiquitous light weight scripting language?

It can be learned, and one can even learn to love it. But scripting is often done by people that very seldom write scripts, the barrier to make decent scripts is much too high and readability isn't much better. The nonsensical syntax is also easy to forget, the result is just an awful lot of headache and poor scripts floating around. Surely we can do better?

deepsun• 7 years ago

ln -s source target

Terminology is broken. Correct is "ln -s TARGET LINK_NAME" per its man page.

arendtio5• 7 years ago

For everybody interested in portable shell programming I like to recommend the following page which offers a good overview over POSIX compliant commands with links directly into the POSIX standard:

https://shellhaters.org

theamk2• 7 years ago

Is POSIX relevant today? I'd think that overwhelming majority of the shell scripts would require either Linux or BSD dialects.

And if one wanted to limit my shell scripts significantly, I'd limit myself to commands supported by busybox's "ash" shell -- this is a limited shell environment which is pretty widely used in initrd's and embedded devices.

rwestergren• 7 years ago

Context and purpose of the bash script in question is important here. In the example, the author is writing a simple bootstrap script for a dev machine. A number of the critiques here, while valid, are aimed at different use-cases.

suprgeek• 7 years ago

A lot of the suggestions in this post are "idempotent" only in the sense of the specific use case that the author is interested in. Please do not take this advise to be generally applicable for the uses of "idempotent" or in general to mean best practice while writing bash scripts.

The advice such as replace rm <FileName> with rm -f <FileName> could lead to Disaster depending on the scenario. So a HUGE YMMV

delinka• 7 years ago

This means you can call it multiple times without any issues.

Unless you have processes that depend on modification time.

ln -sfn

If you don’t mind the possibility of ownership changing, this is OK.

ape4• 7 years ago

Hey I just learned about `mountpoint`. Great. I used to grep the output of `df`. No more.

forty• 7 years ago

I use ansible only not to have to do this :) nice tricks though

nurettin• 7 years ago

To make your bash scripts truly idempotent, go to root and turn your system into a git repo. You're welcome.

jakub_g• 7 years ago

Another tricky thing: recursively copying a folder is not easy to get right in an idempotent way due to arcane bash behaviors:

https://unix.stackexchange.com/questions/228597/how-to-copy-...

posix_me_less5• 7 years ago

That is hardly arcane, the context dependent behaviour of cp and mv is a well-known standard. You are right it does make making a copy in a given path more complicated task.

sharperguy• 7 years ago

I like to use stuff like

mountpoint -q $MOUNTPOINT || mount ...

Instead of

if ! mountpoint -q $MOUNTPOINT; then mount ... fi

It's much more compact in the case of having a lot of commands that need to be checked in case they don't need to be run again.

CameronNemo3• 7 years ago

I was going to mention this, but frankly it is just a style preference. No real difference between either variant. Also, you can do something like:

    mountpoint -q /proc || {
        mount -t proc none /proc
        chown 0400 /proc/slabinfo
    }

if you need multiple statements.

How to write idempotent Bash scripts

Related Stories