Bash patterns I use weekly

https://will-keleher.com/posts/5-Useful-Bash-Patterns.html

by gcmeplz • 5 years ago

278 112 5 years ago

jph• 5 years ago

git bisect is the "real" way to do this, but it's not something I've ever needed

git bisect is great and worth trying; it does what you're doing in your bash loop, plus faster and with more capabilities such as logging, visualizing, skipping, etc.

The syntax is: $ git bisect run <command> [arguments]

https://git-scm.com/docs/git-bisect

OskarS• 5 years ago

Yes, git bisect is the way to go: in addition to the stuff you mentioned, his method only dives into one parent branch of merge commits. git bisect handles that correctly. A gem of a tool, git bisect.

ljm2• 5 years ago

Bisect also does a binary search so if you're looking for one bad commit amongst many others, you'll find it much more quickly than linearly testing commits, one at a time, until you find a working one.

MainJane• 5 years ago

git-bisect is nice if you are looking for a git commit.

If you are looking for a limit or the failing part of a file have a look at: https://gitlab.com/ole.tange/tangetools/-/tree/master/find-f...

geocrasher2• 5 years ago

I want to like this, but the for loop is unnecessarily messy, and not correct.

   for route in foo bar baz do
   curl localhost:8080/$route
   done

That's just begging go wonky. Should be

  stuff="foo bar baz"
  for route in $stuff; do
  echo curl localhost:8080/$route
  done

Some might say that it's not absolutely necessary to abstract the array into a variable and that's true, but it sure does make edits a lot easier. And, the original is missing a semicolon after the 'do'.

I think it's one reason I dislike lists like this- a newb might look at these and stuff them into their toolkit without really knowing why they don't work. It slows down learning. Plus, faulty tooling can be unnecessarily destructive.

masklinn4• 5 years ago

Seems like you could say the same thing about every snippet e.g. running jobs is the original purpose of a shell, (3) can be achieved using job specifications (%n), and using the `jobs` command for oversight.

lambic• 5 years ago

Even more correct would be to use an array:

  stuff=("foo foo" "bar" "baz")
  for route in "${stuff[@]}"; do
    curl localhost:8080/"$route"
  done

gcmeplz6• 5 years ago

"${stuff[@]}" would be turning it back into "foo bar baz" though, right? I think if you were using arrays for this, it'd be something like:

    stuff=("foo" "bar" "baz");
    array_length=${#stuff[@]};
    for i in $(seq 0 $array_length); do
      curl localhost:8080/${stuff[i]}
    done

I'm betting even that isn't right: as soon as bash arrays are a thing, I reach for a different language.

[edit]: trying to get formatting correct

cjvirtucio• 5 years ago

and on the off-chance that `stuff` must be a space-separated string for w/e reason:

  IFS=' ' read -ra routes <<<"${stuff}"
  for route in "${routes[@]}"; do
    curl localhost:8080/"$route"
  done

chubot2• 5 years ago

Yes that is more correct, I describe that here:

Thirteen Incorrect Ways and Two Awkward Ways to Use Arrays http://www.oilshell.org/blog/2016/11/06.html

In Oil the syntax is simplified to

    const stuff = %("foo foo" bar baz)
    for route in @stuff {
      curl localhost:8080/$route  # no quotes needed
    }

xorcist• 5 years ago

The bash syntax for arrays is obtuse and no one will spot when an error creeps in. It's worth to make an effort to say with space separated strings, as long as you are with bash. More advanced data structures is often an indication that it's worth glacing at something like python.

Should you need to handle spaces, it is often much easier to go with newline separated strings and use "| while read".

This contruction has the added benefit of the data not needing to fit in your environment. This can be a real issue, and is not at all obvious when it happens.

gcmeplz6• 5 years ago

Thanks for the note about the semicolon! Added it

geocrasher2• 5 years ago

Sure thing. I apologize if my comment came across as overly negative. My goal was constructive criticism, but based on the downvotes, I'm guessing I missed the mark there.

marginalia_nu7• 5 years ago

These are the types of stuff you typically run on the fly in the command line, not full blown scripts you intended to be reused and shared.

Here's a few examples from my command history:

  for ((i=0;i<49;i++)); do wget https://neocities.org/sitemap/sites-$i.xml.gz ; done

  for f in img*.png; do echo $f; convert $f -dither Riemersma -colors 12 -remap netscape: dit_$f; done

Hell if I know what they do now, they made sense when I ran them. If I need them again, I'll type them up again.

R0flcopt3r• 5 years ago

You can use `wait` to wait for jobs to finish.

    some_command &
    some_other_command &
    wait

masklinn4• 5 years ago

One issue with that is it won't reflect the failure of those commands.

In bash you can fix that by looping around, checking for status 127, and using `-n` (which waits for the first job of the set to complete), but not all shells have `-n`.

guruparan182• 5 years ago

I am confused how this works. I would assume `SECONDS` would just be a shell variable and it was first assigned `0` and then it should stay same, why did it keep counting the seconds?

    > SECONDS
    bash: SECONDS: command not found
    > SECONDS=0; sleep 5; echo $SECONDS;
    5
    > echo "Your command completed after $SECONDS seconds";
    Your command completed after 41 seconds
    > echo "Your command completed after $SECONDS seconds";
    Your command completed after 51 seconds
    > echo "Your command completed after $SECONDS seconds";
    Your command completed after 53 seconds

gcmeplz6• 5 years ago

`SECONDS` is like `PWD`: the shell keeps track of updating it somewhere, and it'll tell you how long your shell has been running.

https://www.oreilly.com/library/view/shell-scripting-expert/...

guruparan182• 5 years ago

Thank you. That make sense now.

ghostly_s• 5 years ago

Huh, never knew about $SECONDS.

usefulcat• 5 years ago

Use for to iterate over simple lists

I definitely use this all the time. Also, generating the list of things over which to iterate using the output of a command:

    for thing in $(cat file_with_one_thing_per_line) ; do ...

phone8675309• 5 years ago

Why not?

    xargs -n1 command < file_with_one_thing_per_line

make3• 5 years ago

that's like bash 101, not sure why it's in there

barbazoo• 5 years ago

git bisect is the "real" way to do this, but it's not something I've ever needed

uh, yeah, you did need it, that's why you came up with "2. Track down a commit when a command started failing". Seriously though, git bisect is really useful to track down that bug in O(log n) rather than O(n).

masklinn4• 5 years ago

Also provides super useful commands like skipping commits because some of your colleagues are assholes and commit non-working code.

deckard1• 5 years ago

For the given command, if the assumption is the command failed recently, it's likely faster than bisect. You can start it and go grab a coffee. It's automatic.

I wish my usage of bisect were that trivial, though. Usually I need to find a bug in a giant web app. Which means finding a good commit, doing the npm install/start dance, etc. for each round.

DylanSp• 5 years ago

As long as you can detect your bug from a script, you can pass the script to git bisect run and it'll run automatically.

phist_mcgee3• 5 years ago

I've had to git bisect a react native app in a monorepo before. Took me nearly a day to track down the commit, due to the size of node_modules and the need to do a `pod install` every time.

bloopernova2• 5 years ago

This thread seems like a good place to ask this:

When you're running a script, what is the expected behaviour if you just run it with no arguments? I think it shouldn't make any changes to your system, and it should print out a help message with common options. Is there anything else you expect a script to do?

Do you prefer a script that has a set of default assumptions about how it's going to work? If you need to modify that, you pass in parameters.

Do you expect that a script will lay out the changes it's about to make, then ask for confirmation? Or should it just get out of your way and do what it was written to do?

I'm asking all these fairly basic questions because I'm trying to put together a list of things everyone expects from a script. Not exactly patterns per se, more conventions or standard behaviours.

mason55• 5 years ago

IMO any script that makes any real changes (either to the local system or remotely) should take some kind of input.

It's one thing if your script reads some stuff and prints output. Defaulting to the current working directory (or whatever makes sense) is fine.

If the script is reading config from a config file or envvars then it should still probably get some kind of confirmation if it's going to make any kind of change (of course with an option to auot-confirm via a flag like --yes).

For really destructive changes it should default to dry run and require an explicit —-execute flag but for less destructive changes I think a path as input on the command line is enough confirmation.

That being said, if it’s an unknown script I’d just read it. And if it’s a binary I’d pass —-help.

bloopernova2• 5 years ago

Thanks for the reply! I really appreciate being able to pick other folks' brains on here :)

gmuslera• 5 years ago

What is your intended audience? It is you? A batch job or called by another program? Or a person that may or not be able to read bash and will call it manually?

A good and descriptive name comes first, then the action and the people that may have to run it are next.

scbrg• 5 years ago

A script is just another command, the only difference in this case is that you wrote it and not someone else. If its purpose is to make changes, and it's obvious what changes it should make without any arguments, I'd say it can do so without further ado. poweroff doesn't ask me what I want to do - I already told it by executing it - and that's a pretty drastic change.

Commands that halt halfway through and expect user confirmation should definitely have an option to skip that behavior. I want to be able to use anything in a script of my own.

gjulianm• 5 years ago

When you're running a script, what is the expected behaviour if you just run it with no arguments? I think it shouldn't make any changes to your system, and it should print out a help message with common options. Is there anything else you expect a script to do?

Most scripts I use, custom-made or not, should be clear enough in their name for what they do. If in doubt, always call with --help/-h. But for example, it doesn't make sense that something like 'update-ca-certificates' requires arguments to execute: it's clear from the name it's going to change something.

Do you prefer a script that has a set of default assumptions about how it's going to work? If you need to modify that, you pass in parameters.

It depends. If there's a "default" way to call the script, then yes. For example, in the 'update-ca-certificates' example, just use some defaults so I don't need to read more documentation about where the certificates are stored or how to do things.

Do you expect that a script will lay out the changes it's about to make, then ask for confirmation? Or should it just get out of your way and do what it was written to do?

I don't care too much, but give me options to switch. If it does everything without asking, give me a "--dry-run" option or something that lets me check before doing anything. On the other hand, if it's asking a lot, let me specify "--yes" as apt does so that it doesn't ask me anything in automated installs or things like that.

auno• 5 years ago

What you're getting at seems to be more about CLI conventions as opposed to script conventions specifically. As such, you might want to have a look at https://clig.dev/ which is a really comprehensive document describing CLI guidelines. I can't say I've read the whole thing yet, but everything I _have_ read very much made sense.

It's been discussed here on HN before.

https://news.ycombinator.com/item?id=25304257

michaelhoffman• 5 years ago

I have something like this in my bashrc:

   preexec ()
   {
       # shellcheck disable=2034
       _CMD_START="$(date +%s)"
   }

   trap 'preexec; trap - DEBUG' DEBUG

   PROMPT_COMMAND="_CMD_STOP=\$(date +%s)
       let _CMD_ELAPSED=_CMD_STOP-_CMD_START

       if [ \$_CMD_ELAPSED -gt 5 ]; then
           _TIME_STR=\" (\${_CMD_ELAPSED}s)\"
       else
           _TIME_STR=''
       fi; "

    PS1="\n\u@\h \w\$_TIME_STR\n\\$ "

    PROMPT_COMMAND+="trap 'preexec; trap - DEBUG' DEBUG"

Whenever a command takes more than 5 s it tells me exactly how long at the next prompt.

I didn't know about `$SECONDS` so I'm going to change it to use that.

masklinn4• 5 years ago

FWIW that is supported by many zsh themes like pure, p9k, p10k, ... (and often enabled by default).

Also:

    REPORTTIME
    If nonnegative, commands whose combined user and system execution times (measured in seconds) are greater than this value have timing statistics printed for them.

which is slightly different but generally useful, and built-in.

spondyl• 5 years ago

Came here to +1 REPORTTIME for zsh

oweiler• 5 years ago

Bash 5.0 also has $EPOCHREALTIME

caymanjim• 5 years ago

Installing GNU stuff with the 'g' prefix (gsed instead of sed) means having to remember to include the 'g' when you're on a Mac and leave it off when you're on Linux, or use aliases, or some other confusing and inconvenient thing, and then if you're writing a script meant for multi-platform use, it still won't work. I find it's a much better idea to install the entire GNU suite without the 'g' prefix and use PATH to control which is used. I use MacPorts to do this (/opt/local/libexec/gnubin), and even Homebrew finally supports this, although it does it in a stupid way that requires adding a PATH element for each individual GNU utility (e.g. /usr/local/opt/gnu-sed/libexec/gnubin).

l0b02• 5 years ago

  > 1. Find and replace a pattern in a codebase with capture groups
  > git grep -l pattern | xargs gsed -ri 's|pat(tern)|\1s are birds|g'

Or, in IDEA, Ctrl-Shift-r, put "pat(tern)" in the first box and "$1s are birds" in the second box, Alt-a, boom. Infinitely easier to remember, and no chance of having to deal with any double escaping.

marginalia_nu7• 5 years ago

Using an IDE kind of handicaps you to only working with your IDE though. The shell works everywhere for every use case.

withinboredom2• 5 years ago

That particular IDE works on Windows though… no idea how to use Powershell…

l0b02• 5 years ago

I've heard this many times, and I don't really understand the argument. I've used the shell plenty of times for this sort of work, but it's much more complicated to do certain things correctly than in an IDE. Things like looping over any filename, escaping arbitrary strings for use within sed patterns, xargs, multi-line replacements and regex lookahead require quite some in-depth knowledge, trial and error, and sometimes installing special tools because the defaults don't support the relevant patterns (like path NUL terminators or lookaheads). I don't have to deal with any of these in IDEA.

turbocon• 5 years ago

Yea, I've yet to come across a regex replacement tool as easy to use as jetbrains find & replace. Invaluable for certain tasks.

DocTomoe• 5 years ago

What you are doing here is to propose a very specialist approach ("Why not just use a SpaceX Merlin Engine, this is ?") when a slightly more cumbersome general approach ("This is how you get from A to B") was described.

IDEA is nice if you have IDEA.

"But not everyone uses Bash" - very correct (more fond of zsh, personally), but this article is specifically about Bash.

stewartbutler• 5 years ago

I've always had trouble getting `for` loops to work predictably, so my common loop pattern is this:

    grep -l -r pattern /path/to/files | while read x; do echo $x; done

or the like.

This uses bash read to split the input line into words, then each word can be accessed in the loop with variable `$x`. Pipe friendly and doesn't use a subshell so no unexpected scoping issues. It also doesn't require futzing around with arrays or the like.

One place I do use bash for loops is when iterating over args, e.g. if you create a bash function:

    function my_func() {
        for arg; do
            echo $arg
        done
    }

This'll take a list of arguments and echo each on a separate line. Useful if you need a function that does some operation against a list of files, for example.

Also, bash expansions (https://www.gnu.org/software/bash/manual/html_node/Shell-Par...) can save you a ton of time for various common operations on variables.

jolmg3• 5 years ago

I've always had trouble getting `for` loops to work predictably, so my common loop pattern is this:

The problem with the pipe-while-read pattern is that you can't modify variables in the loop, since it runs in a subshell.

edgyquant4• 5 years ago

It’s a trade off. I had to use a piped loop earlier this year to extract fallout new Vegas mods and textures since they all had spaces in their file names. For this it was perfect to pipe a list of the names to a loop, but for 99% of things I just use a for loop

chubot2• 5 years ago

BTW you can make it work in bash by setting shopt -s lastpipe. It runs the last part of the pipeline in the main shell, so the mutation variables of will persist.

Both OSH and zsh behave that way by default, which was tangential to a point in the latest release notes https://news.ycombinator.com/item?id=29292187

Another trick I've seen in POSIX shell is to add a subshell after the pipeline until the last time you want to read the variable. Like

    cat foo.txt | ( while read line; do
      f=$line
    done

    echo "we're still in the subshell f=$f"
    )

edgyquant4• 5 years ago

For me I always used for loops and only recently (after a decade of using Linux daily) have learned about the power of piped-loops. It’s strange to me you are more comfortable with those than for loops, but I think it does make sense as you’re letting a program generate the list to iterate over. A pain point in for loops is getting that right, e.g. there isn’t a good way to iterate over files with spaces in them using a for loop (this is why I learned about piped loops recently.)

jolmg3• 5 years ago

A pain point in for loops is getting that right, e.g. there isn’t a good way to iterate over files with spaces in them using a for loop

If those files came as arguments, you can use a for-loop as long as they're kept in an array:

  for f in "${files[@]}";

That handles even newlines in the filenames, while I'm not sure if you can handle that with a while-read-loop. IFS=$'\0' doesn't seem to cut it.

for-loops seem preferable for working with filenames. If a command is generating the list, then something like `xargs -0` is preferable.

zwayhowder• 5 years ago

You can also have your script change the bash file seperator.

https://bash.cyberciti.biz/guide/$IFS

Something I wish I'd learned 23 years ago instead of 3 years ago :(

adityaathalye• 5 years ago

Once I discovered functions are pipeline-friendly, I started pipelining all the things. Almost any `for arg` can be re-designed as a `while read arg` with a function.

Here's what it can look like. Some functions I wrote to bulk-process git repos. Notice they accept arguments and stdin:

  # ID all git repos anywhere under this directory
  ls_git_projects ~/src/bitbucket |
      # filter ones not updated for pre-set num days    
      take_stale |
      # proc repo applies the given op. (a function in this case) to each repo
      proc_repos git_fetch

Source: https://github.com/adityaathalye/bash-toolkit/blob/master/bu...

The best part is sourcing pipeline-friendly functions into a shell session allows me to mix-and-match them with regular unix tools.

Overall, I believe (and my code will betray it) functional programming style is a pretty fine way to live in shell!

hnlmorg2• 5 years ago

I've always had trouble getting `for` loops to work predictably, so my common loop pattern is this:

for loops were exactly the pain point that lead me to write my own shell > 6 years ago.

I can now iterate through structured data (be it JSON, YAML, CSV, `ps` output, log file entries, or whatever) and each item is pulled intelligently rather than having to conciously consider a tonne of dumb edge cases like "what if my file names have spaces in them"

    » open https://api.github.com/repos/lmorg/murex/issues -> foreach issue { out "$issue[number]: $issue[title]" }
    380: Fail if variable is missing
    379: Backslashes and code comments
    378: Improve testing facility documentation
    377: v2.4 release
    361: Deprecate `swivel-table` and `swivel-datatype`
    360: `sort` converts everything to a string
    340: `append` and `prepend` should `ReadArrayWithType`

Github repo: https://github.com/lmorg/murex

Docs on `foreach`: https://murex.rocks/docs/commands/foreach.html

juki• 5 years ago

Powershell is also a good option nowadays (although a lot of people on HN seem to dismiss it for various, imo rather superficial, reasons).

  PS> (irm https://api.github.com/repos/lmorg/murex/issues) | % { echo "$($_.number): $($_.title)" }
  380: Fail if variable is missing
  379: Backslashes and code comments
  378: Improve testing facility documentation
  377: v2.4 release
  361: Deprecate `swivel-table` and `swivel-datatype`
  360: `sort` converts everything to a string
  340: `append` and `prepend` should `ReadArrayWithType`

Or just

  PS> (irm https://api.github.com/repos/lmorg/murex/issues) | format-table number, title

  number title
  ------ -----
     380 Fail if variable is missing
     379 Backslashes and code comments
     378 Improve testing facility documentation
     377 v2.4 release
     361 Deprecate `swivel-table` and `swivel-datatype`
     360 `sort` converts everything to a string
     340 `append` and `prepend` should `ReadArrayWithType`

Or even `(irm https://api.github.com/repos/lmorg/murex/issues) | select number, title | out-gridview`, which would open a GUI list (with sorting and filtering), but I think that only works on Windows.

marcodiego• 5 years ago

We need a simpler regex format. One that allows easy searching and replacing in source code. Of course, some IDE's already to that pretty well, but I'd like to be able to do it from the command line with a stand alone tool I can easily use in scripts.

The simplest thing I know that is able to do that is coccinelle, but even coccinelle is not handy enough.

f0e4c2f7• 5 years ago

It's not perfect but I like sed for this.

marginalia_nu7• 5 years ago

I think what we really need is one regex format. You have POSIX, PCRE, and also various degrees of needing to double-escape the slashes to get past whatever language you're using the regex in. Always adds a large element of guesswork even when you are familiar with regular expressions.

Something1234• 5 years ago

https://regex101.com/

yesenadam2• 5 years ago

I think what we really need is one regex format.

See https://xkcd.com/927/ How standards proliferate

unixhero• 5 years ago

A lot of genious moves here I have never seen or thought of. Brilliant. Grabbing pids was mindblowingly effective.

Hikikomori• 5 years ago

My daily bash pattern.

  cat file
  cat file | grep something

Eduard• 5 years ago

https://security.stackexchange.com/questions/56307/can-cat-i...

phist_mcgee3• 5 years ago

I prefer less tbh, less pollution of the term output.

amp108• 5 years ago

You could just `grep "something" file.

penguin_booze2• 5 years ago

Indeed - it's a known anti-pattern[0].

[0] https://en.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_cat

gnubison• 5 years ago

It’s simple enough to use macos sed here instead of needing gsed: just use ‘-i ''’ instead of ‘-i’.

Arch-TK2• 5 years ago

This post and comments section means I no longer wonder why 99% of shell scripts I come across look inept. I'm sorry guys but seriously please actually learn bash (and ideally not from this blog post). There's so many things wrong in the post and the comments that it's difficult to enumerate.

To start with, if you ever feel the need to write a O(n) for loop for finding which commit broke your build, you DID need git-bisect.

Definitely DONT'T wait for PIDs like that and if you do want to write code like that, maybe actually use the arrays bash provides?

If your complaint is that for a in b c d; do ...; done is unclear, then maybe also use lists there, because what's definitely LESS clear is putting things in a variable and relying on splitting.

And most importantly, DO quote things (except when it's unnecessary).

gcmeplz6• 5 years ago

Definitely DONT'T wait for PIDs like that and if you do want to write code like that, maybe actually use the arrays bash provides?

What pattern would you recommend for waiting for PIDs / parallelizing commands & preserving exit codes? Fair point about arrays being a better fit rather than a string there.

IndexPointer• 5 years ago

Try GNU parallel.

Arch-TK2• 5 years ago

So in this case there's a couple notable things:

`wait` can take no parameters this means that if you just ran a bunch of things in the background in your script and want to wait for all of them to finish, you don't need to track the PIDs or a loop, you can just `wait`.

`wait` is a bash builtin (in this case) and as such it has no parameter limit (although I am told that actually there are some weird limits but it's very unlikely you will be able to spawn enough processes at once from a bash script to hit the limits). Given an array `pids` you can just do: `wait "${pids[@]}"`

The only problem with the two above approaches is that in the former case, wait loses the return status and in the second case wait loses all but the status of the last ID you pass it, so the third option is:

    pids=()
    do_thing_1 &
    pids+=("$!")
    do_thing_2 &
    pids+=("$!")
    for pid in "${pids[@]}"; do
        wait "$pid" || status=$?
    done
    exit "${status-0}"

Now you only have the issue left that this will report the LAST failing status.

dang• 5 years ago

Please don't call names or post supercilious putdowns. It's not the culture we want here.

If you know more than others, that's wonderful and it's great to share some of what you know so the rest of us can learn. But please do it without putdowns, and please do it in a way people can actually learn from. "Definitely DONT'T wait for PIDs like that" doesn't actually explain anything.

https://news.ycombinator.com/newsguidelines.html

1vuio0pswjnm78• 5 years ago

To conserve host resources RFC 2616 recommends making multiple HTTP requests over a single TCP connection ("HTTP/1.1 pipelining").

The cURL project said it never properly suported HTTP/1.1 pipelining and in 2019 it said it was removed once and for all.

https://daniel.haxx.se/blog/2019/04/06/curl-says-bye-bye-to-...

Anyway, curl is not needed. One can write a small program in their language of choice to generate HTTP/1.1, but even a simple shell script will work. Even more, we get easy control over SNI which curl binary does have have.

There are different and more concise ways, but below is an example, using the IFS technique.

This also shows the use of sed's "P" and "D" commands (credit: Eric Pement's sed one-liners).

Assumes valid, non-malicious URLs, all with same host.

Usage: 1.sh < URLs.txt

       #!/bin/sh
       (IFS=/;while read w x y z;do
       case $w in http:|https:);;*)exit;esac;
       case $x in "");;*)exit;esac;
       echo $y > .host
       printf '%s\r\n' "GET /$z HTTP/1.1";
       printf '%s\r\n' "Host: $y";
       # add more headers here if desired;
       printf 'Connection: keep-alive\r\n\r\n';done|sed 'N;$!P;$!D;$d';
       printf 'Connection: close\r\n\r\n';
       ) >.http
       read x < .host;
       # SNI;
       #openssl s_client -connect $x:443 -ign_eof -servername $x < .http;
       # no SNI;
       openssl s_client -connect $x:443 -ign_eof -noservername < .http;
       exec rm .host .http;

themk2• 5 years ago

The curl binary will reuse the TCP connection when fed multiple URLs. Infact it can even use HTTP2 and make the requests in parallel over a single TCP connection. Common pattern I use is to construct URLs with a script and use xargs to feed to curl.

1vuio0pswjnm78• 5 years ago

For HTTP/1.1 pipelining, not HTTP/2 which not all websites support, the curl binary must be slower because the program tries to do more than just make HTTP from URLs and send the text over TCP. It tries to be "smart" and that slows it down. But dont't take my word for it, test it.

For example, compare the retrieval speed of the above to something like

     sed -n '/^http/s/^/url=/p' URLs.txt|curl -K- --http1.1

1vuio0pswjnm78• 5 years ago

Heres another way to do it without the subshell, using tr.

       #!/bin/sh
       IFS=/;while read w x y z;do
       v=$(echo x|tr x '\34');
       case $w in http:|https:);;*)exit;esac;
       case $x in "");;*)exit;esac;
       echo $y > .host
       printf '%s\r\n' "GET /$z HTTP/1.1";
       printf '%s\r\n' "Host: $y";
       printf 'Connection: keep-alive'$v$v;done \
       |sed '$s/keep-alive/close/'|tr '\34\34' '\r\n' > .http;
       read x < .host;
       # SNI;
       #openssl s_client -connect $x:443 -ign_eof -servername $x < .http;
       # no SNI;
       openssl s_client -connect $x:443 -ign_eof -noservername < .http;
       exec rm .host .http;

1vuio0pswjnm78• 5 years ago

   case $x in "");;*)exit;esac

is better written as

   test ${#x} = 0||exit

notatoad• 5 years ago

I really love this style of blog post. Short, practical, no backstory, and not trying to claim one correct way to do anything. Just an unopinionated share that was useful to the author.

It seems like a throwback to a previous time, but honestly can't remember when that was. Maybe back to a time when I hoped this was what blogging could be.

js2• 5 years ago

My git-gsr (global search replace) command:

https://gist.github.com/jaysoffian/0eda35a6a41f500ba5c458f02...

Uses perl instead of gsed, defaults to fixed strings but supports perl regexes, properly handles filenames with whitespace.

penguin_booze2• 5 years ago

If I find myself running a set of commands in parallel, I'd keep a cheap Makefile around: individual commands I want to run in parallel will be written as phony targets:

  all: cmd1 cmd2
  
  cmd1:
      sleep 5
  
  cmd2:
      sleep 10

And then

  make -j[n]

harvie• 5 years ago

curl localhost:8080/{foo,bar,baz}

Bash patterns I use weekly

Related Stories