Bash Golf Part 2

Published at 2022-01-01T23:36:15+00:00; Updated at 2022-01-05

This is the second blog post about my Bash Golf series. This series is random Bash tips, tricks and weirdnesses I came across. It's a collection of smaller articles I wrote in an older (in German language) blog, which I translated and refreshed with some new content.

2021-11-29 Bash Golf Part 1
2022-01-01 Bash Golf Part 2 (You are currently reading this)
2023-12-10 Bash Golf Part 3

    '\       '\                   .  .                |>18>>
      \        \              .         ' .           |
     O>>      O>>         .                 'o        |
      \       .\. ..   .                              |
      /\    .  /\    . .                              |
     / /   .  / /  .'    .                            |
jgs^^^^^^^`^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                        Art by Joan Stark, mod. by Paul Buetow

Bash Golf Part 2
⇢ Redirection
⇢ HERE
⇢ RANDOM
⇢ set -x and set -e and pipefile
⇢ ⇢ -x
⇢ ⇢ -e
⇢ ⇢ pipefail

Redirection

Let's have a closer look at Bash redirection. As you might already know that there are 3 standard file descriptors:

0 aka stdin (standard input)
1 aka stdout (standard output)
2 aka stderr (standard error output)

These are most certainly the ones you are using on regular basis. "/proc/self/fd" lists all file descriptors which are open by the current process (in this case: the current Bash shell itself):

❯ ls -l /proc/self/fd/
total 0
lrwx------. 1 paul paul 64 Nov 23 09:46 0 -> /dev/pts/9
lrwx------. 1 paul paul 64 Nov 23 09:46 1 -> /dev/pts/9
lrwx------. 1 paul paul 64 Nov 23 09:46 2 -> /dev/pts/9
lr-x------. 1 paul paul 64 Nov 23 09:46 3 -> /proc/162912/fd

The following examples demonstrate two different ways to accomplish the same thing. The difference is that the first command is directly printing out "Foo" to stdout and the second command is explicitly redirecting stdout to its own stdout file descriptor:

❯ echo Foo
Foo
❯ echo Foo > /proc/self/fd/0
Foo

Update: A reader pointed out, that the redirection should actually go to /proc/self/fd/1 and not 0. But apparently, either way works for this particular example. Do you know why?

Other useful redirections are:

Redirect stderr to stdin: "echo foo 2>&1"
Redirect stdin to stderr: "echo foo >&2"

It is, however, not possible to redirect multiple times within the same command. E.g. the following won't work. You would expect stdin to be redirected to stderr and then stderr to be redirected to /dev/null. But as the example shows, Foo is still printed out:

❯ echo Foo 1>&2 2>/dev/null
Foo

Update: A reader sent me an email and pointed out that the order of the redirections is important.

As you can see, the following will not print out anything:

❯ echo Foo 2>/dev/null 1>&2
❯

A good description (also pointed out by the reader) can be found here:

Order of redirection

Ok, back to the original blog post. You can also use grouping here (neither of these commands will print out anything to stdout):

❯ { echo Foo 1>&2; } 2>/dev/null
❯ ( echo Foo 1>&2; ) 2>/dev/null
❯ { { { echo Foo 1>&2; } 2>&1; } 1>&2; } 2>/dev/null
❯ ( ( ( echo Foo 1>&2; ) 2>&1; ) 1>&2; ) 2>/dev/null
❯

A handy way to list all open file descriptors is to use the "lsof" command (that's not a Bash built-in), whereas $$ is the process id (pid) of the current shell process:

❯ lsof -a -p $$ -d0,1,2
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
bash    62676 paul    0u   CHR  136,9      0t0   12 /dev/pts/9
bash    62676 paul    1u   CHR  136,9      0t0   12 /dev/pts/9
bash    62676 paul    2u   CHR  136,9      0t0   12 /dev/pts/9

Let's create our own descriptor "3" for redirection to a file named "foo":

❯ touch foo
❯ exec 3>foo # This opens fd 3 and binds it to file foo.
❯ ls -l /proc/self/fd/3
l-wx------. 1 paul paul 64 Nov 23 10:10 \
    /proc/self/fd/3 -> /home/paul/foo
❯ cat foo
❯ echo Bratwurst >&3
❯ cat foo
Bratwurst
❯ exec 3>&- # This closes fd 3.
❯ echo Steak >&3
-bash: 3: Bad file descriptor

You can also override the default file descriptors, as the following example script demonstrates:

❯ cat grandmaster.sh
#!/usr/bin/env bash

# Write a file data-file containing two lines
echo Learn You a Haskell > data-file
echo for Great Good >> data-file

# Link fd with fd 6 (saves default stdin)
exec 6<&0

# Overwrite stdin with data-file
exec < data-file

# Read the first two lines from it
declare LINE1 LINE2
read LINE1
read LINE2

# Print them
echo First line: $LINE1
echo Second line: $LINE2

# Restore default stdin and delete fd 6
exec 0<&6 6<&-

Let's execute it:

❯ chmod 750 ./grandmaster.sh
❯ ./grandmaster.sh
First line: Learn You a Haskell
Second line: for Great Good

HERE

I have mentioned HERE-documents and HERE-strings already in this post. Let's do some more examples. The following "cat" receives a multi line string from stdin. In this case, the input multi line string is a HERE-document. As you can see, it also interpolates variables (in this case the output of "date" running in a subshell).

❯ cat <<END
> Hello World
> It’s $(date)
> END
Hello World
It's Fri 26 Nov 08:46:52 GMT 2021

You can also write it the following way, but that's less readable (it's good for an obfuscation contest):

❯ <<END cat
> Hello Universe
> It’s $(date)
> END
Hello Universe
It's Fri 26 Nov 08:47:32 GMT 2021

Besides of an HERE-document, there is also a so-called HERE-string. Besides of...

❯ declare VAR=foo
❯ if echo "$VAR" | grep -q foo; then
> echo '$VAR ontains foo'
> fi
$VAR ontains foo

...you can use a HERE-string like that:

❯ if grep -q foo <<< "$VAR"; then
> echo '$VAR contains foo'
> fi
$VAR contains foo

Or even shorter, you can do:

❯ grep -q foo <<< "$VAR" && echo '$VAR contains foo'
$VAR contains foo

You can also use a Bash regex to accomplish the same thing, but the points of the examples so far were to demonstrate HERE-{documents,strings} and not Bash regular expressions:

❯ if [[ "$VAR" =~ foo ]]; then echo yay; fi
yay

You can also use it with "read":

❯ read a <<< ja
❯ echo $a
ja
❯ read b <<< 'NEIN!!!'
❯ echo $b
NEIN!!!
❯ dumdidumstring='Learn you a Golang for Great Good'
❯ read -a words <<< "$dumdidumstring"
❯ echo ${words[0]}
Learn
❯ echo ${words[3]}
Golang

The following is good for an obfuscation contest too:

❯ echo 'I like Perl too' > perllove.txt
❯ cat - perllove.txt <<< "$dumdidumstring"
Learn you a Golang for Great Good
I like Perl too

RANDOM

Random is a special built-in variable containing a different pseudo random number each time it's used.

❯ echo $RANDOM
11811
❯ echo $RANDOM
14997
❯ echo $RANDOM
9104

That's very useful if you want to randomly delay the execution of your scripts when you run it on many servers concurrently, just to spread the server load (which might be caused by the script run) better.

Let's say you want to introduce a random delay of 1 minute. You can accomplish it with:

❯ cat ./calc_answer_to_ultimate_question_in_life.sh
#!/usr/bin/env bash

declare -i MAX_DELAY=60

random_delay () {
    local -i sleep_for=$((RANDOM % MAX_DELAY))
    echo "Delaying script execution for $sleep_for seconds..."
    sleep $sleep_for
    echo 'Continuing script execution...'
}

main () {
    random_delay
    # From here, do the real work. Calculating the answer to
    # the ultimate question can take billions of years....
    : ....
}

main

❯
❯ ./calc_answer_to_ultimate_question_in_life.sh
Delaying script execution for 42 seconds...
Continuing script execution...

set -x and set -e and pipefile

In my opinion, -x and -e and pipefile are the most useful Bash options. Let's have a look at them one after another.

-x

-x prints commands and their arguments as they are executed. This helps to develop and debug your Bash code:

❯ set -x
❯ square () { local -i num=$1; echo $((num*num)); }
❯ num=11; echo "Square of $num is $(square $num)"
+ num=11
++ square 11
++ local -i num=11
++ echo 121
+ echo 'Square of 11 is 121'
Square of 11 is 121

You can also set it when calling an external script without modifying the script itself:

❯ bash -x ./half_broken_script_to_be_debugged.sh

Let's do that on one of the example scripts we covered earlier:

❯ bash -x ./grandmaster.sh
+ bash -x ./grandmaster.sh
+ echo Learn You a Haskell
+ echo for Great Good
+ exec
+ exec
+ declare LINE1 LINE2
+ read LINE1
+ read LINE2
+ echo First line: Learn You a Haskell
First line: Learn You a Haskell
+ echo Second line: for Great Good
Second line: for Great Good
+ exec
❯

-e

This is a very important option you want to use when you are paranoid. This means, you should always "set -e" in your scripts when you need to make absolutely sure that your script runs successfully (with that I mean that no command should exit with an unexpected status code).

Ok, let's dig deeper:

❯ help set | grep -- -e
      -e  Exit immediately if a command exits with a non-zero status.

As you can see in the following example, the Bash terminates after the execution of "grep" as "foo" is not matching "bar". Therefore, grep exits with 1 (unsuccessfully) and the shell aborts. And therefore, "bar" will not be printed out anymore:

❯ bash -c 'set -e; echo hello; grep -q bar <<< foo; echo bar'
hello
❯ echo $?
1

Whereas the outcome changes when the regex matches:

❯ bash -c 'set -e; echo hello; grep -q bar <<< barman; echo bar'
hello
bar
❯ echo $?
0

So does it mean that grep will always make the shell terminate whenever its exit code isn't 0? This will render "set -e" quite unusable. Frankly, there are other commands where an exit status other than 0 should not terminate the whole script abruptly. Usually, what you want is to branch your code based on the outcome (exit code) of a command:

❯ bash -c 'set -e
>    grep -q bar <<< foo
>    if [ $? -eq 0 ]; then
>        echo "matching"
>    else
>        echo "not matching"
>    fi'
❯ echo $?
1

...but the example above won't reach any of the branches and won't print out anything, as the script terminates right after grep.

The proper solution is to use grep as an expression in a conditional (e.g. in an if-else statement):

❯ bash -c 'set -e
>    if grep -q bar <<< foo; then
>        echo "matching"
>    else
>        echo "not matching"
>    fi'
not matching
❯ echo $?
0
❯ bash -c 'set -e
>    if grep -q bar <<< barman; then
>        echo "matching"
>    else
>        echo "not matching"
>    fi'
matching
❯ echo $?
0

You can also temporally undo "set -e" if there is no other way:

❯ cat ./e.sh
#!/usr/bin/env bash

set -e

foo () {
    local arg="$1"; shift

    if [ -z "$arg" ]; then
        arg='You!'
    fi
    echo "Hello $arg"
}

bar () {
    # Temporally disable e
    set +e
    local arg="$1"; shift
    # Enable e again.
    set -e

    if [ -z "$arg" ]; then
        arg='You!'
    fi
    echo "Hello $arg"
}

# Will succeed
bar World
foo Universe
bar

# Will terminate the script
foo

❯ ./e.sh
Hello World
Hello Universe
Hello You!

Why does calling "foo" with no arguments make the script terminate? Because as no argument was given, the "shift" won't have anything to do as the argument list $@ is empty, and therefore "shift" fails with a non-zero status.

Why would you want to use "shift" after function-local variable assignments? Have a look at my personal Bash coding style guide for an explanation :-):

./2021-05-16-personal-bash-coding-style-guide.html

pipefail

The pipefail option makes it so that not only the exit code of the last command of the pipe counts regards its exit code but any command of the pipe:

❯ help set | grep pipefail -A 2
    pipefail     the return value of a pipeline is the status of
                 the last command to exit with a non-zero status,
                 or zero if no command exited with a non-zero status

The following greps for paul in passwd and converts all lowercase letters to uppercase letters. The exit code of the pipe is 0, as the last command of the pipe (converting from lowercase to uppercase) succeeded:

❯ grep paul /etc/passwd | tr '[a-z]' '[A-Z]'
PAUL:X:1000:1000:PAUL BUETOW:/HOME/PAUL:/BIN/BASH
❯ echo $?
0

Let's look at another example, where "TheRock" doesn't exist in the passwd file. However, the pipes exit status is still 0 (success). This is so because the last command ("tr" in this case) still succeeded. It is just that it didn't get any input on stdin to process:

❯ grep TheRock /etc/passwd
❯ echo $?
1
❯ grep TheRock /etc/passwd | tr '[a-z]' '[A-Z]'
❯ echo $?
0

To change this behaviour, pipefile can be used. Now, the pipes exit status is 1 (fail), because the pipe contains at least one command (in this case grep) which exited with status 1:

❯ set -o pipefail
❯ grep TheRock /etc/passwd | tr '[a-z]' '[A-Z]'
❯ echo $?
1

E-Mail your comments to paul@nospam.buetow.org :-)

Other related posts are:

2021-05-16 Personal Bash coding style guide
2021-06-05 Gemtexter - One Bash script to rule it all
2021-11-29 Bash Golf Part 1
2022-01-01 Bash Golf Part 2 (You are currently reading this)
2023-12-10 Bash Golf Part 3

Back to the main site