Bash the Pipes

By | September 28, 2013

Bash shell is often seen as simple launcher for other programs, when in fact it is fairly sophisticated piece of software. It has its own language, its own start-up scripts (several of them, executed depending on context) and its own job management service.   In this post I explore the bash redirects, pipes and their typical use cases. I am not going to repeat basic examples that can be found in every tutorial, instead I will go over a few tricks, shortcuts and gotchas I came across while working with bash shell. I am not a bash expert, but still I hope that some of the examples may be useful.

  1. Need to delete a content of a file? Easy, use “>” :
    dd if=/dev/random bs=1 count=16 > myfile # creates a file
    > myfile #removes the content of myfile
  2. Bash allows commands to be grouped. In this case the output streams of all commands in the list are combined:
    (echo  1_error 1>&2; echo 1_output)

    The line above produces two messages, “1_error” goes to the combined STDERR, while “1_output” goes to the combined STDOUT. We will use this construct below.

    For more details on the command grouping see here: reference.

  3. To redirect the standard error to standard output use “2>&1”. There is also a shortcut   “&>” for “2>&1  >”.  Finally, “&|” is a shortcut for “2>&1 |”
    (echo  1_error 1>&2; echo 1_output) #outputs to both STDOUT and STDERR
    (echo  1_error 1>&2; echo 1_output)  2> /dev/null # supresses STDERR
    (echo  1_error 1>&2; echo 1_output)  1> /dev/null # supresses STDOUT
    (echo  1_error 1>&2; echo 1_output) | tr 1 2 # only STDOUT is tr'ed
    (echo  1_error 1>&2; echo 1_output)  2>  >(tr 1 2) # only STDERR is tr'ed (we are using 'process substitution')
    (echo  1_error 1>&2; echo 1_output)  2>&1 | tr 1 2 # both are tr'ed
    (echo  1_error 1>&2; echo 1_output) |& tr 1 2 # both are tr'ed
    (echo  1_error 1>&2; echo 1_output) &> >(tr 1 2) # both are tr'ed
  4. The order of redirects is important:
    (echo  1_error 1>&2; echo 1_output)  1> /dev/null 2>&1 #outputs nothing
    (echo  1_error 1>&2; echo 1_output) 2>&1  1> /dev/null #outputs "1_error" to STDOUT

    It is useful to think of a sequence of redirects as if it’s a program that operates on stream names and their locations. In the first example,  “1> /dev/null” sets STDOUT to location “/dev/null”, then “2>&1” sets STDERR to point to the same location as STDOUT, i.e. “dev/null”. In the second example, STDERR is set to the location of STDOUT, and then STDOUT name is set to point to “/dev/null” location.
    Let’s illustrate this by the following example:

    (echo  1_error 1>&2; echo 1_output) 3>&1 1>&2 2>&3- | tr 1 2

    In this example we swap STDOUT and STDERR. First, we introduce a new file handler “3” and make it point to the current location of STDOUT, then we make STDOUT point to the current location of STDERR, and finally STDERR is made to the location of ‘3’, which is the original location of STDOUT. “-” means “close the file handle after we are done”.
    Some of the examples were borrowed here:   http://stackoverflow.com/questions/1507816/with-bash-how-can-i-pipe-standard-error-into-another-process

  5. An interesting application. We can highlight error messages in the command output:
    color()(set -o pipefail; bash -c "$@" 2>&1>&3|sed $'s,.*,\e[31m&\e[m,'>&2)3>&1

    Now execute

    color  "(echo  1_error 1>&2; echo 1_output)"

    to see it in action. For more details take a look here.

  6. Sudo and tee. Redirects will fail if there is not enough permissions to write a file. As a workaround we can use “sudo tee” trick:
    touch protected.file
    chmod a-xrw protected.file
    echo test > protected.file # fails
    sudo echo test > protected.file #fails as well
    echo test | sudo tee protected.file #this works
    echo test | sudo tee protected.file >/dev/null #works and suppresses STDOUT as we don't need it
  7. Programs can detect that their output is being redirected. For example, run “ls /bin” and “ls /bin > list.txt”. In the former case the output has several columns, while in the later the output is one column. To test create outputtest.sh:
    if [[ -t 1 ]] ; then
      echo stdout is a terminal
    else
      echo stdout is not a terminal
    fi
    

    (taken from http://stackoverflow.com/questions/9340129/test-stdout-and-stderr-redirection-in-bash-script)
    The script can be confused by

    outputtest.sh > /dev/pts/3 #still console but not the current one
  8. Run-time redirection. In Linux it is possible to redirect the output of already running process. This is a bit hacky but doable. Get screenify script from here
    script
    Run the following script that just prints its PID

    ruby -e 'while (1) do print("my pid is " + $$.to_s() + " \n"); sleep(10) end'&

    Execute

    ./screenify <pid> > newoutputdestination.txt
  9. Pipes can have names. I haven’t been in a situation when this functionality is needed, but it still gives an idea how things work under the hood. In one terminal window run:
    mkfifo /tmp/mypipe
    cat /tmp/mypipe

    In another terminal execute:

    ls /bin > /tmp/mypipe

    The output of “ls” will appear in the first terminal window.

  10. Statuses. There are ways to obtain the status of each command in a pipe, and there is an option that ensures that a failing command status is not lost and is returned as the status of the entire pipe:
    # We can get the status for all commands in a pipe
    false | true
    echo "Statuses: ${PIPESTATUS[0]} ${PIPESTATUS[1]}"
    
    # "pipefail" option forces the pipe to return the the first non-zero status (and zero otherwise)
    set -o pipefail
    false | true
    echo "With pipefail on: $?"
    
    # turn "pipefail" off
    set +o pipefail
    false | true
    echo "With pipefail off: $?"
    
    
  11. The last command in a pipe can run in the same process as the script itself. This can be convenient if we want to use the same set of variables:
    #!/bin/bash
    # With this option the last command in the pipe runs in the same process
    # so the updates to the bash variables are visible. 
    shopt -s lastpipe
    n=0
    yes | n=1
    echo "n is $n"
    # If the option is unset then the updates are lost
    shopt -u lastpipe
    n=0
    yes | n=1
    echo "n is $n"
    
  12. Misc.
    • If want to know what “>|” does, then check “noclobber” option. See here.
    • It is possible to statically redirect outputs for all subsequent shell commands:
      exec 2>&1
    • “>&” is the same as “&>”, but the latter is preferable.

Leave a Reply

Your email address will not be published. Required fields are marked *