Bash the Pipes – Geek's Smörgåsbord

Bash shell is often seen as simple launcher for other programs, when in fact it is fairly sophisticated piece of software. It has its own language, its own start-up scripts (several of them, executed depending on context) and its own job management service. In this post I explore the bash redirects, pipes and their typical use cases. I am not going to repeat basic examples that can be found in every tutorial, instead I will go over a few tricks, shortcuts and gotchas I came across while working with bash shell. I am not a bash expert, but still I hope that some of the examples may be useful.

Need to delete a content of a file? Easy, use “>” :

dd if=/dev/random bs=1 count=16 > myfile # creates a file
> myfile #removes the content of myfile

Bash allows commands to be grouped. In this case the output streams of all commands in the list are combined:
```
(echo  1_error 1>&2; echo 1_output)
```
The line above produces two messages, “1_error” goes to the combined STDERR, while “1_output” goes to the combined STDOUT. We will use this construct below.

For more details on the command grouping see here: reference.

To redirect the standard error to standard output use “2>&1”. There is also a shortcut “&>” for “2>&1 >”. Finally, “&|” is a shortcut for “2>&1 |”

(echo  1_error 1>&2; echo 1_output) #outputs to both STDOUT and STDERR
(echo  1_error 1>&2; echo 1_output)  2> /dev/null # supresses STDERR
(echo  1_error 1>&2; echo 1_output)  1> /dev/null # supresses STDOUT
(echo  1_error 1>&2; echo 1_output) | tr 1 2 # only STDOUT is tr'ed
(echo  1_error 1>&2; echo 1_output)  2>  >(tr 1 2) # only STDERR is tr'ed (we are using 'process substitution')
(echo  1_error 1>&2; echo 1_output)  2>&1 | tr 1 2 # both are tr'ed
(echo  1_error 1>&2; echo 1_output) |& tr 1 2 # both are tr'ed
(echo  1_error 1>&2; echo 1_output) &> >(tr 1 2) # both are tr'ed

The order of redirects is important:
```
(echo  1_error 1>&2; echo 1_output)  1> /dev/null 2>&1 #outputs nothing
(echo  1_error 1>&2; echo 1_output) 2>&1  1> /dev/null #outputs "1_error" to STDOUT
```
It is useful to think of a sequence of redirects as if it’s a program that operates on stream names and their locations. In the first example, “1> /dev/null” sets STDOUT to location “/dev/null”, then “2>&1” sets STDERR to point to the same location as STDOUT, i.e. “dev/null”. In the second example, STDERR is set to the location of STDOUT, and then STDOUT name is set to point to “/dev/null” location.
Let’s illustrate this by the following example:
```
(echo  1_error 1>&2; echo 1_output) 3>&1 1>&2 2>&3- | tr 1 2
```
In this example we swap STDOUT and STDERR. First, we introduce a new file handler “3” and make it point to the current location of STDOUT, then we make STDOUT point to the current location of STDERR, and finally STDERR is made to the location of ‘3’, which is the original location of STDOUT. “-” means “close the file handle after we are done”.
Some of the examples were borrowed here: http://stackoverflow.com/questions/1507816/with-bash-how-can-i-pipe-standard-error-into-another-process
An interesting application. We can highlight error messages in the command output:
```
color()(set -o pipefail; bash -c "$@" 2>&1>&3|sed $'s,.*,\e[31m&\e[m,'>&2)3>&1
```
Now execute
```
color  "(echo  1_error 1>&2; echo 1_output)"
```
to see it in action. For more details take a look here.

Sudo and tee. Redirects will fail if there is not enough permissions to write a file. As a workaround we can use “sudo tee” trick:

touch protected.file
chmod a-xrw protected.file
echo test > protected.file # fails
sudo echo test > protected.file #fails as well
echo test | sudo tee protected.file #this works
echo test | sudo tee protected.file >/dev/null #works and suppresses STDOUT as we don't need it

Programs can detect that their output is being redirected. For example, run “ls /bin” and “ls /bin > list.txt”. In the former case the output has several columns, while in the later the output is one column. To test create outputtest.sh:
```
if [[ -t 1 ]] ; then
  echo stdout is a terminal
else
  echo stdout is not a terminal
fi
```
(taken from http://stackoverflow.com/questions/9340129/test-stdout-and-stderr-redirection-in-bash-script)
The script can be confused by
```
outputtest.sh > /dev/pts/3 #still console but not the current one
```
Run-time redirection. In Linux it is possible to redirect the output of already running process. This is a bit hacky but doable. Get screenify script from here
script
Run the following script that just prints its PID
```
ruby -e 'while (1) do print("my pid is " + $$.to_s() + " \n"); sleep(10) end'&
```
Execute
```
./screenify <pid> > newoutputdestination.txt
```
Pipes can have names. I haven’t been in a situation when this functionality is needed, but it still gives an idea how things work under the hood. In one terminal window run:
```
mkfifo /tmp/mypipe
cat /tmp/mypipe
```
In another terminal execute:
```
ls /bin > /tmp/mypipe
```
The output of “ls” will appear in the first terminal window.

Statuses. There are ways to obtain the status of each command in a pipe, and there is an option that ensures that a failing command status is not lost and is returned as the status of the entire pipe:

# We can get the status for all commands in a pipe
false | true
echo "Statuses: ${PIPESTATUS[0]} ${PIPESTATUS[1]}"

# "pipefail" option forces the pipe to return the the first non-zero status (and zero otherwise)
set -o pipefail
false | true
echo "With pipefail on: $?"

# turn "pipefail" off
set +o pipefail
false | true
echo "With pipefail off: $?"

The last command in a pipe can run in the same process as the script itself. This can be convenient if we want to use the same set of variables:

#!/bin/bash
# With this option the last command in the pipe runs in the same process
# so the updates to the bash variables are visible. 
shopt -s lastpipe
n=0
yes | n=1
echo "n is $n"
# If the option is unset then the updates are lost
shopt -u lastpipe
n=0
yes | n=1
echo "n is $n"

Misc.
- If want to know what “>|” does, then check “noclobber” option. See here.
- It is possible to statically redirect outputs for all subsequent shell commands:
```
exec 2>&1
```
- “>&” is the same as “&>”, but the latter is preferable.

Leave a Reply Cancel reply