17. Redirection

Unix shells commonly provide several forms of redirection--ways to change the input source or output destination of a command. But one useful redirection is performed by a separate command, not by the shell; it's described here.

[ < ]

[ > ]

[ << ]

[ Up ]

[ >> ]

[Top]

[Contents]

[Index]

[ ? ]

17.1 `tee`: Redirect output to multiple files or processes

The tee command copies standard input to standard output and also to any files given as arguments. This is useful when you want not only to send some data down a pipe, but also to save a copy. Synopsis:

tee [option]… [file]…

If a file being written to does not already exist, it is created. If a file being written to already exists, the data it previously contained is overwritten unless the `-a' option is used.

A file of `-' causes tee to send another copy of input to standard output, but this is typically not that useful as the copies are interleaved.

The program accepts the following options. Also see Common options.

`-a'
`--append': Append standard input to the given files rather than overwriting them.
`-i'
`--ignore-interrupts': Ignore interrupt signals.

The tee command is useful when you happen to be transferring a large amount of data and also want to summarize that data without reading it a second time. For example, when you are downloading a DVD image, you often want to verify its signature or checksum right away. The inefficient way to do it is simply:

wget http://example.com/some.iso && sha1sum some.iso

One problem with the above is that it makes you wait for the download to complete before starting the time-consuming SHA1 computation. Perhaps even more importantly, the above requires reading the DVD image a second time (the first was from the network).

The efficient way to do it is to interleave the download and SHA1 computation. Then, you'll get the checksum for free, because the entire process parallelizes so well:

# slightly contrived, to demonstrate process substitution
wget -O - http://example.com/dvd.iso \
  | tee >(sha1sum > dvd.sha1) > dvd.iso

That makes tee write not just to the expected output file, but also to a pipe running sha1sum and saving the final checksum in a file named `dvd.sha1'.

Note, however, that this example relies on a feature of modern shells called process substitution (the `>(command)' syntax, above; See (bashref)Process Substitution section `Process Substitution' in The Bash Reference Manual.), so it works with zsh, bash, and ksh, but not with /bin/sh. So if you write code like this in a shell script, be sure to start the script with `#!/bin/bash'.

Since the above example writes to one file and one process, a more conventional and portable use of tee is even better:

wget -O - http://example.com/dvd.iso \
  | tee dvd.iso | sha1sum > dvd.sha1

You can extend this example to make tee write to two processes, computing MD5 and SHA1 checksums in parallel. In this case, process substitution is required:

wget -O - http://example.com/dvd.iso \
  | tee >(sha1sum > dvd.sha1) \
        >(md5sum > dvd.md5) \
  > dvd.iso

This technique is also useful when you want to make a compressed copy of the contents of a pipe. Consider a tool to graphically summarize disk usage data from `du -ak'. For a large hierarchy, `du -ak' can run for a long time, and can easily produce terabytes of data, so you won't want to rerun the command unnecessarily. Nor will you want to save the uncompressed output.

Doing it the inefficient way, you can't even start the GUI until after you've compressed all of the du output:

du -ak | gzip -9 > /tmp/du.gz
gzip -d /tmp/du.gz | xdiskusage -a

With tee and process substitution, you start the GUI right away and eliminate the decompression completely:

du -ak | tee >(gzip -9 > /tmp/du.gz) | xdiskusage -a

Finally, if you regularly create more than one type of compressed tarball at once, for example when make dist creates both gzip-compressed and bzip2-compressed tarballs, there may be a better way. Typical automake-generated `Makefile' rules create the two compressed tar archives with commands in sequence, like this (slightly simplified):

tardir=your-pkg-M.N
tar chof - "$tardir" | gzip  -9 -c > your-pkg-M.N.tar.gz
tar chof - "$tardir" | bzip2 -9 -c > your-pkg-M.N.tar.bz2

However, if the hierarchy you are archiving and compressing is larger than a couple megabytes, and especially if you are using a multi-processor system with plenty of memory, then you can do much better by reading the directory contents only once and running the compression programs in parallel:

tardir=your-pkg-M.N
tar chof - "$tardir" \
  | tee >(gzip -9 -c > your-pkg-M.N.tar.gz) \
  | bzip2 -9 -c > your-pkg-M.N.tar.bz2

An exit status of zero indicates success, and a nonzero value indicates failure.