Chapter 03 · Pipelines, Redirection, and Environment Variables

Chapter Overview

So far we have learned more than fifty commonly used Linux commands. But unless you can combine them flexibly, you won’t truly boost your efficiency. This chapter begins with five flavors of I/O redirection used in file read/write workflows: standard overwrite output redirection, standard append output redirection, error overwrite output redirection, error append output redirection, and input redirection. Through hands-on demos you will see exactly what each mode does and how to persist output cleanly.

We then dive into the pipeline operator, which lets you chain commands and process output more efficiently. Next we cover shell wildcards and the most useful escape characters so your commands say precisely what you intend—laying the groundwork for writing shell scripts in the next chapter.

Finally, we look under the hood at how the Bash interpreter executes commands. You will understand the PATH variable and other key environment variables that shape the Linux runtime environment.

3.1 Input and Output Redirection

Now that Chapter 2 has armed you with most of the foundational commands, your next task is to compose several commands so they cooperate and handle data efficiently. To do this well, you must understand input and output redirection.

In short, input redirection feeds file content into a command, while output redirection takes data that would otherwise go to the screen and writes it to a file. The data flow is shown in Figure 3-1. In daily work, output redirection is more common than input redirection, and it comes in two streams (standard output and standard error) and two modes (overwrite and append).

Standard input redirection (STDIN, file descriptor 0): reads from the keyboard by default, or from a file or another command.
Standard output redirection (STDOUT, file descriptor 1): writes to the screen by default.
Error output redirection (STDERR, file descriptor 2): also writes to the screen by default.

Figure 3-1 Input/output redirection

Consider examining two files—one exists, one does not. Both commands print something to the screen, but the outputs are different in nature:

root@linuxprobe:~# touch linuxprobe
root@linuxprobe:~# ls -l linuxprobe 
-rw-r--r--. 1 root root 0 May 19 10:31 linuxprobe
root@linuxprobe:~# ls -l xxxxxx
ls: cannot access 'xxxxxx': No such file or directory

The output for linuxprobe is regular information about a real file—permissions, owner, group, size, timestamp. That is standard output. The message for xxxxxx reports an error because the file does not exist—that is standard error. If you want to write what would have gone to the screen into a file, you must distinguish these streams.

Table 3-1 lists input-redirection operators.

Table 3-1 Input-redirection operators

Operator	Meaning
command < file	Use file as command’s standard input
command << delimiter	Read from standard input until the given delimiter appears
command < f1 > f2	Read f1 as standard input and redirect standard output to f2

Table 3-2 lists output-redirection operators.

Table 3-2 Output-redirection operators

Operator	Meaning
command > file	Redirect standard output to file (overwrite existing content)
command 2> file	Redirect error output to file (overwrite existing content)
command >> file	Redirect standard output to file (append to existing content)
command 2>> file	Redirect error output to file (append to existing content)
command >> file 2>&1 or command &>> file	Redirect both standard and error output to file (append)

For standard output you may omit the file descriptor 1, but for error output you must specify 2. Let’s try a simple exercise: redirect the output of man poweroff into readme.txt, then display that file.

root@linuxprobe:~# man poweroff > readme.txt
root@linuxprobe:~# cat readme.txt 
POWEROFF(8)                        poweroff                        POWEROFF(8)
NAME
       poweroff, reboot, halt - Power off, reboot, or halt the machine
SYNOPSIS
       poweroff [OPTIONS...]

       reboot [OPTIONS...]

       halt [OPTIONS...]
DESCRIPTION
       poweroff, reboot, and halt may be used to power off, reboot, or halt
       the machine. All three commands take the same options.
… output truncated …

Convenient, right? Next, compare overwrite versus append. First write multiple lines using overwrite mode (each write replaces the previous content, leaving only the last line):

root@linuxprobe:~# echo "Welcome to LinuxProbe.Com" > readme.txt
root@linuxprobe:~# echo "Welcome to LinuxProbe.Com" > readme.txt
root@linuxprobe:~# echo "Welcome to LinuxProbe.Com" > readme.txt
root@linuxprobe:~# echo "Welcome to LinuxProbe.Com" > readme.txt
root@linuxprobe:~# echo "Welcome to LinuxProbe.Com" > readme.txt
root@linuxprobe:~# cat readme.txt
Welcome to LinuxProbe.Com

Then append one more line:

root@linuxprobe:~# echo "Quality linux learning materials" >> readme.txt
root@linuxprobe:~# cat readme.txt
Welcome to LinuxProbe.Com
Quality linux learning materials

Standard and error outputs behave differently. With an existing file, standard output will redirect to the file but error redirection still prints to the screen (because there is no error):

root@linuxprobe:~# ls -l linuxprobe > /root/stderr.txt 
root@linuxprobe:~# ls -l linuxprobe 2> /root/stderr.txt 
-rw-r--r--. 1 root root 0 May 19 10:48 linuxprobe

To capture errors to a file—useful in automated shell scripts for later troubleshooting—try with a non-existent file:

root@linuxprobe:~# ls -l xxxxxx > /root/stderr.txt
ls: cannot access 'xxxxxx': No such file or directory
root@linuxprobe:~# ls -l xxxxxx 2> /root/stderr.txt
root@linuxprobe:~# cat /root/stderr.txt 
ls: cannot access 'xxxxxx': No such file or directory

If you don’t want to distinguish the streams and just want everything appended to one file, use &>>:

root@linuxprobe:~# ls -l linuxprobe &>> result.txt
root@linuxprobe:~# ls -l xxxxxx &>> result.txt
root@linuxprobe:~# cat result.txt
-rw-r--r--. 1 root root 0 May 19 10:48 linuxprobe
ls: cannot access 'xxxxxx': No such file or directory

Input redirection is less common in daily work, but handy when needed. It feeds a file’s contents directly into a command. For example, count the number of lines in readme.txt:

root@linuxprobe:~# wc -l < readme.txt
2

Notice how this differs from Chapter 2’s example, which included the file name:

root@linuxprobe:~# wc -l /etc/passwd
38 /etc/passwd

In the latter form, wc was given a file object explicitly, so it printed the count and the file name. With wc -l < readme.txt, the contents are piped into standard input; wc sees only a byte stream, not the file name. Keep this subtle difference in mind.

3.2 The Pipeline Operator

In Section 2.6 we saw the pipeline operator. Type it by pressing Shift together with the backslash key (). The form is command A | command B. In one sentence: the pipeline takes the output of the command on the left and supplies it as standard input to the command on the right (Figure 3-2).

Figure 3-2 Pipeline concept

In Section 2.6 we used grep to find users whose login shell is /sbin/nologin. We also used wc -l to count lines. You can combine them into a single command by piping grep’s output into wc:

root@linuxprobe:~# grep /sbin/nologin /etc/passwd | wc -l
33

Pipelines make many tasks easier. For example, page through long output when listing /etc so the screen doesn’t scroll by too fast:

root@linuxprobe:~# ls -l /etc/ | more
total 1316
-rw-r--r--.  1 root                 root                     16 Mar  8 20:06 adjtime
-rw-r--r--.  1 root                 root                   1529 Nov 29  2023 aliases
drwxr-xr-x.  3 root                 root                     65 Mar  8 20:04 alsa
drwxr-xr-x.  2 root                 root                   4096 Mar  8 20:04 alternatives
-rw-r--r--.  1 root                 root                    541 Jun 24  2024 anacrontab
-rw-r--r--.  1 root                 root                     55 Jun 17  2024 asound.conf
-rw-r--r--.  1 root                 root                      1 Jun 24  2024 at.deny
drwxr-x---.  4 root                 root                    100 Mar  8 20:02 audit
--More--

When resetting a user’s password, most tools ask you to enter the new value twice, which breaks automation. Combine echo with passwd’s --stdin parameter to set a password in one shot:

root@linuxprobe:~# echo "linuxprobe" | passwd --stdin root

When you first tried ps in Chapter 2, ps aux spewed lines too fast to read. Combine ps with grep to pinpoint the processes you care about, such as those related to bash:

root@linuxprobe:~# ps aux | grep bash
root        3272  0.1  0.1 230020  5552 pts/0    Ss   10:30   0:02 /usr/bin/bash
root        3953  0.0  0.0 227680  1948 pts/0    S+   10:58   0:00 grep --color=auto bash

You are not limited to a single pipeline stage. Chaining like command A | command B | command C is common. To help students remember, I sometimes liken the pipeline to Doraemon’s “Anywhere Door”: data passes instantly from one stage to the next so work flows smoothly.

Tips:
A student once offered a vivid analogy: a street barbecue stand. One person cuts the meat, the next threads it onto skewers, the next grills it. Each step’s output becomes the next step’s input, just like a pipeline, and the final result is handed to the customer in one go.

If you want to display pipeline output on the screen and write it to a file at the same time, add tee:

root@linuxprobe:~# ps aux | grep bash | tee result.txt
root        3272  0.1  0.1 230020  5552 pts/0    Ss   10:30   0:02 /usr/bin/bash
root        3961  0.0  0.0 227680  1884 pts/0    S+   10:58   0:00 grep --color=auto bash
root        3962  0.0  0.0 230020  3300 pts/0    D+   10:58   0:00 /usr/bin/bash
root@linuxprobe:~# cat result.txt
root        3272  0.1  0.1 230020  5552 pts/0    Ss   10:30   0:02 /usr/bin/bash
root        3961  0.0  0.0 227680  1884 pts/0    S+   10:58   0:00 grep --color=auto bash
root        3962  0.0  0.0 230020  3300 pts/0    D+   10:58   0:00 /usr/bin/bash

3.3 Wildcards on the Command Line

We have all drawn a blank on a character while writing. As Linux administrators we sometimes have the same feeling about file names—“it’s on the tip of my tongue.” Suppose you remember only the first few letters of a file name and want to list all files that start with those letters. Or suppose you want to view the attributes of all disk device files at once. You could run these commands one by one:

root@linuxprobe:~# ls -l /dev/sda
brw-rw----. 1 root disk 8, 0 May 19 10:22 /dev/sda
root@linuxprobe:~# ls -l /dev/sda1
brw-rw----. 1 root disk 8, 1 May 19 10:22 /dev/sda1
root@linuxprobe:~# ls -l /dev/sda2
brw-rw----. 1 root disk 8, 2 May 19 10:22 /dev/sda2
root@linuxprobe:~# ls -l /dev/sda3
brw-rw----. 1 root disk 8, 3 May 19 10:22 /dev/sda3
root@linuxprobe:~# ls -l /dev/sda4
ls: cannot access '/dev/sda4': No such file or directory

That’s fine for four items, but hundreds would take all day. Fortunately these devices share a pattern: the files live under /dev and start with sda. Even without knowing the exact partition numbers, you can use wildcards.

As the name suggests, wildcards are symbols that match text. The asterisk (*) matches zero or more characters, the question mark (?) matches exactly one character, [0-9] matches any single digit, and [abc] matches any one of a, b, or c. Table 3-3 lists common shell wildcards.

Table 3-3 Common shell wildcards

Wildcard	Meaning
*	Any string
?	Any single character
[a-z]	Single lowercase letter
[A-Z]	Single uppercase letter
[a-Z]	Any single letter
[0-9]	Any single digit
[[:alpha:]]	Any letter
[[:upper:]]	Any uppercase letter
[[:lower:]]	Any lowercase letter
[[:digit:]]	Any digit
[[:alnum:]]	Any letter or digit
[[:punct:]]	Any punctuation

Match all files in /dev that start with sda:

root@linuxprobe:~# ls -l /dev/sda*
brw-rw----. 1 root disk 8, 0 May 19 10:22 /dev/sda
brw-rw----. 1 root disk 8, 1 May 19 10:22 /dev/sda1
brw-rw----. 1 root disk 8, 2 May 19 10:22 /dev/sda2
brw-rw----. 1 root disk 8, 3 May 19 10:22 /dev/sda3

Match files that start with sda and have one more character after that:

root@linuxprobe:~# ls -l /dev/sda?
brw-rw----. 1 root disk 8, 1 May 19 10:22 /dev/sda1
brw-rw----. 1 root disk 8, 2 May 19 10:22 /dev/sda2
brw-rw----. 1 root disk 8, 3 May 19 10:22 /dev/sda3

Use [0-9] to match a single digit or a bracket list like [135] to match one of those specific digits only:

root@linuxprobe:~# ls -l /dev/sda[0-9]
brw-rw----. 1 root disk 8, 1 May 19 10:22 /dev/sda1
brw-rw----. 1 root disk 8, 2 May 19 10:22 /dev/sda2
brw-rw----. 1 root disk 8, 3 May 19 10:22 /dev/sda3
root@linuxprobe:~# ls -l /dev/sda[135]
brw-rw----. 1 root disk 8, 1 May 19 10:22 /dev/sda1
brw-rw----. 1 root disk 8, 3 May 19 10:22 /dev/sda3

Wildcards need not appear at the end; they can also be at the beginning. For example, find all configuration files under /etc that end with .conf:

root@linuxprobe:~# ls -l /etc/*.conf
-rw-r--r--. 1 root root    55 Jun 17  2024 /etc/asound.conf
-rw-r--r--. 1 root root 30583 Jul  4  2024 /etc/brltty.conf
-rw-r--r--. 1 root root  1370 Dec  5  2023 /etc/chrony.conf
-rw-r--r--. 1 root root 28602 Jun 24  2024 /etc/dnsmasq.conf
-rw-r--r--. 1 root root   117 Aug 28  2024 /etc/dracut.conf
-rw-r--r--. 1 root root    20 Feb 25  2022 /etc/fprintd.conf
-rw-r--r--. 1 root root    38 Jun 24  2024 /etc/fuse.conf
… output truncated …

Wildcards also combine well with commands that create files. Use braces with comma-separated fields to create several files in one shot:

root@linuxprobe:~# touch {AA,BB,CC}.conf
root@linuxprobe:~# ls -l *.conf
-rw-r--r--. 1 root root 0 May 19 11:01 AA.conf
-rw-r--r--. 1 root root 0 May 19 11:01 BB.conf
-rw-r--r--. 1 root root 0 May 19 11:01 CC.conf

Or generate a sequence of tokens:

root@linuxprobe:~# echo file{1,2,3,4,5}
file1 file2 file3 file4 file5

3.4 Frequently Used Escape Characters

To better interpret what you type, the shell provides escape characters for handling special input. Drawing on more than a decade of work and teaching, I distilled dozens down to the four you will use most often.

Backslash (): turns the following special character into a literal character.
Single quotes (' '): treat everything inside as literal text.
Double quotes (" "): preserve variable expansion and most substitutions.
Backticks ( ): run the enclosed command and substitute its output.

First set a variable named PRICE to 5, then print a string with the variable inside double quotes:

root@linuxprobe:~# PRICE=5
root@linuxprobe:~# echo "Price is $PRICE"
Price is 5

Now try to print “Price is $5”. Because $ starts variable expansion, $$ expands to the current process ID, so the output is not what you intended:

root@linuxprobe:~# echo "Price is $$PRICE" 
Price is 3272PRICE

Tips: In a Linux terminal, $$ expands to the current process ID.

To force the first $ to be a dollar sign, escape it with a backslash:

root@linuxprobe:~# echo "Price is \$$PRICE"
Price is $5

If you need just the output of a command, enclose it in backticks. For example, print kernel and OS info with uname -a:

root@linuxprobe:~# echo `uname -a`
Linux linuxprobe.com 6.11.0-0.rc5.23.el10.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Sep 23 04:19:12 EDT 2024 x86_64 GNU/Linux

Backslashes and backticks are distinctive enough that people rarely misuse them. Double quotes cause more confusion, because many commands seem to behave the same with or without them:

root@linuxprobe:~# echo AA BB CC
AA BB CC
root@linuxprobe:~# echo "AA BB CC"
AA BB CC

The difference is ambiguous argument parsing. Without quotes, the program may treat AA, BB, and CC as separate arguments; with quotes, it receives a single argument containing spaces. Here is a simple rule of thumb: if the argument contains spaces, use double quotes; if not, leave them out.

3.5 Important Environment Variables

A variable stores a changeable value. In Linux, variable names are typically uppercase, and commands are lowercase—an informal convention. Environment variables define many aspects of the runtime environment, such as each user’s home directory and mailbox location. You can retrieve a value simply by referencing its variable name.

We will focus on what matters most. Linux uses hundreds of environment variables to run smoothly; you need not learn them all. First, let’s examine how a command is executed in four steps.

Step 1: If you entered an absolute or relative path (such as /bin/ls), the system runs that path directly. Otherwise, proceed to Step 2.

Step 2: The shell checks whether you entered an alias—a custom name that replaces the original command. For example, rm is often aliased to rm -i to guard against accidental deletion:

root@linuxprobe:~# ls
anaconda-ks.cfg  Documents  Music     Public     Videos
Desktop          Downloads  Pictures  Templates
root@linuxprobe:~# rm -r Music
rm: remove directory 'Music'? y

Create an alias with alias name=command. Remove one with unalias name.

Remove the rm alias and try again:

root@linuxprobe:~# unalias rm
root@linuxprobe:~# rm -r Videos
root@linuxprobe:~#

Step 3: Bash determines whether the command is a shell builtin (internal) or an external program. Internal commands execute directly; external commands fall through to Step 4. Use type command to check which kind you are dealing with.

root@linuxprobe:~# type echo
echo is a shell builtin
root@linuxprobe:~# type uptime
uptime is /usr/bin/uptime

Step 4: The system searches several directories for the command file. These directories are set in the PATH variable—the shell’s “helper.” PATH is a colon-separated list of directories. Adding or removing entries changes where the shell searches for commands.

root@linuxprobe:~# echo $PATH
/root/.local/bin:/root/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin
root@linuxprobe:~# PATH=$PATH:/root/bin
root@linuxprobe:~# echo $PATH
/root/.local/bin:/root/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/root/bin

A common question: “Why not add the current directory (.) to PATH?” Technically you can, and it sometimes saves typing. But it creates a security risk. An attacker could drop a Trojan named ls or cd in a shared directory like /tmp; if you happen to run commands there, you could be tricked into executing the Trojan.

As a cautious administrator, always check PATH for suspicious entries before running commands on a new system. Use env to list all environment variables. Table 3-4 highlights ten especially important ones.

Table 3-4 Important environment variables

Variable	Purpose
HOME	User’s home directory
SHELL	User’s shell interpreter
HISTSIZE	Number of history entries shown
HISTFILESIZE	Number of history entries saved
MAIL	Mail storage path
LANG	System language/locale
RANDOM	Generate a random number
PS1	Shell prompt format
PATH	Directories searched for executables
EDITOR	Default text editor

Linux is a multiuser, multitasking OS, and each user can have a tailored environment. The same variable can have different values for different users. For example, HOME differs between root and linuxprobe (su switches users; see Chapter 5):

root@linuxprobe:~# echo $HOME
/root
root@linuxprobe:~# su - linuxprobe
linuxprobe@linuxprobe:~$ echo $HOME
/home/linuxprobe
linuxprobe@linuxprobe:~$ exit

Variables consist of a name and a value; you can create your own to fit your workflow. For example, define WORKDIR to jump quickly into a deep directory:

root@linuxprobe:~# mkdir /home/workdir
root@linuxprobe:~# WORKDIR=/home/workdir
root@linuxprobe:~# cd $WORKDIR 
root@linuxprobe:/home/workdir# pwd
/home/workdir
root@linuxprobe:/home/workdir# cd ~

By default this variable is not global and other users cannot use it:

root@linuxprobe:~# su linuxprobe
linuxprobe@linuxprobe:/root$ cd $WORKDIR
linuxprobe@linuxprobe:~$ echo $WORKDIR

linuxprobe@linuxprobe:~$ exit

Export it to make it available to child shells:

root@linuxprobe:~# export WORKDIR
root@linuxprobe:~# su linuxprobe
linuxprobe@linuxprobe:/root$ cd $WORKDIR
linuxprobe@linuxprobe:/home/workdir$ pwd
/home/workdir
linuxprobe@linuxprobe:/home/workdir$ exit

When you no longer need it, remove it from the current shell with unset:

root@linuxprobe:~# unset WORKDIR
root@linuxprobe:~#

Tips: Variables set directly at the prompt take effect immediately but disappear after a reboot. To make them persistent, add them to .bashrc or .bash_profile. Not sure how to edit files yet? See Chapter 4.

Review Questions

1. What command appends the normal output of ls to error.txt?
   Answer: ls >> error.txt (note the difference between > and >>).

2. If you don’t want to distinguish between normal and error output and want to redirect everything to one file, which operator should you use?
   Answer: Use &> or &>>.

3. Summarize the purpose of the pipeline operator.
   Answer: It feeds the output of the command on the left as the input to the command on the right for further processing.

4. In Bash wildcard syntax, how many characters does the asterisk (*) match?
   Answer: Zero or more.

5. What does the PATH variable do?
   Answer: It defines where the shell searches for commands to execute.

6. In general, what is the advantage of adding double quotes around arguments?
   Answer: Quotes clearly delimit arguments (especially when spaces are present), preventing ambiguity during execution.

7. What command turns a regular variable named LINUX into a global (exported) variable?
   Answer: export LINUX

8. How do you list all current command aliases?
   Answer: alias

9. How do you tell whether a Linux command is internal or external?
   Answer: Use type followed by the command name—for example, type uptime.

10. Besides using unset, what else makes a just-created variable disappear?
    Answer: If it is not written to a config file, closing the terminal or rebooting will make it vanish.