2.4.2. I/O Redirection, Pipelines, and Essential Utilities
💡 First Principle: Every Linux process has three standard streams — stdin (input, fd 0), stdout (output, fd 1), and stderr (errors, fd 2). Redirection operators rewire these streams between files and other processes. Pipes chain programs together so the output of one becomes the input of the next — this is the Unix philosophy made practical.
Redirection Operators:
| Operator | Action | Example |
|---|---|---|
> | Redirect stdout to file (overwrite) | ls > files.txt |
>> | Redirect stdout to file (append) | echo "log" >> app.log |
< | Redirect file to stdin | sort < names.txt |
2> | Redirect stderr to file | cmd 2> errors.log |
2>&1 | Redirect stderr to same destination as stdout | cmd > out.log 2>&1 |
&> | Redirect both stdout and stderr (bash shorthand) | cmd &> all.log |
| | Pipe stdout to next command's stdin | ps aux | grep nginx |
<<EOF | Here-document: multi-line stdin | cat << EOF ... EOF |
<<< | Here-string: single-string stdin | grep "pattern" <<< "$var" |
# Common patterns
command > /dev/null 2>&1 # Discard all output (stdout and stderr)
command 2>&1 | tee output.log # Show output and save to file simultaneously
Essential Text Processing Utilities:
# Filtering and search
grep "error" /var/log/syslog # Find lines matching pattern
grep -i "error" file # Case-insensitive
grep -r "TODO" /project/ # Recursive directory search
grep -v "debug" app.log # Invert match (lines NOT matching)
grep -c "error" file # Count matching lines
# Stream editing
sed 's/foo/bar/g' file # Replace all "foo" with "bar"
sed -n '10,20p' file # Print lines 10–20
sed '/^#/d' config.conf # Delete comment lines
# Column/field processing
awk '{print $1, $3}' file # Print fields 1 and 3
awk -F: '{print $1}' /etc/passwd # Use : as delimiter, print username
awk '$3 > 1000' /etc/passwd # Print lines where field 3 > 1000
# Sorting and deduplication
sort file # Alphabetical sort
sort -n file # Numeric sort
sort -r file # Reverse sort
sort -k2 file # Sort by field 2
uniq file # Remove adjacent duplicates
sort file | uniq -c | sort -rn # Count and rank unique lines
# Extraction and transformation
cut -d: -f1 /etc/passwd # Cut field 1 with : delimiter
cut -c1-10 file # First 10 characters of each line
tr 'a-z' 'A-Z' < file # Translate lowercase to uppercase
tr -d '\r' < windows.txt # Remove carriage returns
# I/O utilities
head -n 20 file # First 20 lines
tail -n 20 file # Last 20 lines
tail -f /var/log/syslog # Follow file as it grows (live logs)
tee output.txt # Write to file AND stdout
wc -l file # Count lines
wc -w file # Count words
xargs rm # Pass stdin lines as arguments
find /tmp -mtime +7 | xargs rm # Delete files older than 7 days
# Miscellaneous
cat file1 file2 # Concatenate files
echo "text" # Print to stdout
printf "%s\n" "formatted" # Formatted print
bc <<< "scale=2; 10/3" # Calculator
Text Editors on the Exam:
The XK0-006 exam tests basic vi/vim and nano proficiency. Minimum vim commands you must know: i (insert mode), Esc (command mode), :w (save), :q (quit), :wq (save and quit), :q! (quit without saving), dd (delete line), yy/p (copy/paste line), /pattern (search).
⚠️ Exam Trap: 2>&1 must come after the stdout redirection — cmd > file 2>&1 redirects stderr to wherever stdout goes (the file). Writing cmd 2>&1 > file redirects stderr to the current stdout (the terminal), then redirects stdout to the file — stderr still goes to the terminal. Order matters.
Reflection Question: You want to find all unique IP addresses in an access log, count how many times each appears, and display the top 10 most frequent. Write the pipeline using only awk, sort, uniq, and head.
