Administering Networking OS — Pipes, Redirection, and REGEX #8

College Online Material

How-to-Geek

CLI and Redirection

CLI Pipes

• The pipe char. (|) can be used between two commands to send the output of the first as input to the second

$ ls | head

The output of ls is sent to head as input

• Multiple commands can be combined to form pipelines

$ ls /etc/passwd | nl | head

I/O Redirection

Three I/O streams associated with every command:

• Standard Input (STDIN) is normally provided by the user via the keyboard
• Standard Output (STDOUT) is the output produced by the command when operating correctly
• Standard Error (STDERR) is the output produced by the command when an error has occurred

Summary of redirection possible with the bash shell

< /path/to/file STDIN from file

> /path/to/file STDOUT overwriting file

>> /path/to/file STDOUT appending file

2> /path/to/file STDERR overwriting file

2>> /path/to/file STDERR appending file

&> /path/to/file STDERR and STDOUT overwriting file

&>> /path/to/file STDERR and STDOUT appending file

The Null Device

• The null device is represented by the /dev/null file (bit bucket)
• Any output redirected to /dev/null is discard
• /dev/null can be used for input to provide a stream of null values

STDIN, STDOUT, and STDERR

STDIN or 0

STDIN normally is provided by the keyboard but can be redirected with the < symbol

The tr command reads its data from STDIN. It translates from one set of char. to another

$ tr 'a-z' 'A-Z'
alpha
ALPHA

tr command translate from lowercase to uppercase. To end the session press CTRL+D

also tr command can be redirected from the /etc/hosts file

STDOUT or 1

STDOUT is the output from the command when operating correctly

The echo command is used to print messages to STDOUT

$ echo Linux 1
Linux 1

Then, the output can redirect to file like .txt

$ echo Linux 1 > a.txt
$ cat a.txt
Linux 1

Appending STDOUT Redirection

Use single arrow > for STDOUT redirection will clober or overwrite the specific file

Using the double arrow >> for STDOUT redirection will either create a new file or append an existing one

STDERR or 2

STDERR is the output of command after an error has occurred

ls /fake is command that will cause an error to be output to STDERR because the /fake file doesn't exist

Redirecting STDERR is possible. ls /fake 2> /tmp/err.msg is a command that would cause an error to be sent to STDERR which is then redirected to the /tmp/err.msg file

The following example demonstrates the find command searching recursively the /etc/pki dir. for any file matching “*.pem”

Redirecting Multiple Streams Separately

STDERR output is sent to the cert.err file and the STDOUT output is sent to the crt.txt file

STDOUT and STERR are redirected into the same file, crt.all

Find Command

The find command is a powerful tool to be able to search for files in different ways including:

• Name
• Size
• Date
• Ownership

$ find [start_dir] [search_option] [criteria] [result]

If the starting dir. start_dir is not specified, then the current dir. is assumed

The search option search_option is how the search will be done. Use the -name option to search by name

The search criteria criteria is the data to be used with the search option. So, if the search option was -name, then the search criteria would be the name of the file to find

The result option result default to -print, which will output the names of the files that are found

$ find /etc/ssh -name "*.crt"

The option -ls will create output similar to the ls -l command

Searching By File Size

• The -size option can be used by find to search by its size
• Large units can be specified as K, M, G, etc.
• +1M means more than one megabyte
• -1M means less than one megabyte

Less Command

less command is a pager command designed to display one one page of data at a time

more command is another pager command that has less features than less command

Less Searching Commands

• / to search from cursor to end file
• ? to search from cursor to beginning of file
• type pattern to search and press enter
• If more than one match found, press n to go to next match or N to go to previous match

Head or Tail

Filtering With Head

head command display the first ten lines of a files by default. -n option allows for the number of line to be displayed to be specified

Filtering With Tail

tail command siplays the last ten lines of file by default. -n option allow for the number of lines to be displayed to be specified.

Sort Command

Sorting Files or Input

sort command will rearrange its output line according to one or more fields you specify for sorting.

• Fields are separated by whitespace, although with the -t option, you can specify the delimiter.
• The default sort is in ascending order, but you can use the -r option to reverse the sorting of a field.
• The default sort is a dictionary sort, but you can use -n option to make it numeric sort.

Example

File Statistic

The wc command output up to three statistic for each file

By default, wc display the line, words, and bytes contained in each file

To view individual statistic, specify -l for lines, -w for words, or -c for bytes

Example

wc command is often used with pipes so that the output of a command can be anayzed

wc -l as the final command in the pipe will count how many lines of output was produced

Cut Command

Filtering with Cut Command

cut command provides two simple techniques

• -d option can let you specify other delimiters
• -f is used to indicate which fields to extract
• -c option with the range of the column to extract

Example

/etc/passwd file is delimited by colon with these fields

account:password:UID:GID:GECOS:directory:shell

To extract the first and gift through seventh fields

$ cut -d: -f1,5-7 /etc/passwd

Grep Command

Filtering with grep

grep command can be used to filter standard input or the contents of a file for lines matching a specified pattern.

Common Grep Options

Basic Regular Expression (BRE)

Basic Regular Expressions (BRE) are able to be used with the grep command without requiring an option to use them (unlike Extended Regular Expression show later).

The backslash \ can be used to escape the meaning of regular expression metacharacters, including the backslash itself.

Example

• The . (period) char. matches exactly one char.

$ echo 'abcdddadas' | grep 'a..' --color
abcdddadas

• The [] (brackets) char. are used to match exactly one char. If the first char. listed is ^ (caret), then it means not the char brackets.

$ echo 'abcdddadas' | grep '[ab][a-d][^x-z]' --color
abcdddadas

• The * (asterisk) char will match zero or more of the previous char. Matching “a*” is not very useful because it might match zero a’s (matches every line). But, matching “abcd*” would be more useful, since you would need an “abc” followed by zero or more d’s

$ echo 'abcddd' | grep 'abcd*'
abcddd

• The ^ (caret) char, when appearing at the beginner of the pattern

$ cat /etc/passwd | grep 'root'
$ cat /etc/passwd | grep '^root'

• The $ (dollar sign), when appearing at the end of the pattern

$ cat /etc/group | grep 'adm'
$ cat /etc/group | grep 'adm$'

• Combining both ^ and $ char. allows for two special matches.

$ cat ~/.bashrc | grep -v '^$'

‘^$’ is a blank line match.

‘^pattern$' matches if the whole line contains only the “pattern“.

Extended REGEX (ERE)

The use of ERE requires the -E option when using the grep command. ERE can combined with BRE. The following ERE char. ? + and |

• The + char will match one or more of the previous char.

• The ? char will optionally match one of the previous char.

• The | char will act like an “or” operator between two regex