Unix pipes and filters

We should have some ways of coupling programs like garden hose — screw in another segment when it becomes when it becomes necessary to massage data in another way.
M. D. McIlroy
October 11, 1964
http://doc.cat-v.org/unix/pipes/

Before the original Unix, programs were generally written as monolith programs, which were huge programs that each completed a certain task. However, with the introduction of Unix and the Unix pipes it meant that the standard output of one program could be piped into the standard input of another program. This allowed monolith programs to be broken down into much smaller programmers. Following the popular DRY programming principle of today, this meant that smaller programmes could be written and then used as a part of the input of another programme. Instead of the code being typed out each time. At the time when Unix was introduced, disk space was much more limited than today, programmers would be concerned around whether the output of one program was too large to be stored before it could be used by another program. The Unix pipeline operates in a way that the output never has to be stored before being used by the next the program.

The pipe operator in Unix is “|” (vertical bar), which enables the standout out of one program to be used by the following program, which is separated by a pipe. A great of this is using the “ls” command, which lists out all the files in a directory.

ls

However, you will see that “ls” will take you straight to the bottom of the list of your files and you are unable to navigate up and down the list using your keyboard. We can change this command to include “ | less”.

ls | less

Using the pipe operator, we have told our bash system to take the output of ls and call the command less on it. Less, provides you with a scroll list of content that you can use your arrow keys to navigate up and down.

Using the arrows key to scroll up and down the list

You can then use “q” to exit this

Filters

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.
M. D. McIlroy — A Quarter Century of Unix https://homepage.cs.uri.edu/~thenry/resources/unix_art/ch01s06.html

Filters is a term used to refer to code that has used the pipelines to perform complex operations on several commands. Filters take in an input, change it in some way and then output that.

You could see this if we added a sort command to the previous code:

ls | sort | less

The sort method takes in our list of files, sorts them into alphabetical order and outputs this to our less command which will provided a page-by-page view of the files.

We could also expand this code using the uniq command.

ls | sort | uniq| less

The sort method takes in our list of files, sorts them into alphabetical order and outputs this to our unique command which removes any duplicates from the list and outputs this to the less command which will provided a page-by-page view of the files.

I hope this has offered a insight into Unix pipes and filters. Please let me know if you have any questions.