Streams in the command line
I’m always amazed when I come across tiny apps that can be used in powerful ways. Linux has a number of single purpose apps, created over 20 years ago which developers today continue to use and repurpose today in new and interesting ways. The magic of these apps is that precisely because they have one function, developers are able to combine them to carry out surprisingly complex tasks.
These tiny apps are like single Lego bricks joined together using pipes. As they all work with text, they take input text, transform it, and then output it as text again. Once I mastered building with these tiny apps, I felt a lot like a Master Builder from the Lego movie. Building tiny efficient apps to the dulcet tune of “everything is awesome!”.
This post will be a whistle stop tour exploring a small selection of these tiny apps in action, to show how their seemingly simple functions can be put to use in powerful ways to save you time as you code. Please note, the context scenarios are heavily based on a ridiculous Katy Perry premise. If you don’t enjoy Katy Cat references, then please read at your own risk!
I’m Katy Perry’s producer, we’re about to launch her next album to the public. We want it to be a success, but we’re a bit worried about it being leaked to the public before we launch. Katy asks me to double check a few things:
- Firstly, she wants a list of all the tracks we recorded from her “A-Side” directory in the “KatyPerryAlbum” directory.
- Secondly, ensure only I (her producer) am able to make changes to those tracks (we don’t want amateur producers trying their hand at this).
- Finally, as we’ve not yet launched the album she wants me to make sure that at this stage no other users can read the files.
Let’s tackle each one step by step:
- I can use a couple of commands here to check the number of tracks. Firstly, I’ll change directory, cd, into our KatyPerryAlbum directory. Then I will list ls the number of files in the repository: ls
2. On to the next request… I need to check the permissions of those tracks now that I’ve found them. Easy peasy, let’s list the hidden files using: ls -l h. Now I can see the permissions. Checking the ‘User’ characters (the characters on the left) I can see that only I, her producer, can: ‘r’ (read), ‘w’ (write) files.
3. To finish up the task though, I can see that the next column of lines- the group and all- can’t write or execute the files but they can read them. So I’ll need to change permissions for all other users (groups and others) to ensure that they can’t read the files ahead of the album launch using the asterisks wildcard that allows me to change permissions on all of the .mp3 files: chmod go-r *.mp3
As we get closer to the album launch, Katy asks me to tidy up the directory so we don’t accidentally release the wrong tracks. She sets me two more tasks:
- Count how many files we have in the “A-Side” album directory.
- Move the “B-Side” files from the “A-Side” album and into the main KatyPerryAlbum directory using the same title.
- At last I get to pipe two apps together! I cd into the “A-side” directory, then list the directory, and pipe the result to the right which then counts the number of lines and prints that number: ls . | wc — l
However, I know I have the “B-Side” directory is also in this directory I include grep in another pipe after ls to ensure I only count the mp3 files, using: ls | grep mp3 | wc -l
2. I then move this file to the main album directory: mv B-Side ../ so this is out of my “A-Side” repository as requested.
There’s still a few more tasks to complete in advance of the album launch. Katy asks me to:
- Search through the Artwork directory to find all of the .txt files because they contain the text that needs to be sent for the artwork. Then add these as a file called ‘foundartwork’ inside the same directory.
- Do another search of the Artwork files to find every mention of the word ‘firework’ and ‘T.G.I.F’. Then create a new file that puts all these lines together called ‘vintage’. As Katy plans to use this as a bonus text for the album’s artwork to please her old fans.
- Search through the entire KatyPerryAlbum directory to find any mentions of ‘Taylor Swift’.Then send add these to a file called bad_blood.txt, including 2 lines above and below every ‘Taylor Swift’ mention to give her some context.
Although the last request strikes me as a bit odd I say nothing and do as Katy asks:
- First I find all the files which end with the name “*.txt” inside the Artwork directory. Using find instead of ls will mean that I can search through all the directories, even if there were nested directories. I then pipe the result into a “foundartwork” file: find Artwork -name “*.txt” > foundartwork
2. Next I use grep to search all the text files ‘firework’ and then ‘T.G.I.F’ and pipe both of these outputs into a new file called ‘vintage’: grep ‘firework’ *.txt > vintage
After I do this, I realise I could actually use egrep in a more intelligent way to search two keywords at the same time- so I use the following instead: egrep “firework|T.G.I.F” *.txt > vintage
3. I use find again to do a search across all of the ”.txt” files in the directories inside the album directory, and grep to search for ‘Taylor Swift’ and ‘Bad Blood’ I then use a grep flag grep -C 2 to give 2 lines above and below and then pipe both of these into the new file called bad_blood and then check the file using cat:
grep -C 2 “Taylor Swift” *.txt > bad_blood.txt
4. Finally Katy didn’t ask me to do this but because I’d like there not to be a feud I append some nice words to this file:
echo “don’t be haters, even if haters guna hate, hate, hate” >> bad_blood
I then look at the last line using tail: tail -1 bad_blood
There you have it! With the help of a few tiny tools, plugged together to do our bidding, we managed to get Katy’s album sorted and out of the door. There are many more such tools out there, ready to be connected, exchanged, piped, and redirected. If you are interested in learning more, run pretty much any command with -h to see its help (and discover more useful flags!) or run man toolname to see its manual page.