3 Programs that will Level Up your Bash Game
Hi again! Glad you’re back, it’s good to see you! We were talking about neat Python libraries last time. This time, we’ll talk about some programs you can use in your shell to become a bash powerhouse. Or maybe like, a bash… hamster…wheel. Either way, by the end of this I’m hoping that you’ll feel at least somewhat more productive.
As an aside, I’m trying to write articles that are good, not just junk to get more views to fund my cocaine-and-vintage-computers habit. My articles are never pay-walled, because I want you ( yes, you! reading this! ) to enjoy yourself and learn something. If you think it’s junk, leave a comment! I’ll do better. Remember, I’m not in this for the big bucks, I’m writing for free, so I have no excuse for bad articles.
Anyway, here we go. So, I’m not gonna talk about like, sed or grep. I assume in this article that those are already in your repertoire. That’s right, folks. We’re off-roading here. As usual, the level of awesome / confusion will increase as the article goes along. I’ve saved the best for last.
rsync
rsync is super cool, and it’s slowly ( but surely ) becoming standard, taking the place of the long-lived scp. It takes a while to move mountains as big as what’s already in place, though; in the mean time, it’s best to get acquainted!
What is it?
rsync is a program that helps you copy files. It’s a bit smarter than cp, because it can detect only things that really need copied by default if you’re doing something like a backup. It’s also a bit safer than scp, which is slowly being deprecated. It works both locally and remotely, which is super cool.
Where do I get it?
Oh, that depends. If you’re on Linux, probably run apt install rsync or yum install rsync or pacman -S rsync depending on your distribution. If you’re on MacOS, you can probably run brew install rsync — in either case, I’d be surprised if you didn’t already have it. If you’re on Windows? It’s uh, a little complicated, but there are guides out there. One approach recommends using WSL, which is Linux, just… under Windows.
How do I use it?
Easy! First we’ll look at the local case. Suppose I have two directories, source-directory and destination-directory and that source-directory contains random junk I want to copy into destination-directory , like so:
Let’s rsync the contents of source-directory into destination-directory! All we need to do is say rsync -r source-directory/ destination-directory/:
That’s it! Just like cp -r , but a lot faster. Don’t forget your trailing / on your directory name!
What, not impressed? Part of the power of rsync is that it syncs. That is, with the right options, you’re only copying what you need to. So, if I add more files to my source-directory and run a new sync ( with verbosity on ), I can demonstrate. Let’s run the command
rsync -a -v --stats source-directory/ destination-directory/:
See that? We only sent the five new files that we added, not the first 10 again. This is great for backing up data. The option that helps us in this case is that -a! Using the -a flag will help us keep directories in sync without doing too much work.
What?? Still not impressed? Come on, I’m really trying here! OK, how about the fact that it can be used remotely? OK, situation. I’ve got a sweet Windows96 album on my plex server, and I want to sync it to my machine. We should get the address of the machine — we can see that for me, it’s 192.168.50.202.
Don’t use SCP — just rsync! The rsync command we want in this instance is
rsync -r 192.168.50.202:/path/to/the/files/on/remote/server .:
It’s just that easy: that synced all of the files on the remote server to my current working directory. The flag -r made sure we recursed into the directory on the server. That’s it!
jq
I did not like jq at first. At all. I didn’t like the way it worked, I was confused by its syntax, and I was put off by the colorful highlighting. I was wrong. A caveman, revolting at the sight of the first bow. jq is awesome.
What is it?
jq is a program that helps you manipulate JSON objects from the terminal. If you work with production code, you’re probably seeing JSON rather often. Getting good at it will absolutely make you more productive.
Where do I get it?
You can download jq from its homepage here! It’s available for Linux, MacOS, and Windows, all 64 bit. If you’re on something else, or you’ve got a 32 bit system, you can build it from source as well ( though that’s outside of the scope of this article ). You might also just apt install jq or brew install jq .
How do I use it?
This is super simple. I’m going to go ahead and grab the second JSON example from the official JSON website ( ? weird ? ), and just paste it into a file. It looks a whole lot like this:
cat thejson.json
{"menu": {
"id": "file",
"value": "File",
"popup": {
"menuitem": [
{"value": "New", "onclick": "CreateNewDoc()"},
{"value": "Open", "onclick": "OpenDoc()"},
{"value": "Close", "onclick": "CloseDoc()"}
]
}
}}Pretty standard, right? Let’s dig in with jq . You can either use cat <file> | jq . or call jq directly on the file. This earns you some nice syntax highlighting:
We have to go deeper. Let’s dig down and get the ‘value’ field from the ‘menu’ key.
This will suffice:
cat thejson.json | jq '.menu.value'
Simple enough! See how we use a . for each level of the document we care about? Very neat.
Now, that document also has an array. See it down there at menuitem underneath the popup key? How do we handle those? With the array operator:
cat thejson.json | jq '.menu.popup.menuitem | .[]'
So, let’s break that down. Like before, we selected down into the JSON object with .s. This time, though, we’re piping inside the jq command. We get down to menuitem who we know is an array, then we pipe it over to .[] so that it spits out three separate entries for us. Now, let’s do some jq magic. We can get all of the values out of these, like so:
cat thejson.json | jq '.menu.popup.menuitem | .[].value'
Cool, right? It was that easy to dig right down in. Now, there are a lot of really cool things you can do with jq even outside the awesomeness we looked at here ( here’s an awesome post about select , which is super powerful ), but for the sake of brevity, we’ll leave it at that. The value of jq is just… too much to handle.
GNU Parallel
Oh man. This one is by far the coolest. It might be one of the coolest programs I use. It’s not something to use on a super regular basis? I suppose? But when you need it, you’ll know, and knowing how to use it in that pinch is just super cool.
What is it?
GNU Parallel is a program that lets you do true parallelization… from your terminal. Yeah, I wasn’t kidding, this is sweet.
Where do I get it?
You can apt install parallel or brew install parallel and maybe if you’re lucky you can pacman -S parallel. Alternatively, you can just go grab it from the GNU Website.
How do I use it?
So I’m going to cover the most basic use case. This program has an absolutely amazing manual, as well as a killer tutorial, and if you want to do any advanced usage, you should definitely take a look.
But here we go.
SITUATION: You find yourself with a file containing a long list of URLs, each of which points to something you want to download. What will you do?
“Ahh, easy,” you say, “I’ll just write a for loop. for file in $(cat download_links.t — “ HALT, you have VIOLATED THE LAW. No, we do not need to do that. In fact, we don’t want to — that queues us up for a sequential download. If we want to get our files downloaded before next week sometime, we should do as many downloads at once as we can, right? GNU Parallel makes this simple. Behold:
Our command ends up being
parallel wget -q {} :::: download_links.txt — let’s break that down.
The {} is the thing we’re going to be wgeting, and it acts as a placeholder for each of the lines in download_links.txt. The :::: tells GNU Parallel that we should read from a file. And that’s all!
You really can’t beat that. It’s super fast, because that’s what it’s for — making things faster. This is just one use for GNU Parallel. There are more uses than you can probably count on all four hands. Really — go explore GNU Parallel!
Conclusion
Here we are, you and I, at the end of another incredible article. We’ve talked about performance techniques, libraries, and the rain together, so far. Well. I’ve talked at you, but you’ve enjoyed it, right? Right. I hope you picked up something interesting this time. I personally use jq almost every day, so hopefully that’ll be a good one to stick in your pocket.
I’m glad you came for this chat. It means a lot to me, you know? Bash and shell scripting have a special place in my heart. Is that weird to say? Wait, where are you going? Why do you do this every time? I wasn’t done talking. Did I make it weird or something? Hey. Hey!
Citations
GNU Parallel: O. Tange (2011): GNU Parallel — The Command-Line Power Tool, ;login: The USENIX Magazine, February 2011:42–47.
