Bash Basics: xargs by example

UNIX utilities tend to follow the philosophy that a program should do one thing and do it well. To accomplish sophisticated tasks on the command line, we need to glue these “small, sharp tools” together — commonly through pipelines. Today we’ll take a look at one small but indispensable variety of UNIX glue: xargs.

I Am A Terrible Programmer

Let’s say, in the course of development, we try to start a Rails server on port 3000 and an EmberJS server on port 4200, only to receive an error. Something like the following:

Address already in use - bind(2) for 0.0.0.0:3000 (Errno::EADDRINUSE)

Uh oh. Looks like some other running process has already bound (gained exclusive access) to our desired ports. How can we discover the identities of these transgressors?

Using lsof with the -i flag, we can search for the PIDs of processes by specifying a range or list of ports (the -t flag limits the output to just process IDs, or PIDs).

> lsof -i :4200,3000 -t 
2497
2682
2684

If we wanted to stop all three of those processes, we would have to use the kill program with each PID. A good programmer would probably investigate why these processes are running and terminate them gracefully. However, I am a terrible programmer, and generally stop processes with little regard for their feelings. We can use the -9 flag to stop processes without asking for their permission.

\\ rudely stops each process 
> kill -9 2497 2682 2684

Imagine if there were 10 or 20 processes. Transcribing the PIDs becomes a time intensive exercise. Fortunately there is xargs.

xargs takes a delimited input from standard in and passes the split sections as arguments to another utility. In the case of lsof, we have newline delimited list of PIDs, and we want to pass all the PIDs to the kill utility.

We can do something like this:

> lsof -i :4200,3000 -t | xargs kill

xargs is simple, but with many useful applications.

Reading Between The Lines

Each night before I go to sleep, I light a candle and transcribe a Drake song. Here’s my progress so far. I’ve transcribed two songs into “.txt” files:

> ls
./01 Legend.mp3
./02 Energy.mp3
./02 Energy.txt
./03 10 Bands.mp3
./04 Know Yourself.mp3
./05 No Tellin'.mp3
./06 Madonna.mp3
./06 Madonna.txt
./07 6 God.mp3
./08 Star67.mp3

One day I get this idea — Drake is sending me a message. If I just take the first verse from each song…Drizzy will bless me with secret illuminati knowledge.

First, I’ll need to isolate just the “.txt” lyrics files

> find . -name ".txt"
./02 Energy.txt
./06 Madonna.txt

Now we need to print the first line of each .txt file. To manually print the first line of a file, we can use head

> head -n 1 Energy.txt
Lickwood means "rewind" and gunshot means "forward"
> head -n 1 Madonna.txt
Breathe through, ride through

But how do we apply the head command to each line output by find?

> find . -name ".txt" | xargs head -n 1
==> ./energy.txt <==
Lickwood means "rewind" and gunshot means "forward"
==> ./madonna.txt <==
Breathe through, ride through

Cool. We have saved ourselves some time, but we’re getting these headers which contain the filename. It makes it harder to decrypt our Drake lyrics. The problem is that head is getting passed each .txt file as a series of arguments

head -n 1 energy.txt madonna.txt

instead of,

head -n 1 energy.txt
head -n 1 madonna.txt

It’s cool, xargs has our back. We can use the -n flag to call the utility with a max of -n arguments. Our arguments will now be fed into the head utility in chunks of size n.

> find . -name ".txt" | xargs -n 1 head -n 1
Lickwood means "rewind" and gunshot means "forward"
Breathe through, ride through

Dealing With Unruly Input

Let’s say I have a good night one night and successfully transcribe the rest of the songs, excitedly I run my bash script to decipher the message.

> find . -name ".txt" | xargs -n 1 head -n 1
xargs: unterminated quote

G dang it! One of our file names contains special characters, specifically the song called No Tellin’. The apostrophe is seen by xargs, which waits for a closing quote but never gets one.

Luckily, find and xargs were built to be used together. We can use -print0 and the -0 flag to change xargs to expect NUL (``\0'’) characters as separators, instead of spaces and newlines.

> find . -name ".txt" -print0 | xargs -0 -n 1 head -n 1

This inputs the entire line as a string and thus ignores the single quote and prints out what we’d expect.

Moving On

After finally decoding my message from Drake, I realize the message I decoded was mostly nonsense. Frustrated, I decide to move these files into a folder where I can review them later once I’ve calmed down.

We can use the mv utility to accomplish this, however the mv utility needs the file name twice. We can use the -I operator to pass xargs an argument list. This gives us access to the argument being passed in so we can use it in multiple places.

> mkdir review_later
> ls
./01 Legend.mp3
./02 Energy.mp3
./02 Energy.txt
./03 10 Bands.mp3
...
review_later
> find . -name "*.txt" -print0 | xargs -0 -I {} mv {}              > ./review_later/{}

Great, now all our .txt files are in a different folder. The {} is just a placeholder for whatever arguments are being fed into xargs, and you can actually use whatever string you want. A more semantic version of our previous command would look like this:

> find . -name "*.txt" -print0 | xargs -0 -I song_lyric mv          > song_lyric ./review_later/song_lyric

These are just a few of the applications of xargs. For some more examples check out this guide. For a list of all possible options and flags check out the xargs man page. For more information about Drake check out this twitter account. Happy scripting!