Xargs — What Is It? When To Use It?

Cover image

You may have come across Xargs commands, but what is it for? Today you’ll learn how it works and how to create powerful commands with it.

TL;DR

When piping commands through the | syntax, the next command receives the output of the command on the left side via standard input. This works fine for commands like grep.

Commands like git revert <commit> receives arguments, thus we would need to prefix it with xargs in order to transform those outputs from the previous command into arguments.

What Is Up With That Weird Command?

If you are a beginner Linux user, or simply use Bash terminals by other means — like using Git Bash on Windows — you end up searching things like “How to create multiple directories at once”. There might be many ways to accomplish that, but chances are you came across this line:

echo 'one two three' | xargs mkdir

Maybe you didn’t really understand what it does, you just know that it did the job — it served the purpose you needed by the time. But if you are reading this, I guess you’re just like me and enjoys the art of understanding the tools you use.

If we take a look at Xargs’ manual page, we get the following:

xargs — build and execute command lines from standard input

Well, that didn’t help much.

The Standard Input

To understand what is really happening, we need to go into something called standard input. According to Wikipedia:

Standard input is a stream from which a program reads its input data. The program requests data transfers by use of the read operation. Not all programs require stream input.

Now we are getting somewhere! Let’s look further into this by using some examples.

You may know Grep, which takes a file and a pattern — a simple string, or even RegEx — and returns the lines within that file that match that given pattern.

A good Grep operation. Nice and simple.

In this example, we passed .zshrc as an argument to Grep. It searched for the pattern 'ls' within that file’s content. We can accomplish the exact same thing with the pipe syntax:

Exactly the same output as the previous image, but piping it with cat.

This syntax doesn’t make much sense if we already have the file we want to Grep, but it does become handy when we are Grepping a command output.

It is also possible to achieve a weird behavior by passing only one argument:

Grep expecting inputs

This same thing happens if you execute xargs and nothing else. But I’ll get there in a bit.

Notice that we didn’t use Xargs in the pipe syntax. Why not using it there, while in the first command snippet we had to use xargs mkdir to execute the command?

Let’s keep on the practical approach and see what happens when we add xargs to the previous example.

It didn’t work, but why?

As we saw earlier, Xargs is supposed to “build and execute command lines from standard input”. What happened here is that Xargs took all of the contents of .zshrc — which were passed to Xargs via standard input — and passed as arguments to Grep.

To visualize it, we can pass --verbose to xargs (or -t shorthand) so that it prints out what it has received.

It crunched the entire file into one line.

Everything that we are seeing there is being passed as arguments to the next command. Notice that it still executed Grep normally, giving us the same error as before. Now let’s take a closer look at Grep’s manual page:

SYNOPSIS
       grep [OPTION...] PATTERNS [FILE...]
       grep [OPTION...] -e PATTERNS ... [FILE...]
       grep [OPTION...] -f PATTERN_FILE ... [FILE...]

Take a look at the second format. Grep accepts many PATTERNS, and at the end it accepts a FILE.

If we pay close attention to the verbose output in the last screenshot, we can observe that ~/.fzf.zsh is the last file path to appear on the content, thus it considers it to be the file to search on for all the arguments that come before it.

I am not entirely sure what it does with the rest of the arguments that come afterwards, neither why it prompted “No such file or directory” since the file actually exists in this path. If anyone knows the reason, I would love to hear it. Reach out to me and I might update the article as well!

Read Operation

But why did it “broke” when we passed only one argument? It actually is exactly the intended behavior. Grep has two ways of processing the content of which it searches into for patterns:

1. Receiving a file as argument;
2. Receiving the content via standard input.

When we passed only one argument, we actually passed the pattern for it to match — even if it was a file path, respecting the argument order — and it expects that the actual content is yet to come from the standard input, which is what happens when we pipe it through.

As soon as the program on the left of the pipe exits, it ends the read operation and passes the content through the next command after the pipe.

It might be weird, but it’s actually a good thing! In my dotfiles repository I setup the following script:

#!/usr/bin/env bash

git clone --bare https://github.com/EduardoRodriguesF/dotfiles $HOME/.dotfiles

function dotfiles {
  /usr/bin/git --git-dir=$HOME/.dotfiles/ --work-tree=$HOME $@
}

dotfiles checkout

if [ $? = 0 ]; then
  echo "Checked out dotfiles.";
  else
    echo "Backing up pre-existing dot files.";

    mkdir -p .dotfiles-backup
    dotfiles checkout 2>&1 | egrep "\s+\." | awk {'print $1'} | xargs -I{} mv {} .dotfiles-backup/{}
fi;

dotfiles checkout

dotfiles submodule init
dotfiles submodule update

dotfiles config status.showUntrackedFiles no

Don’t worry about the actual implementation. Just keep in mind that this script is supposed to download my repository and replace the files in the current machine with the ones that were just cloned.

With that, I also provided a command in the README to execute it at once:

curl -Lks https://gist.githubusercontent.com/EduardoRodriguesF/600ed2f94ad4bdba947fbdf0ca698a9e/raw | bash

So what happens is that the script’s content will be cURL’d and passed to bash as an input. It only executes the script when the response is received. This would be equivalent to downloading the Gist to a file and then running Bash with the path as argument.

This is why we are “free to type” when we omit some arguments earlier. Standard input is expecting something to end so that it moves on the command.

Using Xargs The Right Way

Now that we managed to observe the different behaviors of standard input versus arguments, we have a better understanding of what Xargs does, so we can go on to use it like masters.

Previously we saw the --verbose flag, but there are a couple of helpful arguments that we can pass to Xargs to achieve certain things. For that manner, I really encourage you to look into Xargs’ manual page and experiment with those. For the sake of that article, I will show you an example with my favorite argument.

Let’s say that we want to revert a Git commit. We actually still don’t know which commit we want to revert, so we’re going to search for it using FZF. After finding, we want to execute git revert to do the job. You might initially think of doing something like that:

git log --oneline | fzf | xargs git revert

Just to ensure we’re on the same page, xargs is being used because git revert receives a commit as an argument, and doesn’t take any inputs from standard input, so Xargs will transform the output of fzf into arguments.

It didn’t work. That’s because it took the entire line and passed it as arguments. It would be the same as executing the line below:

git revert 2b8d302 refactor: separate input_stream from fighter input commands system

That is not exacly what we want . What we need is the commit hash (2b8d302) to make the revert. Xargs actually has a way of limiting the maximum number of arguments we want to pass, dropping the rest.

git log --oneline | fzf | xargs -n 1 git revert

Let’s see what happens now…

Despite the merge errors that we would need to resolve, it worked like a charm! Xargs got the hash into account and ignored all the rest, avoiding any invalid syntax.

Final Thoughts

Xargs is awesome! It enables us to write powerful commands by transforming inputs into arguments. That’s the power of automation right there.

I learned a lot about this subject while writing this article and I hope I managed to pass that knowledge to you, dear reader.

If any of the information here is wrong or expressed in a confused manner, please let me know.

I am planning to be much more present on this platform from now on, so I am hoping to see what are your thoughts on these. :)