Nov 22, 2021

6 min read

Commanding the Line

Many of us live and work on the Linux command line.

I’ve been living in that neighborhood since 1986. For most of that time, the command line commanded me. But eventually, I started commanding the command line.

In this article, I will share with you an example of commanding the command line. I’m going to take the command line by its shirt collar and shake it until it does what I want. The target of my aggression will be a TrueBlocks command line provided by one of our users. (Thanks Pintail.)

Using Linux Pipes

First, I’ll start by showing you a command our user shared with us:

chifra export --cache --articulate \
    --first_block 12550000 --last_block 13520000 \
    --fmt csv 0x6b175474e89094c44da98b954eedeac495271d0f | \
    grep 'approve' | \
    tr -d '"' | \
    cut -d ',' -f1,2,5,6,13-20 \
    > dai_uniswapv2_approvals.csv

Translated into English, this says:

Show every transaction against the DAI StableCoin (0x6b17…71d0f), starting at block 12,550,000 and continuing until block 13,520,000 inclusive reporting the results in CSV format. Furthermore, articulate those results and cache the query as you go. Then…
shove the results through grep to pick off only the token approval transactions; remove quotes; and, using comma as the field delimiter, extract the first, second, fifth, sixth and thirteenth through the 20th fields (blockNumber, transactionIndex, from, to, compressedTx); and, finally, store the results in a file called dai_uniswapv2_approvals.csv

See — I told you. The command line tends to command the user. That’s a crazy-assed command.

Can We Improve on This?

Yes.

How?

There’s are five things going on in the above command:

Producing the list of transactions with chifra export
Filtering the records for approvals with grep
Extracting only five of the many fields in each record with cut
Removing quotes with tr
Storing results in a file with a redirect (>)

We’ll start by simplifying things.

We remove any part of the above command that is irrelevant for this article (the —-cache options and the --first_block and --last_end block options. We’ll keep the --articulate option. We’ll also remove the piping, grepping, transforming, and re-direct.

So, we’re left with this command:

chifra export --fmt csv \
    --articulate 0x6b175474e89094c44da98b954eedeac495271d0f

This command shows every transaction in the entire history of the DAI Stablecoin. We’ll add back options from here. Note: that’s a very large amount of data. You can limit the results with --first_block/--last_block or --first_record/--max_records. See the docs.

Filtering Records for Approvals

If one runs the commandchifra export --help, one sees this summary:

chifra export [flags] \
    <address> [address...] [topics...] [fourbytes...]

You can see here that it’s possible to provide not only addresses to the export command, but also one or more topics and/or one or more fourbyte signatures.

In order to find the fourbyte signature that we’re interested in, we can do this command:

chifra abis 0x6b175474e89094c44da98b954eedeac495271d0f | grep approve

which returns:

fourbyte       type         name        signature
-----------------------------------------------------------------
0x095ea7b3     function     approve     approve(address,uint256)

We can add that fourbyte signature to our growing command:

chifra export --fmt csv --articulate \
    0x6b175474e89094c44da98b954eedeac495271d0f \
    0x095ea7b3

This command shows every approve transaction that was every run against DAI.

The above command replaces the need for grep in the original command. This is also faster than the original command since many transactions (those that aren’t approve) are never generated. There’s no need to grep them out.

Field Selection

Note: The following section describes an experimental feature of TrueBlocks. I may be removed or altered in future releases. Use with caution.

The original command uses tr to remove quotes from the output and cut to select fields 1,2,5,6, and 13–20 from the data. This can be accomplished using an undocumented (and therefore experimental) feature of TrueBlocks called “Display Strings.” I won’t explain Display Strings, because this article is already getting long, but they allow you to select fields and (in effect) describe the exact output of the data you want.

Run this command:

DISPLAY_FORMAT="[{BLOCKNUMBER}]" \
    chifra export --fmt txt --articulate \
    0x6b175474e89094c44da98b954eedeac495271d0f \
    0x095ea7b3 2>/dev/null

Note: I’ve changed the export format from csv to txt in order to make the displayed data clearer. This also fixes an issue with the original command. The compressedTx field contains commas, so using CSV doesn’t really work as the fields get garbled.

The above command produces row upon row showing only the blockNumber of each transaction.

Run this command:

DISPLAY_FORMAT="[{BLOCKNUMBER}][{TRANSACTIONINDEX}]" \
    chifra export --fmt csv --articulate \
    0x6b175474e89094c44da98b954eedeac495271d0f \
    0x095ea7b3 2>/dev/null

Row upon row of blockNumber and transactionIndex. You get the idea.

You can tell TrueBlocks which fields to display using an environment variable.

Fields 1,2,,5,6, and 13–20 correspond to blockNumber, transactionIndex, from, to, and compressedTx. You can see the exported fields here.

Extending this to the fields we want, the command becomes:

DISPLAY_FORMAT="[{BLOCKNUMBER}]\t[{TRANSACTIONINDEX}]\t[{FROM}]\t[{TO}]\t[{COMPRESSEDTX}]" \
    chifra export --articulate \
    0x6b175474e89094c44da98b954eedeac495271d0f \
    0x095ea7b3 2>/dev/null

Note: You must fix the above command to put it on a single line.

If you run the above command trough | head -1 you will see that only the fields we’re interested in are being exported:

blocknumber    transactionindex   from   to   compressedtx

No quotes. That’s what we want. This is command is faster because you don’t have to pipe extraneous data through the cut command. When you’re doing command-line data processing, every little tweak counts.

Storing the Results in a File

So far, we’ve eliminated grep, cut, and tr. Can we eliminate the redirect?

Yes.

Chifra has an option called --output <filename> which lets the user specify the destination of the resulting data.

The final, improved command, then, is:

DISPLAY_FORMAT="[{BLOCKNUMBER}]\t[{TRANSACTIONINDEX}]\t[{FROM}]\t[{TO}]\t[{COMPRESSEDTX}]" \
    chifra export --articulate \
    0x6b175474e89094c44da98b954eedeac495271d0f \
    0x095ea7b3 --output dai_uniswapv2_approvals.csv

Note: Edited to be all on one line.

Performance

Here are the results of testing the two commands for speed:

9.79 seconds for the original command. 8.89 seconds with the new command. A speedup of 10%. Not as good as I would have liked, but at least something.

The above timing tests were run against 1,360 transactions. The DAI smart contract has (as of the time of this writing) 12,956,972 transactions. We found this number with:

chifra list --count 0x6b175474e89094c44da98b954eedeac495271d0f`

That means that in order to extract every transaction, we would have to wait 9,527.18 times longer than our above tests.

For the first command, this would be 9.79 seconds * 9527.18 = 25.9 hours.

For the second command, it would have taken 8.89 * 9527.18 = 23.52 hours.

A savings of more than two hours. Take yourself out to dinner on the difference.

Support Our Work

TrueBlocks is totally self-funded from our own personal funds and a few grants such as The Etheruem Foundation (2018), Consensys (2019), Moloch DAO (2021), and most recently Filecoin/IPFS (2021).

If you like this article or you simply wish to support our work go to our GitCoin grant https://gitcoin.co/grants/184/trueblocks. Donate to the next matching round. We get the added benefit of a larger matching grant. Even small amounts have a big impact.

If you’d rather, feel free to send any token to our public Ethereum address at trueblocks.eth or 0xf503017d7baf7fbc0fff7492b751025c6a78179b.

Join Coinmonks Telegram Channel and Youtube Channel learn about crypto trading and investing