Commanding the Line
Many of us live and work on the Linux command line.
I’ve been living in that neighborhood since 1986. For most of that time, the command line commanded me. But eventually, I started commanding the command line.
In this article, I will share with you an example of commanding the command line. I’m going to take the command line by its shirt collar and shake it until it does what I want. The target of my aggression will be a TrueBlocks command line provided by one of our users. (Thanks Pintail.)
Using Linux Pipes
First, I’ll start by showing you a command our user shared with us:
chifra export --cache --articulate \
--first_block 12550000 --last_block 13520000 \
--fmt csv 0x6b175474e89094c44da98b954eedeac495271d0f | \
grep 'approve' | \
tr -d '"' | \
cut -d ',' -f1,2,5,6,13-20 \
> dai_uniswapv2_approvals.csv
Translated into English, this says:
Show every transaction against the DAI StableCoin (0x6b17…71d0f), starting at block 12,550,000 and continuing until block 13,520,000 inclusive reporting the results in CSV format. Furthermore, articulate those results and cache the query as you go. Then…
shove the results through grep to pick off only the token approval transactions; remove quotes; and, using comma as the field delimiter, extract the first, second, fifth, sixth and thirteenth through the 20th fields (blockNumber, transactionIndex, from, to, compressedTx); and, finally, store the results in a file called dai_uniswapv2_approvals.csv
See — I told you. The command line tends to command the user. That’s a crazy-assed command.
Can We Improve on This?
Yes.
How?
There’s are five things going on in the above command:
- Producing the list of transactions with chifra export
- Filtering the records for approvals with grep
- Extracting only five of the many fields in each record with cut
- Removing quotes with tr
- Storing results in a file with a redirect (>)
We’ll start by simplifying things.
We remove any part of the above command that is irrelevant for this article (the —-cache
options and the --first_block
and --last_end
block options. We’ll keep the --articulate
option. We’ll also remove the piping, grepping, transforming, and re-direct.
So, we’re left with this command:
chifra export --fmt csv \
--articulate 0x6b175474e89094c44da98b954eedeac495271d0f
This command shows every transaction in the entire history of the DAI Stablecoin. We’ll add back options from here. Note: that’s a very large amount of data. You can limit the results with --first_block/--last_block
or --first_record/--max_records
. See the docs.
Filtering Records for Approvals
If one runs the commandchifra export --help
, one sees this summary:
chifra export [flags] \
<address> [address...] [topics...] [fourbytes...]
You can see here that it’s possible to provide not only addresses
to the export command, but also one or more topics
and/or one or more fourbyte
signatures.
In order to find the fourbyte
signature that we’re interested in, we can do this command:
chifra abis 0x6b175474e89094c44da98b954eedeac495271d0f | grep approve
which returns:
fourbyte type name signature
-----------------------------------------------------------------
0x095ea7b3 function approve approve(address,uint256)
We can add that fourbyte
signature to our growing command:
chifra export --fmt csv --articulate \
0x6b175474e89094c44da98b954eedeac495271d0f \
0x095ea7b3
This command shows every approve
transaction that was every run against DAI.
The above command replaces the need for grep
in the original command. This is also faster than the original command since many transactions (those that aren’t approve
) are never generated. There’s no need to grep them out.
Field Selection
Note: The following section describes an experimental feature of TrueBlocks. I may be removed or altered in future releases. Use with caution.
The original command uses tr
to remove quotes from the output and cut
to select fields 1,2,5,6, and 13–20 from the data. This can be accomplished using an undocumented (and therefore experimental) feature of TrueBlocks called “Display Strings.” I won’t explain Display Strings, because this article is already getting long, but they allow you to select fields and (in effect) describe the exact output of the data you want.
Run this command:
DISPLAY_FORMAT="[{BLOCKNUMBER}]" \
chifra export --fmt txt --articulate \
0x6b175474e89094c44da98b954eedeac495271d0f \
0x095ea7b3 2>/dev/null
Note: I’ve changed the export format from csv
to txt
in order to make the displayed data clearer. This also fixes an issue with the original command. The compressedTx
field contains commas, so using CSV doesn’t really work as the fields get garbled.
The above command produces row upon row showing only the blockNumber
of each transaction.
Run this command:
DISPLAY_FORMAT="[{BLOCKNUMBER}][{TRANSACTIONINDEX}]" \
chifra export --fmt csv --articulate \
0x6b175474e89094c44da98b954eedeac495271d0f \
0x095ea7b3 2>/dev/null
Row upon row of blockNumber
and transactionIndex
. You get the idea.
You can tell TrueBlocks which fields to display using an environment variable.
Fields 1,2,,5,6, and 13–20 correspond to blockNumber
, transactionIndex
, from
, to
, and compressedTx
. You can see the exported fields here.
Extending this to the fields we want, the command becomes:
DISPLAY_FORMAT="[{BLOCKNUMBER}]\t[{TRANSACTIONINDEX}]\t[{FROM}]\t[{TO}]\t[{COMPRESSEDTX}]" \
chifra export --articulate \
0x6b175474e89094c44da98b954eedeac495271d0f \
0x095ea7b3 2>/dev/null
Note: You must fix the above command to put it on a single line.
If you run the above command trough | head -1
you will see that only the fields we’re interested in are being exported:
blocknumber transactionindex from to compressedtx
No quotes. That’s what we want. This is command is faster because you don’t have to pipe extraneous data through the cut
command. When you’re doing command-line data processing, every little tweak counts.
Storing the Results in a File
So far, we’ve eliminated grep
, cut
, and tr
. Can we eliminate the redirect?
Yes.
Chifra has an option called --output <filename>
which lets the user specify the destination of the resulting data.
The final, improved command, then, is:
DISPLAY_FORMAT="[{BLOCKNUMBER}]\t[{TRANSACTIONINDEX}]\t[{FROM}]\t[{TO}]\t[{COMPRESSEDTX}]" \
chifra export --articulate \
0x6b175474e89094c44da98b954eedeac495271d0f \
0x095ea7b3 --output dai_uniswapv2_approvals.csv
Note: Edited to be all on one line.
Performance
Here are the results of testing the two commands for speed:
9.79 seconds for the original command. 8.89 seconds with the new command. A speedup of 10%. Not as good as I would have liked, but at least something.
The above timing tests were run against 1,360 transactions. The DAI smart contract has (as of the time of this writing) 12,956,972 transactions. We found this number with:
chifra list --count 0x6b175474e89094c44da98b954eedeac495271d0f`
That means that in order to extract every transaction, we would have to wait 9,527.18
times longer than our above tests.
For the first command, this would be 9.79 seconds * 9527.18 = 25.9 hours.
For the second command, it would have taken 8.89 * 9527.18 = 23.52 hours.
A savings of more than two hours. Take yourself out to dinner on the difference.
Support Our Work
TrueBlocks is totally self-funded from our own personal funds and a few grants such as The Etheruem Foundation (2018), Consensys (2019), Moloch DAO (2021), and most recently Filecoin/IPFS (2021).
If you like this article or you simply wish to support our work go to our GitCoin grant https://gitcoin.co/grants/184/trueblocks. Donate to the next matching round. We get the added benefit of a larger matching grant. Even small amounts have a big impact.
If you’d rather, feel free to send any token to our public Ethereum address at trueblocks.eth or 0xf503017d7baf7fbc0fff7492b751025c6a78179b.
Join Coinmonks Telegram Channel and Youtube Channel learn about crypto trading and investing
Also, Read
- Blockfi vs Coinbase | BitKan Review | Binance Review
- Coldcard Review | BOXtradEX Review | A Guide to Uniswap
- 5 Best Crypto Exchanges in UAE | SimpleSwap Review
- 7 Best Ways to buy Dogecoin | ZebPay Review
- iTop VPN Review | Mandala Exchange Review
- Coinbase vs WazirX | Bitrue Review | Poloniex vs Bittrex
- Best Crypto Trading Bots in the US | Changelly Review
- A-Ads Review | Bingbon Review | Mudrex Invest
- Best Cardano Wallets | Bingbon Copy Trading
- Best P2P Crypto Exchanges in India | Shiba Inu Wallets
- Top 8 Crypto Affiliate Programs | eToro vs Coinbase
- Best Ethereum Wallets | Cryptocurrency Bots on Telegram
- Best Exchanges to Trade Leveraged Tokens