The Startup
Published in

The Startup

How to package a JavaScript library for use in BigQuery

In this article you’ll learn how to package a JavaScript library for use in a BigQuery UDF. We’ll consider a particular example — ethers.js —a complete Ethereum library and wallet implementation in JavaScript. It will allow us to decode raw transactions and logs data in the bigquery-public-data.crypto_ethereum dataset in BigQuery.

The whole process can be broken down into three steps:

You can find the source code for bundling and using ethers.js in this Github repo:

Now, let’s go through each step in detail.

Create a package.js file

In the image below I highlighted the webpack build script, which we will use later; the dependency on the “ethers” library with its version; and dependencies on webpack and webpack-cli:

Run npm install to download the dependencies.

Create webpack.config.js and build a JS file using webpack

The image below highlights the entry point, starting from which webpack will aggregate all transitive dependencies and assemble them in a single file:

The output configuration tells it to save the result todist/ethers.js file, and that the library will be available as ethers variable when included in a UDF.

Run npm run-script build to generate dist/ethers.js.

Upload the generated JS file to GCS and use it in BigQuery UDF

You’ll need to create a GCS bucket, upload the generated file to it, and make the file publicly readable:

gsutil mb gs://your-bucket
gsutil cp dist/ethers.js gs://your-bucket
gsutil acl ch -u AllUsers:R gs://your-bucket/ethers.js

Now you are ready to use it in a UDF for parsing Ethereum data. The image below highlights the usage of the ethers variable available inside the UDF; how the library is included; and the logs table in the crypto_ethereum dataset:

You can run it by pasting into the BigQuery console query
editor: And this is how the result looks like (all CryptoKitties Transfer events):

Also read this article from my fellow D5 member Alex Svanevik on How to get any Ethereum smart contract into BigQuery (in 8 mins).



Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +768K followers.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store