Start programming http2 client mode (part I): to call BigQuery services, using Node.js Version 8 core API only (zero external dependency)

One of the “5 Fantastic Features Shipping with Node.js 8 LTS” is HTTP/2, this has just happened since v8.4.0 released in this August 2017 “Experimental support for the built-in http2 has been added”, and since earlier this year there were many ones blogging on tutorials how to make a nodejs app support HTTP/2 server mode, while, if you ask how many client side libraries already support HTTP/2 mode? from current npm packages I’m not seeing any one yet, include the most popular request and axios do not have HTTP/2 client mode support yet.

Credit: Rob Bates

From the http2 API doc, you may see that has well support of HTTP/2 client mode, so we can start playing with it, right? from my recent project I found this could be a cool idea and start writing this as a HTTP/2 client tutorial.

Before starting with HTTP/2 client mode, we need to ask a question, which is how many servers are already an HTTP/2 server? after all, an HTTP/2 client can only talk with a HTTP/2 server, because if you know somewhat HTTP/2 protocol, it’s a binary protocol and is not compatible with HTTP/1.x with many differences; if you’re not very familiar with HTTP/2, I would recommend you have a read of Ilya Grigorik‘s well written book «High Performance Browser Networking» from 2013, he put the book as freely readable and it has a whole chapter on HTTP/2.

The answer to above question also has a good website isthewebhttp2yet maintained how many public servers are already supporting HTTP/2; the simple answer is THERE ARE MANY! and that adoption list is still steadily fast growing.

Just because my project is Google BigQuery related, and Google is the main one promoting HTTP/2, most googleapis have good HTTP/2 support, if not all (I’m not sure if google ever announced all their apis support HTTP/2, but as far as I know, all the Google Cloud services I’ve played with, like BigQuery, ComputeEngine, AppEngine, CloudStorage, DataStore, PubSub, and many others, all support HTTP/2 on server side).

So, First, Let’s start http2 client programming

Goal: in this tutorial, I’m going to make a simple script run as cli, to list all tables in bigquery and retrieve metadata about each one; also, because of no 3rd party library support http2 client mode yet, this script does not use any 3rd party library, hence core API only;
as long as you have installed latest version of Nodejs Version 8, you can run it; (the latest is v8.8.1 as I am writing this.)

Why I want to do this? there is @google-cloud/bigquery library maintained by google-cloud team and you can call their api like listTable and do a loop getMetadata to get all of them, but however it isn’t efficient because can’t pass in fieldsparameters, especially when I already have some tens of thousands of tables, it’s slow to call that library wrapped API; it’s also because that library calls request underlyingly that I know it’s calling by HTTP/1.x protocol, but here I want to show it’s now possible to write nodejs code to call bigquery from HTTP/2 protocol!

From the examples from nodejs http2 api doc, we can derive a simple structure like this:

If you have a google-cloud project and have been using the bigquery, you may change the const projectId and access_token to a valid one, and this code can run, it just list your datasets, from BigQuery’s REST API Reference you may find all models;

Update: if you have nodejs version 8 but lower than v8.8, don’t forget to run your node version 8 with --expose-http2 because it’s not exposed by default: but if you have nodejs v8.8+ you don’t need this flag:

$ node --expose-http2 ./http2client.js

Hint: use gcloud auth application-default print-access-token can get a token for you; in later writeup I’ll talk about how to call token API to request one dynamically, but for this tutorial, to keep it short, just get one from gcloud command, it’s valid for 1 hour session, probably enough for a tutorial.

Secondly: to make it more useful

just list the datasets isn’t interesting, let’s list all tables as well:

things changed:

1. when we’re looping all datasets, we need to wait all listTable function finish in the callback; since Nodejs v6 we have the native Promise API and v8 added the async await, keeps code way clear than callbacks hell before;
   so here use async await on native Promise api, let the code look like a sequential blocking call but nodejs event loop is still doing pretty good single-thread asynchronous non-blocking IO in the background;
2. the reason for having to call clientSession.destroy() is because this connection isn’t by default unref’ed; would hold the event loop if do not call it; if remove you’ll find the script is hanging itself until 3 minutes later server side shutdown connection.
3. You see the benefit of HTTP/2 client? Right, it doesn’t need to create another connection, and list tables from different datasets don’t depend on each other, the results from each dataset might come back out of order, because it’s multplex reusing the same TCP connection; if you read Ilya’s book, on most TCP client implementation, there is an TCP slow start problem affecting HTTP1’s performance, and even though HTTP1 has keep-alive, that connection reusing can only call one after another, never like the HTTP/2 concurrently re-using the single TCP connection as of today!

Thirdly: show table information as well

the listTables api can return some basic information on each table, but even if you print the whole response from listTables, it isn’t that much information on each table, so how to get more metadata for each table? we need to call the get metadata api: https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/get

It is that simple! this script can do so far: 1) list all datasets, 2) for each dataset list all its tables, 3) for all tables, list how much storage does it use; based on that output you may recognize which table uses the most storage, (and hence cost the most!)

In a summary, to call Nodejs Version 8 http2 client mode, you just need to

1. call http2.connect(...) to make the connection session, and then
2. callsession.request({ ':path': '/request-path', ... }) to make a GET request with specified path, and you may specify any other headers as well, like authorization is required when interacting with Google’s cloud services.
3. within the same session, you can make as many requests as you want, unless server side shut down the connection or some other errors; (later post will talk more about handling errors for http2 client)

In the following write, I’ll talk about how to interact google’s token server, in the HTTP/2 way, and with Nodejs core API only, that refers POST method, and others.

Update:

• Part II: use POST method to get access_token from google’s token server https://medium.com/@ty0h/start-programming-node-js-6c2bf588c511
• part III: get access_token with service account https://medium.com/@ty0h/start-programming-node-js-http2-client-mode-part-iii-get-access-token-with-service-account-59f5715c15c3

Do you like this post? like more on Nodejs Version 8 latest features? like more on HTTP/2 ? if so, please let me know.
Click 👏 (Clap) below to help others discover this story. Please follow me if you want to get updates about new posts or boost work on future stories.