Start programming Node.js http2 client mode (part III): get access_token with service account

8 min readNov 7, 2017

from last post we talked about POST method with http2 client, to get access token from google’s token server, that was relying on the user’s application_default_credentials.json file if the user has ever called gcloud auth application-default login the gcloud is part of google-cloud-sdk package, it’s a shell script wrapper of python script which does pop up a browser page for user to authorize the app (gcloud) and get an refresh_token, the same behavior can be done in the Node.js script as well, it’s an easy one I’m not planning to talk in detail, have the code share later (if you give feedback of interest to that).

Today I’d like to talk about another kind of authentication supported by google’s token server, which is service account, more oriented to server application, it’s also called headless account in some context.

If you’re using Google Cloud services but never used service account, you may refer to Using OAuth 2.0 for Server to Server Applications on Google Developer site about what is it and how to create one, here I just assume you have one already, and downloaded the JSON format private key, the content should look similar like this:

{
  "type": "service_account",
  "project_id": "project-id",
  "private_key_id": "437a4c3...........",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANB.................\n-----END PRIVATE KEY-----\n",
  "client_email": "account-name@project-id.iam.gserviceaccount.com",
  "client_id": "10599...............",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/......."
}

it’s a valid JSON file, with most important fields are client_email and private_key we will use later.

The HTTP/REST interaction to Google’s token server

Here I just have a copy of the interaction process to google’s token server

from this process, we can see 3 major steps: 1) create JWT and sign it; 2) send a POST request; 3) got the access_token.

The JWT is JSON Web Token needs to be created as this format:

{Base64url encoded header}.{Base64url encoded claim set}.{Base64url encoded signature}

As in above google link creating JWT in HTTP/REST mentioned, the header is always as {“alg”: ”RS256", ”typ”: ”JWT”} because google’s token server uses RSA SHA-256 algorithm, from Node.js crypto module’s doc we see that Node.js builtin crypto module can work it out, hence no need to call an external dependency, the code would be something likecrypto.createSing(‘RSA-SHA256’) and then call signer’s .update method to write in the data to be signed, lastly is signer.sign(‘private-key’, ‘output-format’) to get signature.

the base64url encoding

Another step is to call base64url encode it; from last writing of current @google-cloud/bigquery dependencies analysis, many of such need is calling the base64url package on npm registry , it has some millions of downloads in each past month; but, is it really a good quality module and really necessary? from Node.js buffer doc we know that Nodejs support to convert a Buffer to base64 output format string, so a question is can we just use the builtin module? how far is it to base64url encoding? we can read the base64url package’s source code, or read the rfc mentioned in the nodejs doc:

'base64' - Base64 encoding. When creating a Buffer from a string, this encoding will also correctly accept "URL and Filename Safe Alphabet" as specified in RFC4648, Section 5.

the base64 encoding uses 26 uppercase letters, 26 lowercase letters, 10 digits, and still need 2 extra symbol charaters to form all 64 characters for encoding, the original design purpose of base64 was for email content safe in the 1980s, uses ‘+’ and ‘/’ for the last two (octet 62 and 63), because email was invented much earlier (1970s) than the web (1990s), as far as I understand, around that early time some routing (or called relaying) system in the email delivery uses 7bit bytes only, was cutting the high bit to zero for characters above 128, hence sending email with 8bits bytes weren’t safe and need encoding all bytes to a 7bits safe system, the use of base64 actually use 6bits for each byte, to be further safe; when the time come to 1990s the web was invented, people tried to use base64 for the web, in the URLs and some download link, then realized that the last two charters of ‘+’ and ‘/’ are not safe on the web, the ‘+’ has special meaning in the URL parsing, and ‘/’ is not valid filename charters, to many filesystems; hence people created a variant to replace the last two to ‘-’ and ‘_’ for octet 62 and 63;

So with this knowledge, can we use builtin API only and create something better than base64url package? I think so

first step is to convert any content to a Buffer, and call the buffer’s toString with ‘base64’ Buffer.from('to-be-encoded').toString('base64') this way we get the base64 encoded string;
second step is to convert any ‘+’ to ‘-’ (minus) and ‘/’ to ‘_’ (underscore); there are many stackoverflow threads talked about this, seems no way to use just one regex replace; here I have to mention if any readers also have a Python background, should remember Python’s string module has a translate table concept, which would be perfect to translate characters in a one-to-one mapping;
through re-read the MDN String.replace documentation, I found we can use the replacer function to achieve that translate table, hence get a one loop only:

// input is a string
function base64url(toEncoded) {
  return Buffer.from(toEncoded).toString('base64')
    .replace(/[+/]/g, x => {
      switch (x) {
        case '+': return '-';
        case '/': return '_';
      }
      return x; // shouldn't run to here
    });
}

this should work; I haven’t benchmark is it faster than two .replace calls as in the base64url package does? that depends on each Javascript runtime’s String.replace implementation, for short strings it may be faster to call .replace twice, but its complexity is O(n) anyway, and twice of the loop to the length of input, for longer string, replace twice won’t be efficient anyway; if we can loop the input only once, it should be faster than run loop twice.

Just for completeness, I can also make a maketrans function similar to Python’s:
(but do not use in production because not complete checking invalid input)

// use as  thebase64encoded.replace(/[+/]/g, maketrans('+/', '-_'))
// get output is base64url encoded
function maketrans(from, to) {
  const table = {};
  for (let i = 0; i < from.length; i++)
    table[from[i]] = to[i];
  return function (x) {
    return table[x];
  }
}// and we can also create a polyfill; use of the ES7 spread operator
if (!String.prototype.translate) {
  String.prototype.translate = function(from, to) {
    const table = Array.from(from)
      .reduce((acc, x, i) => ({ ...acc, [x]: to[i] }), {});
    return this.replace(
      new RegExp(`[${from}]`, 'g'), x => table[x]);
  }
}
// use like
> 'abcdefghi+/+++++'.translate('+/', '-_')
'abcdefghi-_-----'

the claim set

the second portion in the JWT to-sign content is the claim set, from the documentation, it need:

iss: the client email of the service account;
scope: the permissions to request
aud: A descriptor of the intended target of the assertion
iat and exp: in seconds since unix epoch.

once composed the claim set inJSON.stringify we can also call base64url function to get encoded;

const seconds = Math.ceil(Date.now() / 1000);
const claim = JSON.stringify({
  iss: client_email, scope, aud, iat: seconds, exp: seconds+3600 });// got a claim string like this:
{"iss":"account-name@project-id.iam.gserviceaccount.com","scope":"https://www.googleapis.com/auth/cloud-platform.read-only","aud":"https://www.googleapis.com/oauth2/v4/token","iat":1510092000,"exp":1510095600}

Hint here: someone might ask here can we provide a iat in the future or in the past? and exp not exactly one hour long? I did that test, however, I found that google’s token server will always override the iat to current timestamp, and exp to be one hour session; but if omit iat and exp in the post request, that ends an error

to sign the content so far

the code to sign is to use crypto module as described earlier:

const signer = crypto.createSign('RSA-SHA256');
const tosigncontent = base64urlencodedHeader + '.' +
            base64urlencodedClaim;
signer.update(tosigncontent);
const signature = signer.sign(private_key, 'base64')

the post body

once get the signature, just concatenate with sign header and claim set; that is assertion string, from google’s doc, need to provide another grant_type: ‘urn:ietf:params:oauth:grant-type:jwt-bearer’, parameter in the postbody, by form-urlencoding; again, we can do this by querystring module;

lastly is the http2 POST method as the same way as previous writing.

The full code

and some extra explanation:

it’s still, using builtin modules; tested on Node.js v8.8.1 and v9.0.0
the GoogleToken constructor, now take a keyfile parameter, by convention, it can default to the GOOGLE_APPLICATION_CREDENTIALS environment variable; just because @google-cloud/bigquery and many other google’s library support read keyfile from environment; it also support a given path to a keyfile;
now the constructor function need to read the keyfile content, after loaded file content, read it by JSON.parse ; this place didn’t catch invalid file content, so don’t just copy use this code to production; use this.ready a promise indicator, if it is resolved to a true value, this.ready will be set to true; in some cases, this trick could save one event loop in Node.js execution.
the getToken function now takes a scope parameter, and defaults to read-only use of retrieving information only;
the Line30 of checking this.ready === true isn’t really necessary, because Line32 of Promise.resolve(this.ready) would take them all, works for both cases when this.ready is resolved or not resolved yet;
while, for some particular cases, by checking if resolved already, can save one event-loop runtime.
the Symbol use of postrequest is the special way of really private data, or private method in this case; Symbol was first time brought into Javascript standard in ES6 (or ES2015); because every Symbol is unique, if the Symbol is not exported, no users can get this Symbol, hence no way to directly access the postrequest method here;
in the long time past ES5 and earlier time, people use all kinds of _ (underscore) prefixes for private methods, those were not really private;
on Line32: the method this[postrequest] is not bound by default, if need, like in this case, can call bind the object this, and first parameter scope; alternative way to write this line is:
Promise.resolve(this.ready).then(() => this[postrequest](scope));
which works because of ES6 rule, the arrow function will bind this, always
the postrequest function is to create the signHeader, claim set, and signature; concatenate them all, and call querystring.stringify to get the postbody.
the actual post method is similar as before; there were someone suggested always include a user-agent that’s for sure we can do, on Line70;
the String.prototype.translate and tobase64url polyfill, as explained already;
the BufferChunks class, is a helper class, to save one loop, and save utf8 decoding on every single data chunk, on Line135, if Buffer.concat is not provided with a total_length, it would do a loop first, the calculate what’s total length needed for the new buffer, but if we already have that, we should provide it;
lastly is the main function, as a simple test case, or to make it as a simple useful script, to be able to get a token; it also call getToken again to see that cache one hour works, but the 2nd call has different scope than the first call, a more feature-full way should cache tokens by scope, however it could be beyond this simple tutorial

run as follow:

$ ~/opt/node-v9.0.0-linux-x64/bin/node ~/tmp/gtoken.js
(node:21607) ExperimentalWarning: The http2 module is an experimental API.
'ya29.c.El79BII...'
get token again: ya29.c.El79BII........