My journey to bring Web Push support to Node and Electron
Push notifications are now used on all platforms (mobile, Web and desktop). When developing a cross-platform application, the main goal is to bring a shared set of features to all users whatever the platform they are using.
The easiest way to send and receive notifications is to use Firebase Cloud Messaging (FCM), which provides integrations for Android, iOS and Web.
Unfortunately, if you’re like me and you use Electron to build a desktop application, you can’t receive push notifications sent through FCM.
Electron and Push notifications
A few months ago, I had to add push notifications to a cross-platform application built with Electron. I started working on the Web app and it was pretty straightforward using Firebase and the Firebase JS SDK. I first thought that it would also work seamlessly with Electron. After all Electron is just Chromium + Node right? Well… not really (more on this later).
I was surprised to encounter the following error :
AbortError: Registration failed - push service not availableThis error was raised by the browser when the Firebase JS SDK was trying to subscribe to push notifications via the browser’s PushManager
serviceWorkerRegistration.pushManager.subscribe()To receive push notifications in Chrome, you first have to register a service worker, which will receive the push notifications when your app is not opened, then subscribe to notifications through the PushManager.
The problem is that the PushManager is missing in Electron.
See this issue on the Electron repo for more context.
Websockets to the rescue
I use a lot of Electron-based desktop apps daily, like Slack.
I asked myself “How does Slack manage to send me push notifications?”.
Going through Slack source code (yes, you can easily unpack an Electron asar), I found that they were using Websockets to send & receive notifications.
The problem with this solution is that you have to set up a notification service, and store the notifications somewhere in case that your user is, for example, disconnected. Moreover, this service is only needed for your Electron users and you have to handle two ways of sending notifications (FCM + websocket).
A bummer.
Can’t we just bring Web Push support to Electron?
The missing parts
While investigating, I discovered that Electron was not built on top of Chromium itself but only on a subset called Chromium Content (using libchromiumcontent), which is the core part of Chromium needed to render a web page. For example, it doesn’t include all Google APIs and Services built in Chromium and, unfortunately, it’s missing the Push Notification API & Push Service needed to receive push notifications.
Using Chromium Content allows to have a smaller binary and disable a lot of services that are not needed for an Electron application.
Bringing those parts to Electron
Electron does provide more features than Chromium Content by bringing modules directly from the Chromium’s source code and patching them. So I thought about adding the needed files from the Chromium’s source code to make push notifications work.
I searched through the Chromium’s source code to extract the minimal amount of code which would need to be added to Electron to support Web Push.
At that time, I had to clone the Chromium repository (which takes quite some time) and used Atom’s “Find in project” feature to search through the source code, which could take several minutes each time. Later, I found Chromium Code Search which allows you to search very efficiently through Chromium’s source code and provides neat IDE-like features, such as click to definition etc… and saved me precious hours.
There were several modules and folders that needed to be copypasted & adapted like google_apis/gcm, chrome/browser/gcm, components/gcm_driver & more. All those files where using even more files from other modules. I’m not a C++ developer so to make it work, it felt like I had to bring all Chromium to Electron.
So I gave up…
Reverse engineering Chromium Web Push client
But not for long.
Chromium Push Service is just a service which can register to Google server & maintain a live connection to an endpoint to receive notifications.
I decided that I could build from scratch a minimal JavaScript Push Service by reverse engineering the Chromium Web Push client.
Protocol Buffers
While going through the Chromium’s source code, I found out that in order to communicate with Google servers, Chromium uses Protocol Buffers.
What are protocol buffers?
Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data — think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.
When using protocol buffers, you are sending raw buffers between your client and your server and you need to share schemas between clients and servers that describe the content of your messages (as for GraphQL for example).
Those schemas are defined in .proto files. Three proto files were needed to build the JavaScript client: mcs.proto, android_checkin.proto and checkin.proto .
An example of a proto schema:
message LoginResponse {
required string id = 1;
// Not used.
optional string jid = 2;
// Null if login was ok.
optional ErrorInfo error = 3;
repeated Setting setting = 4;
optional int32 stream_id = 5;
// Should be "1"
optional int32 last_stream_id_received = 6;
optional HeartbeatConfig heartbeat_config = 7;
// used by the client to synchronize with the server timestamp.
optional int64 server_timestamp = 8;
}The downside of protobuffers is that it’s very difficult to decode a message without the schema and that you need to know what kind of messages the server is sending to use the right schema for decoding. Sometimes, a message can be valid for different schemas… which isn’t nice when you’re reverse engineering.
Check-in and registration
Even though Google Cloud Messaging (GCM) has been replaced by Firebase Cloud Messaging (FCM), GCM is still used internally by Google to send push notifications. Actually, when you’re sending notifications to FCM, these notifications are then forwarded to GCM which handles delivery to the mobile & web clients.
So we need to register to both GCM & FCM if we want to be able to send and receive notifications.
To register to Google Cloud Messaging (GCM), we need to first check in to the same GCM to retrieve an androidId and a securityToken by sending information about our client like version numbers, etc…
Yes an androidId even though we’re not an Android device. Actually there are a lot of references to Android in Chromium source code because GCM was first built only for Android devices. I guess that’s also why the endpoint is named android.clients.google.com.
Once checked in, we need to register to GCM to retrieve a GCM token by sending back the androidId, the securityToken, a user generated appId and the server key that will be used to send notifications. Here, the server key is the FCM one, because as said above, FCM servers will send notifications on our behalf.
Lastly, we have to register to FCM using the GCM token to receive a FCM token that we will use to send notifications to this client. We also have to generate an ECDH public-private key pair and an auth secret that will be used to decrypt the notifications (more on this later).
MCS Client
To receive notifications, Chromium uses a MCS client (I didn’t find the meaning of MCS, so if anyone has the answer, feel free to leave a comment).
This client maintains a live socket connection to a Google server at mtalk.google.com.
To successfully connect to this server, the client has to write a login request proto message with the androidId, security token and other information.
The login request schema:
message LoginRequest {
enum AuthService {
ANDROID_ID = 2;
}
required string id = 1; // Must be present ( proto required ), may be empty
// string.
// mcs.android.com.
required string domain = 2;
// Decimal android ID
required string user = 3;required string resource = 4;// Secret
required string auth_token = 5;// Format is: android-HEX_DEVICE_ID
// The user is the decimal value.
optional string device_id = 6;// RMQ1 - no longer used
optional int64 last_rmq_id = 7;repeated Setting setting = 8;
//optional int32 compress = 9;
repeated string received_persistent_id = 10;// Replaced by "rmq2v" setting
// optional bool include_stream_ids = 11;optional bool adaptive_heartbeat = 12;
optional HeartbeatStat heartbeat_stat = 13;
// Must be true.
optional bool use_rmq2 = 14;
optional int64 account_id = 15;// ANDROID_ID = 2
optional AuthService auth_service = 16;optional int32 network_type = 17;
optional int64 status = 18;// 19, 20, and 21 are not currently populated by Chrome.
reserved 19, 20, 21;// Events recorded on the client after the last successful connection.
repeated ClientEvent client_event = 22;
}
At first, I wasn’t able to maintain a live connection and was always kicked out after 10 seconds by the server.
I was stuck. For quite some time.
Debugging Chromium
As looking through the Chromium’s source code and guessing what could be the value of variables was clearly not going to cut it, I decided to build Chromium from source to add debug statements.
I set up all the tools needed to build Chromium and the first build took more than 8 hours on my MacBook Air 😱 😱. But the next builds were, thankfully, way quicker.
Going through the documentation, it said that I could use Xcode to debug Chromium. To do so I would have to change the build options and rebuild Chromium from scratch 😱 😱 😢. No way!
As I have very little experience with C++ debugging tool and lldb, I gave up on trying to debug Chromium.
Proxies for the win
An easier way to know what was sent by Chromium to Google servers was to set up a proxy which would intercept and forward all requests made to mtalk.google.com.
As the connection to mtalk was using TLS, I had to generate a self-signed certificate, add it to the macOS Keychain and mark it as trusted.
Doing so, I discovered that my LoginRequest proto buffer was missing 4 bytes at the begining and 3 of those 4 bytes had always the same value for the two versions of Chrome/Chromium I used (my own build and the Chrome version installed on my MacBook). By adding those 4 bytes to my login buffer, I wasn’t been kicked out anymore 🎉.
I thought I was done but I couldn’t receive any notifications 😞.
Stuck I was again.
As my login request seemed to be ok, I thought I had an issue with the registration requests.
I set up a proxy for all requests made to android.clients.google.com. Doing so I could fine tune the registration parameters. Actually, I didn’t find any issue with these requests.
By adding some logs, I discovered something odd. The androidId I was sending in the login request was different from the one I received from the registration servers. It seems somewhat rounded. Instead of 5302112185968951120 I was sending 5302112185968951000 for example.
My problem was that I had to convert the androidId to hexadecimal and for this I was first converting it to a Number using
const hexa = parseInt(androidId, 10).toString(16);But the androidId was greater than Number.MAX_SAFE_INTEGER and that’s why it got truncated.
The solution was to use the long.js library:
const hexa = Long.fromString(androidId).toString(16)And it worked! I was abled to receive my first notification! 🎉 🎉 🎉
Decrypt notifications
As I said earlier, when you’re registering to FCM, you need to generate a ECDH key pair and to send the public key and an auth secret so that notifications can be encrypted.
Encryption and decryption of FCM notifications are following this IETF spec and use Encryped Content-Encoding for HTTP.
The npm package http_ece implements several versions of Encrypted Content-Encoding, which makes it trivial to decrypt FCM notifications.
A decrypted Hello World notification:
{
"from": "60070000000",
"collapse_key": "do_not_collapse",
"notification": {
"title": "Hello world",
"body": "Test"
}
}
Send them all
Almost done. My last issue was that every time the MCS client would reconnect to the mtalk server, it would receive all the notifications ever sent, even the already received ones.
From what I understand, the MCS client has to remember all the notification ids it received and send them on the login request to avoid receiving duplicates. Those ids are called persistentIds.
I tried sending all of them or only the last one but no matter what I tried, I ended up being kicked out again 😞.
What’s odd is that when proxying Chrome requests, I saw that it wasn’t always sending all the persistentIds: sometimes none and sometimes only one. And as a result, it was receiving all the notifications back but were just ignoring them.
Remember, those 4 bytes I spoke about earlier? Well, I saw that one of those 4 bytes was changing depending on the persistentIds length and values. This part needs more investigation.
But for now, I just skip them if they have already been received.
push-receiver, a Push Service for Node
Introducing push-receiver! 🎊 The result of this journey inside the Chromium’s source code.
push-receiver is a library which brings push notifications to Node. It handles FCM/GCM registration & notifications reception.
As the goal of this journey was, afterall, bringing Web Push support to Electron, I’m also introducing electron-push-receiver which is a convenient wrapper around push-receiver. It handles auto registration and credentials storage so that the only thing you have to do is to listen for incoming notifications.
End of journey?
This journey made me learn of lot about Chromium & Push notifications. It was one of the most challenging projects I have worked on with a lot of problems to overcome.
This is not the end as there’s still work to do. I’m looking for early testers and contributors for both projects, as they are still in an early development stage.
Examples of things that can be done:
- A better persitentIds management
- Error management : registration error/handle socket closing etc…
- Babel build: push-receiver uses async/await so it supports only Node v7.6+ for now
- Mirror browser API: electron-push-receiver uses ipc communication but we could abstract this away by providing a API closer to Web Push’s.
- Handle FCM token update/expiration
- and more…
Feel free to contribute! 😄
DISCLAIMER: this post only reflects my understanding of the Web Push client built in Chromium as a non Chromium developer and often contains simplifications to avoid boring explanations. It doesn’t intend to provide a proper and in-depth presentation of Chromium Web Push client.
