Realm Logs: Alert Logging System with Twilio

Motivation

This tutorial is focused on those developers who have a Realm application and want to create an alert monitoring system that sends an SMS every time a new error appears in our application.

Currently, there are three ways to access the logs of our system:

1. Through the Logs section of our graphical interface: From here we will be able to access the logs and we will be able to filter between the different origins, dates, etc.
2. Using the Admin API Rest: this will allow us to request the logs of an application programmatically by accessing the logging endpoints of the Realm Admin API.
3. Through the command interface tools or realm-cli

Example of Log view in Realm

Example of Admin API

In this tutorial, we will create a monitoring system that will use a schedule trigger to monitor the Logs through the Admin API and when it detects a new error, it will send a message to a registered phone number to receive the error and be alerted of it using Twilio.

Let’s assume that you have already created a MongoDB Atlas account.

Prerequisites

1. Have already created a MongoDB Atlas account and an associated cluster. https://www.mongodb.com/cloud/atlas/register
2. Have a Twilio account. https://www.twilio.com/

Overview

The system is composed of several parts that we will explain below:

System design

1. Schedule Trigger: This trigger will be in charge of monitoring our Admin API to request the error logs. Here it is important to define the monitoring time so that this time will be the delay that we will have between a new error that arises in our application and the alert message that we will receive. For this tutorial, the monitoring is indicated every 10 minutes.
2. MongoDB collection for monitoring: We have created a new database called “Logging” where we will store the three collections we need for our alert monitoring system.
   2.1. Control collection: We will use this collection to store the access_token and the refresh_token needed to make the requests to our Admin API. The access_token is valid for half an hour, so the refresh_token will be automatically used to generate a new token every time it expires and it will be stored in this collection. We will be able to consult the last_modified property to consult the date of the last access_token renewal request.
   2.2. Error Collection: We will use this collection to store any error derived from the requests to our Admin API.
   2.3. Alert Collection: We will use this collection to store the last error log of our application that we have not yet processed and sent via SMS to our phone.
3. Database Trigger: The function of this trigger is to monitor the Alert collection, every time a new document is introduced in this collection, this trigger will be in charge of sending an alert to the phone numbers configured through Twilio.

Enable Realm CLI

In order to follow this tutorial, a repository has been created that will be used to implement this alert system in your own Realm application. To do this, we will need to enable Realm CLI.

We will have to follow the steps described here:

https://docs.mongodb.com/realm/deploy/realm-cli-reference/ to install and be able to use Realm CLI in your operating system. My recommendation is to install version 2.

Once installed, we can check the version we have with the command:

realm-cli --version

Create a Realm application

We will create a Realm application in our MongoDB Atlas account for our alert system.

Screenshot of the create new Realm App widget

Generate an API Key

In order to use realm-cli we must log in with the command:

realm-cli login --api-key="<my api key>" --private-api-key="<my private api key>"

To generate a public key and private key, we will have to access the project options, then Access Manager, and in API Keys

Create an API Key for realm-cli

We will create a new public and private key with the necessary permissions, which in this case will be Project Owner’s.

Create an API Key for realm-cli

We must copy both the public key and the private key, the public key can be consulted at any time, but the private key can only be accessed once on this screen. We must also add the IP addresses that will have access to using these keys.

The next step to verify that everything has been done correctly will be to log in using these keys, to do this we will use the command described above, the result should be similar to this:

Terminal output of realm-cli login command

Download our application using Realm-CLI

Once we have logged in, the next step will be to download our Realm application in a directory in our local machine.

To do this we can follow step 4 indicated in the Deployments -> Export App section.

Screenshot of Export App section in Realm UI

This step would be to execute the following command:

realm-cli pull --remote <<APP_ID>>

This command creates a directory at the given path and exports the application configuration into the new directory. If a file or directory already exists at the specified path, the export will fail.

Download the template project from GitHub

The next step will be to download the project https://github.com/josmanperez/realmAlertSystem in our system. This will serve as a template to initialize our alert system.

Once downloaded, we will have to copy all its content inside the directory where we previously downloaded the Realm application is located using realm-cli pull.

We will be able to observe that the structure is similar since both projects were created using version 2 of realm-cli. After copying and pasting, the content will be replaced keeping the same structure, we will have to use replace if we do it from a Mac.

An important thing to consider

1. When copying the downloaded GitHub project files to our Realm application, we must be careful not to delete the config.json file located in the /data_sources/mongodb-atlas/directory because the downloaded template project does not have it and copying and pasting this file may delete it.
2. The name of the folder /data_sources/<<name>>will match the name of our Linked Data Sources Service Name. In the template project, this name is mongodb-atlas but if in your project the name is different, please update it.

Update the Realm application with the required data

In order to be able to use our application from the template created above, we need to create several files first. This is because the Twilio API access data or the phone numbers needed to send the SMS are not available in the GitHub project.

Secrets

• private_key: We must create a secret that will store our private key to be able to request an access_token to our Admin API. At this point, we can create a new API Key, following the steps described above in the section “Generate an API Key” or we can use the same API Key that we have generated for Realm CLI.
  To generate the next secret, we can create it directly from the graphical interface in the Value section of our Realm application or we can use realm-cli for this purpose.
  The command to create this secret with realm-cli will be the following:

realm-cli secret create --app=<APP_ID> --name=private_key --value=<PRIVATE_KEY>

• Auth_Token_Twilio: We must create another secret with the name Auth_Token_Twilio, where we will store the Auth token that Twilio provides us when we create an account in their platform.
  For them we must execute the following command:

realm-cli secret create --app=<APP_ID> --name=Auth_Token_Twilio --value=<Auth token>

Screenshot of Twilio Dashboard

Values

Twilio_Numbers.json

We must overwrite the value called Twilio_Numbers.json that has the following format:

{
    "from_secret": false,
    "name": "Twilio_Numbers",
    "value": {
        "from": "<<Twilio number>>",
        "to": "<<Number>>"
    }
}

To do this we will do it from the values/ directory in our Realm application using our favorite text editor, i.e.

vim values/Twilio_Numbers.json

We must replace the value <<Twilio number>>> with our phone number provided by Twilio to send SMS and the string <<Number>>, with the target value of the phone we want to receive the alert.

Once finished, we will save the new document.

Realm_App.json

In this file, we will store the data corresponding to the application we want to monitor.

We need two fields: group_id and app_id. Both fields can be found in the URL of the Realm application.

To create them, we will do it again in the location of the values

vim values/Realm_App.json

The format will be the following:

{
    "from_secret": false,
    "name": "Realm_App",
    "value": {
        "appId": "<<app_id>>",
        "groupId": "<<group_id>>"
    }
}

In this we will establish the following:

• The app_id corresponds to the id of the Realm application we are going to monitor.
• The group_id or project_id corresponds to the id of the project where our Realm application is located.

Twilio_SID_Value.json:

In order to send messages using our Twilio account, we must add the SID provided by Twilio as a value. For this, we will replace <<Twilio SID value>> located in “values/Twilio_SID_Value.json” with our own.

{
    "from_secret": false,
    "name": "Twilio_SID_Value",
    "value": {
        "sid": "<<Twilio SID value>>"
    }
}

public_key.json

In the previous section, we have created a secret “private_key” that has associated with it the private key needed to make requests to the Admin API.

However, the template project has a value, public_key, that we must replace with its real value. To do this, we can access our favorite text editor “values/public_key.json”.

Once here, we will replace the value associated with “value” with ours.

{
    "name": "public_key",
    "value": "<<public key>>",
    "from_secret": false
}

Publish changes to our application

In the section “Download the project from GitHub”, we have downloaded the template from the GitHub repository and we have copied and pasted the folders in our own Realm project.

After creating in the previous section the necessary dependencies to be able to publish the changes, we must then publish the changes, for them we must use this command in the directory where we have our application.

realm-cli push --include-dependencies --remote <APP_ID>

When we execute the command, the terminal will ask us if we want to accept the changes, we will press the “y” key and it will continue publishing the changes.

In the end, the terminal will display information similar to the following:

Screenshot of terminal output after realm-cli push command

Things to consider

One thing worth mentioning is that to make Twilio API calls, we are using the lazyLoading option in the constructor. This ensures faster execution time and is highly recommended when interacting with Twilio.

const client = require('twilio')(accountSid, authToken, {    lazyLoading: true  });

Wrapping up

After performing this action the application will send us a text message every time a new error is registered. The mode of operation will depend on the time that we have the schedule trigger since it is this who monitors the new errors that have occurred in the app_id that we have defined. By default, it checks this operation every 2 minutes. It can be changed without problem directly from the interface in the Triggers section by accessing the trigger with the name: “MonitorLogging”.

If several errors are registered in a time interval where the monitoring trigger is not active, we will only receive the last error. If no new errors appear, the application will not send anything to the user.

The message received will have the following format:

SMS received by the Alert System