Synthetic Testing/ Synthetic Monitoring

Synthetic testing is a monitoring technique where user flows are simulated to make sure that the system is working fine. These tests are scheduled at regular intervals, as a continuous check on the heartbeat and health of the system. In today’s world of micro services and micro ETL pipelines, such tests are even more valuable to trigger a flow and get the stats on the health of all of the different micro services involved. Synthetic tests can proactively catch and help resolve issues before it affects our clients and customers.

Synthetic tests can be used to test both frontend and backend systems. An example of a front end synthetic test could be loading a webpage every 10 minutes and making sure all the various UI components are loaded fine. Getting to know a failure in less than 10 minutes is such a confidence booster.

We used New Relic for triggering and monitoring synthetic tests. In this article, let us go through how we automated backend API synthetic tests to monitor our business events micro services.

Brainstorm and design the solution:

Every new event schema created needs a synthetic test created to monitor it. The synthetic test in this case, can be a simple REST POST API call that is triggered every 15 minutes. The event payload should include all the required (non nullable) attributes in the event, so the event passes validation and marches through all the micro services steps in the pipeline. The schema registry could be leveraged to store default test values for all the required attributes. These default values could be used in the synthetic tests. Any required attribute change in the schema registry needs to result in corresponding synthetic monitor change.

The synthetic events should flow through all the micro services like a normal event , so they can be traced all along and represented in dashboards and alerted in case of an issue. These synthetic test data can be routed to different tables in the very final step, so these test data doesn’t pollute the customer data.

Implementation Details:

As all our resources are in AWS Cloud, we published schema create, delete and required attribute changes to an AWS SQS queue. NerdGraph is NewRelic’s GraphQL API and it can be leveraged for our use case to create and modify NewRelic synthetic tests.

Creating Synthetic Monitor:

SQS listeners in our monitoring micro service check if the synthetic monitor for the given schema exists using NerdGraph entitySearch actor and if no monitor exists, it creates a new synthetic monitor using syntheticsCreateScriptApiMonitor mutation.

Example entitySearch request made to NerdGraph:

Example entitySearch request made to nerdgraph:
{
  actor {
    account(id: $NEWREIC_ACCOUNT_ID) {
      id
    }
    entitySearch(query: "domain IN ('SYNTH') and name IN ('$SCHEMA_NAME’)”) {
      results {
        entities {
          guid
          name
          ... on SyntheticMonitorEntityOutline {
            name
            accountId
            monitorId
            domain
          }
        }
      }
    }
  }
}

Example syntheticsCreateScriptApiMonitor mutation:

mutation {
  syntheticsCreateScriptApiMonitor(
    accountId: $ACCOUNT_ID
    monitor: {status: ENABLED, runtime: {runtimeTypeVersion: "16.10", scriptLanguage: "JAVASCRIPT", runtimeType: "NODE_API"}, name: "$SCHEMA_NAME", period: EVERY_15_MINUTES, script: "", locations: {private: {guid: "$GUID", vsePassword: "$PASS"}}}
  ) {
    errors {
      description
      type
    }
    monitor {
      id
      guid
      name
    }
  }
}

From the monitorId , monitorURL can be constructed as

https://synthetics.newrelic.com/synthetics/api/v3/monitors/$MONITORID

Editing the Synthetic Monitor Script:

To create or update the monitor script, PUT the base64 encoded payload details to the MonitorScriptURL.

Example script:

let assert = require('assert');
let timestamp = Math.round(Date.now()/1000);
var options = {
    url: "$SERVICE_FACADE_URL",
    body: 
//schema payload
`{"schemaVersion":4,"stackId":"$MONITOR_STACK","field1":"123","timestampEvent":${timestamp},"eventName":"$SCHHEMA_NAME",field2":"345"}`,
    headers: {
        'Content-Type': 'application/json'
        }
};

function monitor() {
  // trigger the event
  $http.post(options, function (e, r, healthResponse) {
    let testStatus = r.statusCode > 199 && r.statusCode < 300 ? "SUCCESS" : "FAILED"
   // post the synthetic test status , which can be used to visually surface the status in schema registry. 
    $http.post(""$SERVICE_FACADE_URL" + "api/schemas/$SCHHEMA_NAME/synthetic-test-status/" + testStatus, function () {
        assert.ok(r.statusCode > 199 && r.statusCode < 300, 'Request should succeed');
    })
  });
}

monitor();

Example Base64 encoded script payload PUT to the MonitorScriptURL.

{"scriptText":”bGV0IGFzc2VydCA9IHJlcXVpcmUoJ2Fzc2VydCcpOwpsZXQgdGltZXN0YW1wID0gTWF0aC5yb3VuZChEYXRlLm5vdygpLzEwMDApOwp2YXIgb3B0aW9ucyA9IHsKICAgIHVybDogIiRTRVJWSUNFX0ZBQ0FERV9VUkwiLAogICAgYm9keTogYHsic2NoZW1hVmVyc2lvbiI6NCwic3RhY2tJZCI6IiRNT05JVE9SX1NUQUNLIiwiZmllbGQxIjoiMTIzIiwidGltZXN0YW1wRXZlbnQiOiR7dGltZXN0YW1wfSwiZXZlbnROYW1lIjoiJFNDSEhFTUFfTkFNRSIsZmllbGQyIjoiMzQ1In1gLAogICAgaGVhZGVyczogewogICAgICAgICdDb250ZW50LVR5cGUnOiAnYXBwbGljYXRpb24vanNvbicKICAgICAgICB9Cn07CgpmdW5jdGlvbiBtb25pdG9yKCkgewogIC8vIHRyaWdnZXIgdGhlIGV2ZW50CiAgJGh0dHAucG9zdChvcHRpb25zLCBmdW5jdGlvbiAoZSwgciwgaGVhbHRoUmVzcG9uc2UpIHsKICAgIGxldCB0ZXN0U3RhdHVzID0gci5zdGF0dXNDb2RlID4gMTk5ICYmIHIuc3RhdHVzQ29kZSA8IDMwMCA/ICJTVUNDRVNTIiA6ICJGQUlMRUQiCiAgIC8vIHBvc3QgdGhlIHN5bnRoZXRpYyB0ZXN0IHN0YXR1cyAsIHdoaWNoIGNhbiBiZSB1c2VkIHRvIHZpc3VhbGx5IHN1cmZhY2UgdGhlIHN0YXR1cyBpbiBzY2hlbWEgcmVnaXN0cnkuIAogICAgJGh0dHAucG9zdCgiIiRTRVJWSUNFX0ZBQ0FERV9VUkwiICsgImFwaS9zY2hlbWFzLyRTQ0hIRU1BX05BTUUvc3ludGhldGljLXRlc3Qtc3RhdHVzLyIgKyB0ZXN0U3RhdHVzLCBmdW5jdGlvbiAoKSB7CiAgICAgICAgYXNzZXJ0Lm9rKHIuc3RhdHVzQ29kZSA+IDE5OSAmJiByLnN0YXR1c0NvZGUgPCAzMDAsICdSZXF1ZXN0IHNob3VsZCBzdWNjZWVkJyk7CiAgICB9KQogIH0pOwp9Cgptb25pdG9yKCk7Cg=="}

Deleting Synthetic Monitor:

When a schema is deleted, the synthetic test associated with it should also be deleted. This can be accomplished by calling entitySearch actor for the deleted schema name and then calling DELETE on the MonitorURL.

Example stats on a synthetic monitor — Everything’s A-OK!!!

Dashboard showing synthetic events across multiple micro services

Dashboards are also automated that show these synthetic tests across various micro services and will alert if a service wouldn’t receive the synthetic event in the expected timeframe. More details about dashboard creation in my previous article here.

Conclusion:

Synthetic tests help in monitoring services’ health . They can be created manually by scheduling CRON jobs or using various observability tools available in the market. These tools allow developers to easily configure the tests using their console or programmatically using their API. Using the NewRelic NerdGraph API to create, edit and delete the Synthetic tests have helped us scale the synthetic tests and thus monitoring of our business events.

If you’re interested in joining Ancestry, we’re hiring! Feel free to check out our careers page for more info. Also, please see our medium page to see what Ancestry is up to and read more articles like this.