The Microservice Dream and Connecting SendGrid Events to Mixpanel with Serverless and AWS Lambda

tl;dr Microservices are cool and this repo can help you setup a Node.js/Serverless/AWS Lambda service that sends SendGrid webhook data to Mixpanel.

I daydream about microservices. As Airbo grows, and along with it, our technical debt, I sit at my desk imagining some wonderful future where everything isn’t coupled so tightly.

I wake from these dreams with two conclusions:

  1. Microservices are not the panacea I dream about. Our pathway out of technical debt requires a deliberate approach to refactoring as well as a more accountable culture of code quality.
  2. Microservices will play a role in this endeavor. But, to get started there must be a cultural shift in the way we have historically approached the development of new features. The shift is simple and likely sounds likely sounds like “Dev Team 101” for some. We must always ask the questions:
“Should this be part of the core application?”
AND
“Are we comfortable with this taking longer than it would if we built it in the core application?”

Recently, I was able to take a small (and reactionary) step towards creating a culture where we ask these questions. Here’s the context:

  • Product: “We need to track SendGrid Events in Mixpanel ASAP”
  • Dev: “No problem”
  • Deploy.
  • Dev: “Scount APM is getting rate limited, should we upgrade? Wait, SHOULD this even be part of the core application?”

No, it shouldn’t. Let’s write a (micro) microservice 🎉 🏖.

What We Are Building

An AWS Lambda function, written in Node.js and deployed with Serverless, that takes the data sent by SendGrid’s Events Webhook and pings Mixpanel.

Getting Started

To get started writing our service, you’ll need the Serverless Framework installed as well as Yarn or NPM. You’ll also need your environment configured with AWS credentials.

Creating the Service

Let’s start by creating a new directory with a package.json file:

mkdir sendgrid-webhook-to-mixpanel-lambda
cd sendgrid-webhook-to-mixpanel-lambda
yarn init

Then, let’s add our dependencies. We’ll need mixpanel and serverless-http:

yarn add mixpanel
yarn add serverless-http

The mixpanel package is the standard Mixpanel Node.js API. The serverless-http package is some middleware that manages the connection between our Node.js application and API Gateway.

With our libraries installed, let’s create an index.js file that has our application code:

Here, we create the Lambda function that will get called when our Lambda endpoint is hit by the SendGrid Event Webhook. We parse the event.body, which contains and array of SendGrid events. Each event includes metadata about the event as well as any custom attributes you can configured in your SendGrid implementation.

Then, we iterate through each event and use the Mixpanel Node.js library to track each event, specifying the event name and all the associated data. This example also increments the number of events tracked for the response.

Lastly, we create the response and call the callback parameter, where we specify there was no error with null and return the response.

Deploying Our Service

To get our microservice deployed, we need to do a few things.

First, we need a place to store our ENV vars that won’t accidentally get pushed to GitHub. Create a file called serverless.env.yml and add the following information:

staging:
MIXPANEL_TOKEN: "STAGING TOKEN"
prod:
MIXPANEL_TOKEN: "PROD TOKEN"
whatever_enviroment:
MIXPANEL_TOKEN: "WHATEVER TOKEN"

Now add a .gitignore that includes this file (as well as node_modules).

Next, let’s create a serverless.yml:

Here, we tell Serverless the information it needs to know about our AWS credentials as well as any environment variables it needs to send to AWS. Additionally, we tell AWS Lambda what function it should run (index.handler ), the endpoint it should use, and the HTTP method it should use.

Now, we can deploy our function:

sls deploy --stage prod

After it deploys, we should be able to see our function in the AWS Lambda console and we can test it out via your SendGrid Event Webhook configuration GUI. In the SendGrid Event Webhook configuration, we simply paste in our Lambda function’s endpoint as the HTTP Post URL.

Testing the Service

In order to test our service, we’ll need a few more dependencies. Here’s the full list of dependencies used:

I won’t go into too much detail on how to test the service, but I do want to point out the serverless-offline is a great package for working with Serverless in your development environment and lambda-local was super useful for testing this function. Using these packages along with Mocha/Chai and Sinon makes it easy to ship a fully tested microservice that connects SendGrid Events Webhook data to Mixpanel. The full test suite can be found here.

Conclusion

Our pathway out of technical debt requires a deliberate approach to refactoring as well as a more accountable culture of code quality, but this will only happen if we take small steps in the right direction.

By questioning whether a piece of code belonged in our core application, we not only saved money (by not having to upgrade our Scout APM integration), but also took a step in the right direction towards a culture of code quality.

This is the full repo that can help you setup a Node.js/Serverless/AWS Lambda service that sends SendGrid Events webhook data to Mixpanel.

*Some familiar with Scout APM might point out that we could have simply excluded that endpoint from tracking, but what’s the fun in that 🙃