The Microservice Dream and Connecting SendGrid Events to Mixpanel with Serverless and AWS Lambda
tl;dr Microservices are cool and this repo can help you setup a Node.js/Serverless/AWS Lambda service that sends SendGrid webhook data to Mixpanel.
I daydream about microservices. As Airbo grows, and along with it, our technical debt, I sit at my desk imagining some wonderful future where everything isn’t coupled so tightly.
I wake from these dreams with two conclusions:
- Microservices are not the panacea I dream about. Our pathway out of technical debt requires a deliberate approach to refactoring as well as a more accountable culture of code quality.
- Microservices will play a role in this endeavor. But, to get started there must be a cultural shift in the way we have historically approached the development of new features. The shift is simple and likely sounds likely sounds like “Dev Team 101” for some. We must always ask the questions:
“Should this be part of the core application?”
“Are we comfortable with this taking longer than it would if we built it in the core application?”
Recently, I was able to take a small (and reactionary) step towards creating a culture where we ask these questions. Here’s the context:
- Product: “We need to track SendGrid Events in Mixpanel ASAP”
- Dev: “No problem”
- Dev: “Scount APM is getting rate limited, should we upgrade? Wait, SHOULD this even be part of the core application?”
No, it shouldn’t. Let’s write a (micro) microservice 🎉 🏖.
What We Are Building
Creating the Service
Let’s start by creating a new directory with a
Then, let’s add our dependencies. We’ll need
yarn add mixpanel
yarn add serverless-http
With our libraries installed, let’s create an
index.js file that has our application code:
Here, we create the Lambda function that will get called when our Lambda endpoint is hit by the SendGrid Event Webhook. We parse the
event.body, which contains and array of SendGrid events. Each event includes metadata about the event as well as any custom attributes you can configured in your SendGrid implementation.
Then, we iterate through each event and use the Mixpanel Node.js library to track each event, specifying the event name and all the associated data. This example also increments the number of events tracked for the response.
Lastly, we create the response and call the callback parameter, where we specify there was no error with
null and return the
Deploying Our Service
To get our microservice deployed, we need to do a few things.
First, we need a place to store our ENV vars that won’t accidentally get pushed to GitHub. Create a file called
serverless.env.yml and add the following information:
MIXPANEL_TOKEN: "STAGING TOKEN"
MIXPANEL_TOKEN: "PROD TOKEN"
MIXPANEL_TOKEN: "WHATEVER TOKEN"
Now add a
.gitignore that includes this file (as well as
Next, let’s create a
Here, we tell Serverless the information it needs to know about our AWS credentials as well as any environment variables it needs to send to AWS. Additionally, we tell AWS Lambda what function it should run (
index.handler ), the endpoint it should use, and the HTTP method it should use.
Now, we can deploy our function:
sls deploy --stage prod
After it deploys, we should be able to see our function in the AWS Lambda console and we can test it out via your SendGrid Event Webhook configuration GUI. In the SendGrid Event Webhook configuration, we simply paste in our Lambda function’s endpoint as the
HTTP Post URL.
Testing the Service
In order to test our service, we’ll need a few more dependencies. Here’s the full list of dependencies used:
I won’t go into too much detail on how to test the service, but I do want to point out the
serverless-offline is a great package for working with Serverless in your development environment and
lambda-local was super useful for testing this function. Using these packages along with Mocha/Chai and Sinon makes it easy to ship a fully tested microservice that connects SendGrid Events Webhook data to Mixpanel. The full test suite can be found here.
Our pathway out of technical debt requires a deliberate approach to refactoring as well as a more accountable culture of code quality, but this will only happen if we take small steps in the right direction.
By questioning whether a piece of code belonged in our core application, we not only saved money (by not having to upgrade our Scout APM integration), but also took a step in the right direction towards a culture of code quality.
This is the full repo that can help you setup a Node.js/Serverless/AWS Lambda service that sends SendGrid Events webhook data to Mixpanel.
*Some familiar with Scout APM might point out that we could have simply excluded that endpoint from tracking, but what’s the fun in that 🙃