Custom Sentry Reporting

As with most organisations, we like to be informed of errors and exceptions in production. For that we use Sentry.

The vanilla Sentry configuration works quite nicely for most applications — it captures everything we send, categorises it intelligently and pushes useful alerts to Slack:

Understand the error and action accordingly

Unfortunately, some of our applications use logback and the logback Sentry integration. The Slack notifications into Slack typically lack useful information and we often find ourselves having to click into Sentry for more details.

Typically requires clicking into Sentry or going to our ELK stack to see logs

When our engineers are on the go, being able to get a sense of a problem direct from Slack makes life a lot easier, so we set out to build a better solution.

The Approach

Sentry allows you to build internal integrations that trigger webhooks for certain actions. Looking through the payload, it looked like we could extract what we needed and then push the relevant information to Slack.

To make this a reality we needed to:

• Create a service to accept a webhook and push to Slack
• Configure Slack to receive the notification
• Configure Sentry to send the webhook to the application above

In a picture:

Create a Service for the Sentry Webhook

We’ve recently started using serverless to configure lambda functions for a variety of uses from marketing data automation to a merge request reminder bot.

Gitlab has a great template for getting started with serverless and we used something similar with our project.

All we need to do is write a bit of node:

slack.js

index.js

A few notes:

• We map Sentry project ids to slack channels
• We tried to mimic the Sentry Slack message format as closely as possible
• The Slack token / Sentry secret are from the steps below
• We put Sentry in our Sentry

With the application developed, a simple command sends it up to the cloud and gives us the endpoint we need to use:

sls deploy --stage production

Configure Slack to Receive the Notification

(You’ll need admin access to Slack to do this)

There are a few approaches you can take to push messages into slack. We wanted to use the chat.postMessage endpoint and created a new application to give us the correct permissions. These steps are widely documented, but we’ll cover them quickly here.

1. Build a new app

2. Click Permissions when you see this screen:

3. Configure Scopes by adding chat:write.public

Now you can install the app into your workspace and the token you get will allow your app to write to whatever public channel it needs.

Configure Sentry to Send the Webhook to our Serverless Application

(You’ll need admin access to Sentry to do this)

From developer settings (https://sentry.io/settings/(your org name)/developer-settings/), you can add an Internal Integration.

Enter the name and details for your project (using the Serverless url with endpoint from the Serverless step above) and enable the Alert Rule Action.

You’ll need to enable Read access for Issue & Event.

Once created, you’ll provide the Client Secret from the Credentials section for the Serverless app above.

With the above configured, you can now configure the Sentry applications to send alerts to this new integration:

The Final Result

With all the above in place, we now have our own message in Slack that we can customise in whatever way we like:

If you are up for a new challenge in 2020 please check out our careers page and feel free to reach out to us directly — we are looking for capable engineers across most of our teams in all seniorities.