Building link tracking infrastructure

Streak has millions of users sending many millions of emails a day through their Gmail account. Our users love the ability to track when someone has viewed emails they send, whether it’s a one-off email or when sending a mail merge. We’ve had a long-standing desire to also include click tracking of any links within the email.

Streak link tracking toggle UI within Gmail

Design goals

In building this, we had a number of goals for the link tracking functionality:

1. Must work all the time. We need to ensure that any system we implement for link tracking is highly reliable. It’s one thing for our code to have to retry a request to the API if there’s a transient issue, but our users will be sending emails to their contacts where we don’t have such control, and errors would reflect not only poorly on Streak but also on our users.
2. Must be fast. If you’ve ever received a marketing email that has links in it, you’ve likely clicked a link and it takes forever to go to the destination because some slow CRM system is processing the click. We don’t like building slow functionality.
3. Links must always work. Because our users are sending emails to their contacts, any tracked links in those emails should work. Forever. This means we shouldn’t tie the implementation to any one specific technology or require any database of valid links.
4. Revocable. When a Streak users sends out any email, the email goes out through their Gmail account, which has the benefit of Google’s extensive anti-spam measures, quota enforcement, best in class deliverability, and keeping all their email in one place. However, should we need to revoke a link (for example, due to malicious use), we want to be able to prevent those specific links from working. Note that this is at odds with links always working.
5. Spoof proof. Links must only work if they were generated by Streak. When the system generates a link, any modification of the link should render the link invalid. This prevents a malicious attacker from taking a valid link and modifying the destination to go somewhere else, or otherwise circumventing checks added during link generation.

Technology

Since Streak runs directly within Gmail and we are a Google Technology Partner, our infrastructure runs within GCP (Google Cloud Platform).

We considered Google Cloud Run as it seemed like a cool technology where you deploy a containerized microservice and leave it up to Google to manage. Also under consideration was Google Cloud Functions which is essentially a containerless version of Cloud Run and similar to AWS Lambda.

We eventually chose Cloudflare Workers to power our link tracking. Cloudflare boasts that their network reaches 95% of the world’s population within 50ms. Additionally, as they power some of the largest properties on the web we were comfortable that their infrastructure’s reliability meets or exceeds our own.

Code written for Workers runs directly in each of their edge locations, meaning we get fast distribution across more than 200 points of presence. Additionally, since the code is isolated from our existing infrastructure it’s not dependent on whichever technology stack we happen to use.

Constructing a link specification

When users send emails out through Streak, our system records which links are within which emails. Then, when a recipient clicks on a link, our system associates the click with the correct email. And because Streak has millions of users sending, cumulatively, millions of emails, maintaining a copy of all that data on Cloudflare or anywhere else was a non-starter. So we needed a much simpler system.

The format of the URL that we settled on looks like this:

https://somestreakdomain.ext/uniqueidentifier/destinationurl

This consists of the following parts:

1. The domain that we host the link on. For our links, we wanted to be completely transparent about the functionality and chose streaklinks.com for the domain.
2. The destination URL. Because we want the links to work regardless of platform or infrastructure, we chose a simple URL encoded version of the destination. So if you are linking to https://example.com then the encoded version is https%3A%2F%2fexample.com. While simple URL encoding does make a link easily decodable by a recipient should Streak later block the URL, the main goal is that Streak’s infrastructure isn’t directly used to link to an unwanted site.
3. This leaves the unique identifier. As mentioned, we use this to later associate which link in which email the recipient clicked on. At the same time, we don’t want to perform an expensive database lookup nor do we want a malicious attacker to be able to spoof arbitrary URLs in our system. The solution is that the identifier combines an identifier within Streak’s CRM with a cryptographic checksum of the data.

Because the complete URL contains all the data necessary to both validate the URL and redirect the user to the destination, we don’t need to maintain any storage.

Validating a link

When a recipient clicks on a link, we extract the unique identifier and destination URL from the request. Validation is then a simple matter of performing an HMAC SHA-256 calculation of the unique identifier and the destination URL, which we use to ensure the integrity and authenticity of the link.

Cloudflare has some docs on Web Crypto at https://developers.cloudflare.com/workers/runtime-apis/web-crypto/ which outlines how to calculate a message digest. In our case, this looks something like this:

const encoder = new TextEncoder();
const secretKeyData = encoder.encode(SECRET_KEY);
const key = await crypto.subtle.importKey(
  'raw',
  secretKeyData,
  {name: 'HMAC', hash: 'SHA-256'},
  false,
  ['sign']
);
const mac = await crypto.subtle.sign('HMAC', key, encoder.encode(message));

In the above code, SECRET_KEY is a secret that has been set via wrangler:

wrangler secret put SECRET_KEY

The value of this secret key is shared between Cloudflare and our API code which generated the link in the first place. The mac (Message Authentication Code) value contains an ArrayBuffer with the bytes, which we extract and use to verify that the link was generated by our API without having to contact the API or lookup any data.

Additionally, we store a blocklist of undesirable links in Cloudflare Workers KV (a key/value store) and we verify that the link has not been blocklisted. Workers makes this quite simple. In our case, we created a new KV namespace like blocklisted-links and then bound it to the worker using the name BLOCKLIST. To check if a destination URL is on the blocklist, it’s simply:

const MAX_KEY_LENGTH = 512;
const blocklistKey = ('url:' + destinationUrl).substring(0, MAX_KEY_LENGTH);
const blocklistEntry = await BLOCKLIST.get(blocklistKey);

If blocklistEntry is anything other than null, that means we’ve blocklisted the URL and we can reject the request. Note the use of the url: prefix here for checking the full URL. This allows us to use various prefixes to expand the blocklist to support any kind of blocking, whether via hostname, ip address, user-agent, and so on.

If everything checks out, the Cloudflare Worker code immediately redirects the user to the destination:

return new Response('', {
  status: 302,
  headers: {
    location: destinationUrl
  }
});

Overall request flow

Fast performance

The Cloudflare Worker code redirects requests to the destination URL in single digit milliseconds. Real world performance testing shows that link redirection can occur in approximately 20ms including making the request, processing the request inside the Cloudflare Worker code, and sending the redirection response. Most requests are so quick that the browser does not have time to display the streaklinks.com domain before the request to the destination occurs.

Behind the scenes, we take advantage of the fact that inside a Worker, fetch events (see https://developers.cloudflare.com/workers/runtime-apis/fetch-event/) run asynchronously and live beyond the lifecycle of the user request. While the email recipient’s click has already been redirected to their destination, the Worker code pushes the click data to our system where we update the Streak user’s CRM data:

event.waitUntil(postToStreak(data));

We run this code just prior to returning the redirection response. Here, data is an object that contains information about the request including unique ID, destination, timestamp, and so on. When processed, our API associates this data with the original sent email.

Launching link tracking

Soon after we launched, we saw some incredible success with the system. Cloudflare’s performance excelled at our goal of having very fast redirection and handled a massive number of requests:

Geeking out on logs

Cloudflare lets you easily live stream requests using wrangler. A simple wrangler tail -f pretty from the terminal and requests flood in, including any console logging performed within the worker.

console.error('Something went wrong!');
console.debug('Everything is good');

See https://developers.cloudflare.com/workers/learning/logging-workers/ for details. During development, this was invaluable to quickly verify details of the request and response. And once rolled out to production, streaming only errors caught a few issues where we weren’t gracefully handling invalid requests. And, when we want to see everything, we stream the debug logs as well.

In examining the logs, there were a few interesting things to note:

1. Browsers, of course, attempt to fetch a favicon.ico file. We didn’t plan to serve one initially, but it was an easy matter of serving up Streak’s favicon directly from the Worker. Similarly for people who are curious about what streaklinks.com is all about, when they visit https://streaklinks.com/ we redirect them to https://www.streak.com/
2. Anyone who has run any internet-facing system is familiar with automated exploit attempts. Soon after the domain went live, we saw all manner of malicious traffic. Fortunately, Cloudflare doesn’t break a sweat with all those requests and the Worker code trivially rejects invalid requests without effort.

As part of our commitment to overall security, we allow ethical hackers to try and discover vulnerabilities our systems via Hacker One and link tracking is fair game. More information at https://www.streak.com/security

Wrapping it up

Exploring the benefits that Cloudflare has to offer has been fun. With the new feature successfully launched to everyone, Streak users now get additional valuable insights into how recipients are interacting with their emails without having to use third party tools or leave Gmail.

Cloudflare Workers has proven to be extremely reliable and fast and our link tracking functionality has operated flawlessly. Having proven the benefits of Cloudflare Workers as part of our engineering toolkit, we now have a reference on how to build out fast, reliable functionality that integrates with our existing infrastructure.

Engineering at Streak

We work on many interesting challenges affecting millions of users and many terabytes of data. For more information, visit https://www.streak.com/careers