How to build your own serverless URL shortener service

Mohamed Barakat
Feb 1 · 6 min read
Image for post
Image for post
Photo by Wicked Monday on Unsplash

You will never have a solution before having the problem.
But we’ll try to tell you before actually facing the problem :]

The problem

In September 2020 we exceeded our monthly bit.ly rate limit:
We checked the stats and had created 6905 links so far.

We used Bit.ly to shorten URLs for our different reminders and notifications
during the selling process, we have many notifications to be sent
for collecting data and informing about a specific state,
we send those notifications as emails and SMSs
so we went for bitly mainly because of SMS restrictions,
but recently we started using it for emails as well.
This was because, in Mandrill, long links can cause issues appending tracking parameters correctly, leading to incorrect tracking statistics.
Exceeding our quota caused all the services to rely on shortening via Bit.ly
in a way or another to fail, which means:

  • 248 failed requests.
  • 29 users got affected on time.
  • 28 users got an error message (failed request) and couldn’t retry it, because the endpoint was constantly returning 500.
  • We got approx. 100 sentry issues reported. Given that we retry 3 times, so it's around ~40 mails/SMS that could not be sent until then.

A simply upgrade to another bit.ly plan was not really reasonable as it would only provide us with 3k additional links for 199$/month!
While Bitly’s enterprise plan could be a solution, it starts at $15,000 per year!

Image for post
Image for post
Source: https://www.kiplinger.com/slideshow/spending/t050-s000-ways-to-stop-wasting-money/index.html

Alternatives?

We discussed the issue with our Engineering Managers,
Yuri Veremeyenko, Damir Mukimov, and agreed to search for alternatives as we were growing fast and paying too much money for the service.

A SaaS alternative to bit.ly that we also considered was Rebrand.ly.
But it also was too expensive for our use case,
specifically considering what kind of features we actually need.

Upgrading Bitly was not helpful, it was only 3k links for 199$/month! So we hot-swapped over to rebrandly as a temporary solution still using bitly as a fallback. But monitoring the increased shortening needs we realized the extended quota needed to be increased several times during a months period

In the end, we decided to build our own URL shortener service.

When?

Every quarter at Homeday we have something called PEW.
PEW stands for
Product Engineering Week, which translates to a week where you cancel all your meetings and work on a topic you would like to explore.
It can be done in groups or alone. It is totally up to you.

I had a technical meeting with our Lead Engineers, Carsten Wirth and Ilyas Amezouar, we decided to build our in-house service to shorten URLs,
during the PEW, using a serverless approach.

Why build a service like that in-house?

  • Less expensive.
  • More adjustable.
  • More scalable in regards to our needs.
  • No third-party SaaS being used.
  • You can have your own branded link, without paying extra for it.
  • Our Engineering kitchen is full of skilled chefs 👩‍🍳👨‍🍳.

Things to think about before building your own service

  • Cost of maintenance and team capacity.
  • Things that you might not think about and come out of the box –
    we had a similar case that will be mentioned later (the check for already existing URLs was not threadsafe).
  • The effort in terms of time, requirements, maintenance, and usage.

Why serverless?

Before you go for a serverless solution, you should think about the output from this as well as the limitations. It's not just a modern technology that you simply want to try.

For sure everything has limitations, but in our case, it was more than helpful and gave us what we need and more.

So, based on what you need, you can decide to go for it or not.

From a technical perspective:

  • It scales with demand automatically.
  • It eliminates server maintenance.
  • No backend infrastructure to be responsible for.

For sure you will have some negatives:

  • Cold starts for lambda
  • The costs automatically scale too
  • Having to support the solution by ourselves.

but again, as we mentioned, the requirements define the pros and cons.

From a business perspective:

  • Reduced operational costs.
  • Smaller deployable units result in faster delivery of features.
  • Increasing the ability to adapt to change.

Time for the technical part 👨‍🚀 🚀

Developed with ❤ : by Homeday

Image for post
Image for post
Source: https://twitter.com/eisabai/status/998514987827122176

We are planning to launch a technical series for this service which should be detailed technical learnings but we aim to do it in different articles, so it’s nicer and independent.

The project was very simple but very valuable for us.
Our main cloud provider is AWS. The project consists of 2 public
and 1 internal Lambda function.

One is called shorten and is used to shorten long URLs (it will use the internal function verifyToken to check if the caller is allowed to consume the function based on the passed Authentication Header).

The other one is called resolve and it's used to resolve a shortened URL
and redirect the user to the original destination.

How does it work?

Known URLs

We will check for already known URLs in the shorten function (second index used on DynamoDB).
On a match, we will return the shortUrl with status 200.
When we create a new entry the shortUrl will be returned with status 201.

Note: This existence check is not threadsafe. So in case of a race condition we might end up with the same longUrl multiple times.
But since this is rather for protecting against flooding the DB with too many entries it won’t affect the service. In this case, one can retrieve the same longUrl with different shortIds.

Known ShortID
Will respond with a 301 redirect to the stored long URL.

Unknown ShortID
Will respond with a 301 redirect to a static default Url
(currently https://www.homeday.de/de/faq/#footer)

Why, not the 404 status code?
Mainly to send the customer to a page he can get help from.
Since this is customer-facing mainly a 404 would not help here much
(in contrast to an API consumed by a machine)

Image for post
Image for post

Our service is now up and running, with zero errors(the 1 you see is for testing).
Having generated ~7K links in a little more than 20 days.

aws dynamodb scan --table-name url-shortener-prod --select "COUNT" --filter-expression 'resolveCount > :count' --expression-attribute-values '{":count":{"N":"0"}}'
{
"Count": 975,
"ScannedCount": 7093,
"ConsumedCapacity": null
}

Guess what?

Still the overall costs with all AWS services involved in providing the new functionality is under 100$ a month.
And this although there are quite more heavy lambda apps in Homeday.

Image for post
Image for post
Costs for Lambda, DynamoDB, KMS, API Gateway in December

Congratulations, you have your own service 🎉

Next steps:

  • Since we do not expect any frequent API changes for this service,
    we stick to the minimal requirements so far shortening and resolving links.
  • The first next step and new challenges will be about tracking
    and analyzing the data, eg. clicks/visits/source, etc…

Learnings (stay tuned for the technical series) :

  • Secrets using KMS.
  • JWT token verification for post methods.
  • Local testing with DynamoDB and serverless offline
    (skipping auth with noAuth).
  • Using Global Secondary Indexes in DynamoDB.
  • Handler functions folder structure (organizing the project).
  • Testing with jest.
  • Package optimization.
  • Dynamic hosting URL configuration.
  • Set up CI/CD with Github actions.
  • Collision detections and shortId regeneration retries using Nano ID.
  • DynamoDB Time to Live (TTL) for auto expiring keys ( after 6 months )
    (The TTL feature can be disabled on DynamoDB level in the AWS console).

Do you want to know more technical details?

Follow us on Medium @Homeday for technical series about different topics or join our amazing engineering department at Homeday

Reviewers:

Vinicius Kiatkoski Neves , Damir Mukimov

Homeday

HOMEDAY – The next generation estate agent.

Thanks to Vinicius Kiatkoski Neves and Damir Mukimov

Mohamed Barakat

Written by

Homeday

Homeday

HOMEDAY is the next generation estate agent. Easier. Transparent. Digital. And cheaper. Because we believe: Good brokerage does not have to be expensive.

Mohamed Barakat

Written by

Homeday

Homeday

HOMEDAY is the next generation estate agent. Easier. Transparent. Digital. And cheaper. Because we believe: Good brokerage does not have to be expensive.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store