How to build your own serverless URL shortener service
You will never have a solution before having the problem.
But we’ll try to tell you before actually facing the problem :]
The problem
In September 2020 we exceeded our monthly bit.ly rate limit:
We checked the stats and had created 6905 links so far.
We used Bit.ly to shorten URLs for our different reminders and notifications
during the selling process, we have many notifications to be sent
for collecting data and informing about a specific state,
we send those notifications as emails and SMSs
so we went for bitly mainly because of SMS restrictions,
but recently we started using it for emails as well.
This was because, in Mandrill, long links can cause issues appending tracking parameters correctly, leading to incorrect tracking statistics.
Exceeding our quota caused all the services to rely on shortening via Bit.ly
in a way or another to fail, which means:
- 248 failed requests.
- 29 users got affected on time.
- 28 users got an error message (failed request) and couldn’t retry it, because the endpoint was constantly returning
500. - We got approx. 100 sentry issues reported. Given that we retry 3 times, so it's around ~40 mails/SMS that could not be sent until then.
A simply upgrade to another bit.ly plan was not really reasonable as it would only provide us with 3k additional links for 199$/month!
While Bitly’s enterprise plan could be a solution, it starts at $15,000 per year!
Alternatives?
We discussed the issue with our Engineering Managers,
Yuri Veremeyenko, Damir Mukimov, and agreed to search for alternatives as we were growing fast and paying too much money for the service.
A SaaS alternative to bit.ly that we also considered was Rebrand.ly.
But it also was too expensive for our use case,
specifically considering what kind of features we actually need.
Upgrading Bitly was not helpful, it was only 3k links for 199$/month! So we hot-swapped over to rebrandly as a temporary solution still using bitly as a fallback. But monitoring the increased shortening needs we realized the extended quota needed to be increased several times during a months period
In the end, we decided to build our own URL shortener service.
When?
Every quarter at Homeday we have something called PEW.
PEW stands for Product Engineering Week, which translates to a week where you cancel all your meetings and work on a topic you would like to explore.
It can be done in groups or alone. It is totally up to you.
I had a technical meeting with our Lead Engineers, Carsten Wirth and Ilyas Amezouar, we decided to build our in-house service to shorten URLs,
during the PEW, using a serverless approach.
Why build a service like that in-house?
- Less expensive.
- More adjustable.
- More scalable in regards to our needs.
- No third-party SaaS being used.
- You can have your own branded link, without paying extra for it.
- Our Engineering kitchen is full of skilled chefs 👩🍳👨🍳.
Things to think about before building your own service
- Cost of maintenance and team capacity.
- Things that you might not think about and come out of the box –
we had a similar case that will be mentioned later (the check for already existing URLs was not threadsafe). - The effort in terms of time, requirements, maintenance, and usage.
Why serverless?
Before you go for a serverless solution, you should think about the output from this as well as the limitations. It's not just a modern technology that you simply want to try.
For sure everything has limitations, but in our case, it was more than helpful and gave us what we need and more.
So, based on what you need, you can decide to go for it or not.
From a technical perspective:
- It scales with demand automatically.
- It eliminates server maintenance.
- No backend infrastructure to be responsible for.
For sure you will have some negatives:
- Cold starts for lambda
- The costs automatically scale too
- Having to support the solution by ourselves.
but again, as we mentioned, the requirements define the pros and cons.
From a business perspective:
- Reduced operational costs.
- Smaller deployable units result in faster delivery of features.
- Increasing the ability to adapt to change.
Time for the technical part 👨🚀 🚀
Developed with ❤ : by Homeday
We are planning to launch a technical series for this service which should be detailed technical learnings but we aim to do it in different articles, so it’s nicer and independent.
The project was very simple but very valuable for us.
Our main cloud provider is AWS. The project consists of 2 public
and 1 internal Lambda function.
One is called shorten and is used to shorten long URLs (it will use the internal function verifyToken to check if the caller is allowed to consume the function based on the passed Authentication Header).
The other one is called resolve and it's used to resolve a shortened URL
and redirect the user to the original destination.
How does it work?
Known URLs
We will check for already known URLs in the shorten function (second index used on DynamoDB).
On a match, we will return the shortUrl with status 200.
When we create a new entry the shortUrl will be returned with status 201.
Note: This existence check is not threadsafe. So in case of a race condition we might end up with the same longUrl multiple times.
But since this is rather for protecting against flooding the DB with too many entries it won’t affect the service. In this case, one can retrieve the same longUrl with different shortIds.
Known ShortID
Will respond with a 301 redirect to the stored long URL.
Unknown ShortID
Will respond with a 301 redirect to a static default Url
(currently https://www.homeday.de/de/faq/#footer)
Why, not the 404 status code?
Mainly to send the customer to a page he can get help from.
Since this is customer-facing mainly a 404 would not help here much
(in contrast to an API consumed by a machine)
Our service is now up and running, with zero errors(the 1 you see is for testing).
Having generated ~7K links in a little more than 20 days.
aws dynamodb scan --table-name url-shortener-prod --select "COUNT" --filter-expression 'resolveCount > :count' --expression-attribute-values '{":count":{"N":"0"}}'
{
"Count": 975,
"ScannedCount": 7093,
"ConsumedCapacity": null
}Guess what?
Still the overall costs with all AWS services involved in providing the new functionality is under 100$ a month.
And this although there are quite more heavy lambda apps in Homeday.
Congratulations, you have your own service 🎉
Next steps:
- Since we do not expect any frequent API changes for this service,
we stick to the minimal requirements so far shortening and resolving links. - The first next step and new challenges will be about tracking
and analyzing the data, eg. clicks/visits/source, etc…
Learnings (stay tuned for the technical series) :
- Secrets using KMS.
- JWT token verification for post methods.
- Local testing with DynamoDB and serverless offline
(skipping auth with noAuth). - Using Global Secondary Indexes in DynamoDB.
- Handler functions folder structure (organizing the project).
- Testing with jest.
- Package optimization.
- Dynamic hosting URL configuration.
- Set up CI/CD with Github actions.
- Collision detections and shortId regeneration retries using Nano ID.
- DynamoDB Time to Live (TTL) for auto expiring keys ( after 6 months )
(The TTL feature can be disabled on DynamoDB level in the AWS console).
Do you want to know more technical details?
Follow us on Medium @Homeday for technical series about different topics or join our amazing engineering department at Homeday
