How I’ve rebuilt and scaled the biggest nightlife media report company in Portugal. [Part 1]
The story of one of my early career challenges.
How I’ve rebuilt and scaled the biggest nightlife media report company in Portugal. [Part 1]
The story of one of my early career challenges.
Hello guys,
I’m posting this article to share with you how I’ve faced the challenge of rethinking all of technological side of 7Noites.
For those of you who don’t know, 7Noites.com is the biggest ( arguably ) nightlife media report company in Portugal. It has tones of gigabytes of photo reports, several profissional photographers working all nights and thousands of visits every day.
The service was already running for two years and everything seemed ok. Thing is, it wasn’t ok.
What happened ?
After two years being online, 7Noites owner contacted me troubled with some situation.
7Noites had received a notice saying they were violating their SLA (Service Level Agreement) and this lead to some problems with the company responsible for hosting it. After some chat between 7Noites owners and the company, it was decided that they had to move 7Noites elsewhere in very few days, and I was in charge of doing this transition smoothly.
Identifying the problem
So I started analyzing what lead to the violation of the SLA.
In a minute I came across that multiple slow queries to the database were being made and even worse, the database was badly designed and the files were not being placed correctly.
So beyond insecure and poorly written code, the main bottlenecks the project had were ridiculous unnormalized database schemas ( really, who keeps all kinds of images in a simple images table, from banners, to photo reports, to avatars, etc.? ) and how the files were being saved, which by the way, all images were being saved in a ‘/images’ folder.
The Challenge
- Redesign the database.
- Rewrite the project with maintainable code and improved queries.
- Restructure the way how images are placed.
- Migrate the project to another host without the service being offline.
Solution — Restructure the way how images are placed
Well, point 1 and 2 of the challenge are pretty basic and we’ll not cover them. One challenge ( from my perspective ) was to find a way to store the files in a way that could scale. In some systems it’s important to partition your files into separate folders and if you happen to be in this situation where you need to place millions of files there’s a neat solution to ‘randomly’ partition your files.
So imagine you have a file named new_photo1.png, one thing you can do is to create an MD5 hash and use it to name the location of the picture.
For instance,
md5(“new_photo1.png”) = “91256e78a1ed4f6497527933dad02294"
and the path of the picture could be,
9125/6e78/a1ed/4f64/9752/7933/dad0/2294.png
and if there are two files named new_photo1.png ? Just do this method with a timestamp in the end of each name☺
Solution — Migrate the project to another host without the service being offline
Since the owners wanted to have the files in a dedicated server and didn’t want to host the files in a CDN like Amazon S3 we had to move tones of gigabytes from one host to another.
Due to the extent regarding this solution, a future post will be made on how this was achieved, keep following!
