My Journey in KASKUS TV
Hi guys, this is an article about my contemplation about being a software engineer for KASKUS TV since it came into this world. For about two years, I learned many things about software development in general. I want to share what I did, what I regret, and what I expect going forward. All of these are based on my observations, so I may not capture everything here.
What have I done?
Come to think of it, this is the first time I contribute to a project from scratch. My past professional experience has been me joining already established projects (well, some of those projects are no longer with us; RIP). Kinda makes me feel, “So is this what starting a startup feels like?”.
When we were directed to start this project, we prepared ourselves by learning all we could think of that could be relevant. After we feel enough, we start developing KASKUS TV based on what we’ve learned back then. Along the way, we keep discovering something new every day (well, not literally).
Explaining all of what we’ve done as a team will take a lot longer than a simple article could cover. So I would focus on what I’ve done that leaves quite an impression on myself.
“Do you have any videos about Bali?”
I remember when our PM notified us when he searched “bali” in our site’s search bar. He said that there’s a video on our site that contains “bali” in its title and in description, but the first video that shows up after the search is a completely irrelevant video. I was baffled.
Turned out, that said video contained the word “kembali” in its description (I didn’t really remember, but the word was supposed to contain a substring “bali”). OK, at that time, we only utilized our database’s query using LIKE feature. Well, this was an eye-opener for me that we needed a proper search engine on our site.
When I was thinking about what we should use for our search engine, one thing came to mind: Elasticsearch. After I told the team that I wanted to try using Elasticsearch, they gave it a green light. So, I started to learn about the basics; check whether there is a library I could use (unsurprisingly, there was); integrate it with our current system; and do a weighted search (I wanted the video that contains “bali” in its title to appear higher than video that contains “bali” in its description).
After everything is settled in place, now whenever I search “bali”, I will get a list of relevant videos sorted by relevance. My only regret is that I didn’t spend more time experimenting with other similar products (e.g. Apache Solr) before deciding to go with Elasticsearch. Going forward, maybe some things that we can improve regarding search:
- Handling typo in search. Sometimes, I mistype “bali” into “baali” (and get nothing from the search result)
- Adding suggestions; My colleague suggests that when the user types “Denpasar”, perhaps it can also give a suggestion to search for “bali” instead (you know, Denpasar is a city in Bali, in case you didn’t know).
API, API, API!!
There was a time when a client of ours wanted to manage their video contents via API. Well, we have a CMS dashboard (in my opinion, a nice looking one). But I guess, they didn’t want to manage too many CMS dashboard in their day-to-day operations. So, time to build APIs!
Long story short, we finally had our API for content management (thanks to our solid architecture, adding APIs is not complicated). I built APIs for managing videos and programs, because that’s what we thought would be the most used API. I could add more API if the need arises. But for now, it’s basically “testing the water”. To secure those APIs, I used Basic authorization using token.
After all of those APIs are ready to use, then comes preparing an API docs. I was thinking,”This is an API docs for clients, surely we need to make it as beautiful as possible”. So, using Google docs is out of the question.
At that time, I tested Slate and Swagger. Both look sweet, but I decided to use Slate. Mainly because we had experiences using it before, so I could deliver the docs faster (and if I had any troubles when developing it, I could just ask my peers that had experience with Slate before).
Our API for content management is ready now (and we even have a nice API docs too). We have not considered expanding our API since there are no immediate needs right now. Should the needs arise (that is, we get clients that need more advanced APIs), we are ready to expand it.
Time to save more money
When we started this project, we didn’t know much about video delivery or video platforms. So, in order to speed up learning + development, we use third-party services to help us in delivering those VODs (Video on Demand) and live streaming to our users. After some time, our VOD numbers grow, and so does our bills. To reduce our bills, we decided to be independent from our current third party service that helps us deliver VODs and live streaming. Yeah, time to make our own services!
We start off by listing all possible alternatives for transcoding VODs and hosting a livestream. After comparing those options based on performance and price, we decided to go with AWS MediaConvert for transcoding and Wowza for live streaming. I was assigned with building the transcoding service using AWS MediaConvert.
Currently, we already had a transcoding system developed by other teams in our company. It was using AWS ElasticTranscoder. But when we opened the product site, AWS told us to use MediaConvert instead, promising that it will be more sophisticated and cheaper (I did the math, it was indeed cheaper). So, instead of spinning a new AWS server to host our newly-built transcoding service using MediaConvert, we just enhanced the existing transcoding service to support transcoding VODs using MediaConvert.
It was a challenging task for me. Because the current system is using a technology I had never used before. So before starting the task, I learnt the technology and familiarized myself with the codebase. Also, the feature I was about to implement needed a database, but the current server didn’t have it. So, I just stored the data into files (since the data is small, so I think it was safe). Although it’s good enough for now, we need to implement a proper database once the data’s growth is large enough. Funnily enough, when I was experimenting with MediaConvert, it turned out it could transcode more video formats than what Elastic Transcoder does. We discovered this when looking at some old raw VOD that failed to be transcoded, and we managed to transcode it using MediaConvert.
After building the feature, it turned out we needed more than just MediaConvert. To get status updates regarding transcoding jobs, we needed to set up AWS CloudWatch. And then, the CloudWatch instance needed to send notification to our service, so we also needed AWS SNS (we already had one, so we just needed to create a new topic for listening to the CloudWatch).
By building our own transcoding service using AWS MediaConvert, we expected to save 50% cost for transcoding. That number could be higher overall, because we could stop subscribing to our current third-party service for transcoding and delivering VODs and live streaming (that’s where most of our bills came from).
What’s next?
For over two years, I believe that our team has done a splendid job building KASKUS TV. I learned a lot about stuff I may not even be interested in learning, if not because of this project. Going forward, I’m thinking about some improvement that our team can do:
Next videos, please!
Our boss told us that the secret recipe of a currently popular video app is its ability to recommend its user the next video, after said user finished watching the current video. And not just any video, the next video is calculated by machine learning to increase the engagement of users.
We’ve been made aware of this, and are trying to implement similar features. Initially, we want each user to have their own recommendation system tailored to their own preferences. After discussing it with our AI consultant, unfortunately we need to put this idea on hold. They said that our idea needs more data than we can provide right now / we simply don’t generate enough data.
If building a personalized recommendation system is not feasible, perhaps we could tweak the idea a bit. Because, well, our main target is providing a way to recommend the next video after our user finished watching a specific video. Machine learning is just one way to achieve this. Perhaps we can hold a brainstorming session in the future with our AI consultant about this.
Laptop fans go brr..
I’m starting to hate building and running our system locally. Why? Well, the project is becoming bigger and more complex every day, with continuous addition of features and enhancements. When I’m running it locally, it starts to make my laptop slow, and make its fans go brr….
When we started this project, we decided to go monolith because, well, it’s faster to develop. Also, it was designed with modularization in mind. So if the time comes when we have to split it into smaller services, we don’t have to change the code too much. It was ok at the beginning, but I start to feel like the time we split our codebase into smaller services is coming. Why do I feel this way? Well, let’s say my laptop fan’s noise is getting louder (even though I only open the necessary browser tabs).
Serverless is one alternative we could experience. We have a service that is not frequently used, and we run it serverless. Perhaps if there are more features that we can deploy as serverless, we can save some more money on infrastructure?
How do my videos perform?
If you have a production house (you know, the one that makes video content), surely you want to know about your audience’s response to your content. If they like your content, then maybe you’re going in the right direction. But if they don’t, well, you might want to consider getting back to the drawing board. How do we know our audience’s response? One way is by processing the data.
When a user is watching a video, we can get various types of information (view count is one of those). If you have a video with a lot of view count, a simple conclusion is that your audiences like that particular video. So, making another video that shares similar traits with the last video might be worth it.
Because we are building a video platform, I think it’s necessary to integrate analytics and insight into our system. Currently, if our client wants data or insights, our data team will crunch the numbers and make an analytics report. The problem is that it’s not scalable. What if our client grew larger? Our data team will be overwhelmed with requests.
Conclusion
I think that is all I want to brag/rant about my current project. I have learnt a lot about things, and believe that there will always be something new to discover while being in this project and team. Even though there’s some regret when I’m developing features (like the one when I didn’t consider more options when choosing a search engine to implement), all those experiences helped me grow as a software engineer (and as a person, although I didn’t write about it in this article).
