Building a Serverless Podcast Search Engine on AWS — Intro (Part 1)

System Architecture (initial version)

update (MVP is done): https://www.BrainyNinja.com

But why??

I’ve been pondering lately on what is the best way to demonstrate the power of AWS. One idea that came to mind is to create a sample application that uses a bunch of AWS services and to walk through/document the creation process from start to finish.

So then I started thinking: “what do I want to architect and develop?” The one itch that I always wanted to scratch is to create a really fast and reliable podcast search engine. I’m a night-owl and have a lot of free time after I finished binge-watching Dark and Carnival Row, so I decided to work on that idea and bring it to fruition on my free time. I will be doing this in my own AWS account, hence paying for this myself. At the end of the project I will showcase my total bill for this app and go into right-sizing my Elasticsearch cluster instances, choosing the right Lambda and Fargate parameters, and evaluating which microservices I want to consolidate, at least for now.

I will walk you through my thought process, architecture designs, and code (I will open-source a lot of it). Feel free to ask questions and participate in the process — this is meant as a learning exercise for you, so don’t hold off.

If I were working on it full-time, and not at night, it would take me about a month to complete. Since I’ll be doing this part-time it’ll take a bit longer, but I think I should be able to finish most of it in 2 months. Each blog post will go over what I did or what I plan on doing.

System Architecture

Although the architecture will change (it always does) I wanted to create a very rough-draft of it. I didn’t spend too much time on the smaller details (took about 10 minutes to come up with it), but I wanted to demonstrate the big picture here. As you can see there are about 12 AWS services that are involved as of now (more will be added later). I will review why I chose these services and what you can substitute them with; there’s always at least 3 different ways of achieving the same goal. I will go over the architecture and the other technical components in the next blog post which should come out shortly after this one.

Initial business requirements/use cases

This is a very short list, for now. I’ll leave it at 10 use cases, since each one of them is more of a story that expands into multiple subsequent use cases

1. Users should be able to search for podcasts using any keyword that comes to mind
2. The podcast results should showcase not only podcasts themselves, but events and people (both famous and not)
3. The system should allow the admin to add an RSS feed for a podcast. This in turn should result in automatic retrieval of new episodes
4. Users should have the ability to leave comments on each podcast episode
5. Users should have the ability to “favorite” podcasts
6. Users should have the ability to save podcast episodes for future review
7. Users should have the ability to browse episodes for each podcast in the system (list them)
8. Users should have the ability to browse podcasts by category
9. Users should have the ability to get recommendations from friends using the app
10. Users should be able to view the app in a browser and on a mobile device

Conclusion

In the next blog post I will go over the tech stack considerations and the system architecture that I’ve chosen to implement. I will dive deeper into alternate options for a lot of those components.