This article is part of my, very loosely connected, series that explores different challenges of designing large, distributed system. While working on this system, I have had a lot of opportunities to learn about system design, security, and Artificial Intelligence.
You can read the previous articles here:
Our primary use case is following:
When users of our app decide to share content to their friends, one of two things can happen:
1. Friend already has our app.
In that case, content is shared to this friend in-app, and he or she gets notified via push notification. …
Let’s explore the challenges involved in building a backend system to store and retrieve high-dimensional data vectors, typical to modern systems that use “artificial intelligence” — image recognition, text comprehension, document search, music recommendations, …
In my last article, I mentioned a system that I wrote to replace a 3rd party API. The new system is composed of three components:
1. Domain logic service —”artificial intelligence”
2. Storage & retrieval service
3. Backing data-store (Redis)
Domain logic service processes input data and produces a 1024-dimensional normalized vector that represents the data. Which is then passed onto the Storage & retrieval service to be persisted.
At retrieval time, Domain logic service produces another vector from the input that is transformed using same processing logic and Storage & retrieval service is tasked with producing a set of vectors that are already persisted in the system that are most similar to the given input. …
It’s a mouthful, I know!
I am building an asynchronous distributed system that serves as a backend for a mobile app. The system does a lot of processing and communicates with couple of external components. From the start, I decided to design the system as a choreography of micro-services. But other than this basic constraint, I decided on iterative approach to the design. Starting with something simple and evolving it over time…
In this article, I will walk you through couple of challenges I had designing the system and how I choose to solve it. To me, it was interesting to see how the system started coming together and how patterns started to emerge. Because I designed the system from scratch adding layers of complexity over time, I feel I now have better grasp of some of the concepts and patterns in distributed computing and I would like to share that experience with you. This is a first article that focuses on system resiliency but I plan on writing several more, covering various aspects of the whole system. …