The front-end ecosystem has exploded in recent years. Although the “winner” at the time of this writing appears to be React, it’s not the only possible solution, and is often used in cases where it shouldn’t be.
The technology being used isn’t as important as the philosophy — there are a whole bunch of component frameworks at this point, but they all use the same basic paradigm. React generates everything from the ground up — it can’t be easily “decorated” onto an existing HTML document, but is more aimed at “componentizing” everything about the application. …
Sometimes as you go about your business of being a software engineer, a problem crops up that is unusually interesting, potentially really useful, and also seems like it should be relatively easy to do, given the age of your chosen programming language and the tools already available.
Such a situation happened recently with the introduction of a new “item indexer” project at Flipp. This was an attempt to replace a patchwork of existing systems that fetched item information from retailer websites with a single microservice that could do it for all requests.
In the microservices world, communication between services provides a host of problems — one of them being “how do we ensure that the contract between services doesn’t break as those services change?”
Communication in event-driven land usually happens through some kind of messaging system. Here at Flipp, we’ve standardized on Apache Kafka, a streaming platform with a number of advantages which have seen it play a central role in our architecture. This system will have a number of topics (also known as queues, buses, etc. in other products), each of which have a number of messages (records, events, etc.)
In my previous article, I discussed some of the advantages and disadvantages of single vs. multiple services. In this article, I’m going to assume that you went the multi-service route.
When we talk about “services”, there are actually multiple different things we can mean. (I know that these terms may not be the commonly used definitions!) In this article, I will be discussing three aspects of a service:
Here at Flipp, we use Event-Driven Microservices pretty heavily. In particular, we use Apache Kafka as our data backbone. Microservices are (from a historical perspective) a relatively new pattern in computer science. One of the main touted advantages of microservices is the ability to decrease coupling between different pieces of functionality.
“Low coupling” is a term originally used to refer to different modules or classes within a single codebase. More recently, it’s been co-opted to be used within a service-based architecture to discuss the advantages of keeping the different services independent.
Some of these advantages include:
In the previous article, I detailed the microservice pattern of using Kafka as a data backbone for your architecture — Kafka acts as the source of truth and each microservice consumes data from it. This allows each service to be independent and to own only the data it cares about. We also said that there should be some way to marry traditional relational databases like MySQL and Postgres with the Kafka data backbone with a minimum of fuss.
We left off with a couple of problems:
Apache Kafka, a distributed, asynchronous streaming platform, has exploded in popularity over the last few years. It boasts a number of advantages, including fault-tolerance, availability, reliability and scalability, and is being used by hundreds of companies ranging from tiny startups to enormous companies like PayPal and Twitter.
In this article, I’ll describe how Kafka can be used to act as the data backbone for your microservice architecture, and provide a host of advantages while being able to solve many of the inherent disadvantages that come with that pattern. I’ll also cruelly hint at an awesome open-source toolkit we built that…
Note: This is a post that originally appeared on the Flipp Engineering blog before we figured out how to use Medium Publications.
by Daniel Orner, Staff Engineer at Flipp “Some are born great, some achieve greatness, and some have greatness thrust upon ‘em.” — Malvolio, Twelfth Night II:5 So what does the term “team lead” mean to you?
Many moons ago, I joined Flipp as a senior software engineer. At the time, Flipp was still a relatively raw startup, with under 40 employees. …
Staff Engineer at Flipp