Recently I have learned a lot about what qualities make for a user friendly (and non-user friendly) client library.
Over the last 1.5 years I have been part of a team at Wix that’s developing event-driven data streaming infra on top of the Kafka message broker. The most important part of this infra is a JVM-based client library called Greyhound, which has been in continuous development for the past 5 years and was recently re-written from scratch and open-sourced.
Following are 5 tips for writing great libraries, touching on design, implementation and APIs.
Before jumping into the deep water of writing a library from scratch, first make sure there is no good open-source solution already available.
Search Github, Maven Central or NPM for current implementations. There are millions of NPM packages and JVM jars out there. But not all search hits will be a good fit. Make sure to checkout these important…
Kafka consumers and producers are used as the event-driven messaging backbone between Wix’s various microservices — 1500 in total. With such large scale, many different requirements were needed to be addressed and great tools provided for many different teams.
We talked about Greyhound, Wix’s own Kafka client wrapper library that has very useful features like easy to setup retry policies and trivial configuration of how much parallel work each client should handle.
Most of the discussion revolved around the valuable lessons Wix has learned in order to be able to support the huge scale of developers, use cases, meta-data, and traffic on its Kafka based distributed system. …
Back In January 2020, I wrote two posts (I, II) about pitfalls to avoid when starting to work with ZIO.
9 months have passed. In the interim period, ZIO 1.0 was released and ZIO Environment has been tremendously improved with the introduction of ZLayer.
From my experience it is a real joy to write code with ZIO, especially in cases where you have complex concurrent/async requirements which are much simpler to implement with the features this library provides.
Since January, I’ve written a lot more code with ZIO and had a few more mistakes I’ve made and corrected along the way. Below I gathered 5 more ZIO lessons learned. Lessons ranging from the best way to write recursions with ZIO, to the correct way to write test assertions when using TestClock. …
As promised, this is the second part of a two part series on key patterns of event-driven messaging designs that have been implemented at Wix and that have facilitated creating a robust distributed system that can easily handle increasing traffic and storage needs by more than 1400 microservices.
I recommend reading part 1 first, where I write about ‘consume and project’, ‘event-driven end to end’, and ‘in memory Key-Value stores’. Here are the next 3 patterns:
…when you need to make sure scheduled events are eventually processed
There are many cases where Wix microservices are required to execute jobs according to some schedule. …
For the past year, I’ve been part of the data-streams team that is in charge of Wix’s event-driven messaging infrastructure (on top of Kafka). This infrastructure is used by more than 1400 microservices.
During this period I’ve implemented or have witnessed implementations of several key patterns of event-driven messaging designs that have facilitated creating a robust distributed system that can easily handle increasing traffic and storage needs. 3 of these are in this post and another 3 are in part 2.
…for those very popular services that become a bottleneck
This pattern can help when you have a bottleneck legacy service that stores “popular” data of large domain objects. …
Recently, I managed to deploy several OSS artifacts built with Bazel to a Maven Central repository. As this process is quite complicated and involves many steps, I’ve decided to write a detailed step-by-step guide. I sure would have liked to have had one before I got started.
I work at Wix, as part of the data-streams team, that is in charge of all our Kafka infrastructure. Recently we started to open source our tools and libraries, starting with Greyhound — a Scala/Java High-level rich SDK for Apache Kafka.
Greyhound is built with Bazel — an open source build tool designed for speed and accuracy with truly incremental builds and a high degree of test parallelism. …
Recently, I managed to dramatically reduce the memory usage of a widely-used JVM app container on Kubernetes and save a lot of money. I figured out which JVM flags are more important, how to set them correctly and how to easily measure the impact of my actions on various parts of the app’s memory usage. Here are my Cliff notes.
The story starts with a use case. I work at Wix, as part of the data-streams team, that is in charge of all our Kafka infrastructure. Recently I was tasked with creating a Kafka client proxy for our Node.js services.
The idea was to delegate all Kafka-related actions (e.g. produce and consume) from a Node.js app to a separate JVM app. The motivation behind this is that a lot of our own infrastructure for Kafka clients is written in Scala (It’s called greyhound — the open source version can be found here).
With a sidecar, the Scala code doesn’t need to get duplicated in other languages. …
In this post I will share how I created a synchronisation primitive called
Gate that allows to potentially suspend execution by setting a boolean flag using a few lines of ZIO STM code.
At Wix, we take testing seriously. Each commonly used micro-service supplies a testkit to its users.
Recently, I wrote a testkit that mimics our Kafka-client Sidecar (separate container that communicates with Kafka and implements additional client-side features).
The testkit simulates the Kafka broker by directly putting messages-to-be-produced (that the sidecar received) into fake consumer queues for further processing.
One of the Kafka-client sidecar features is to globally pause message consumption (and then resume it). For the testkit logic I was required to implement the following…
My first ZIO post - 5 pitfalls to avoid when starting to work with ZIO was well received and I even got encouragement by some of my colleagues at Wix to share more of my beginner experiences with ZIO. So I’ve decided to write a sequel (like any current Hollywood screenwriter is easily persuaded to do).
If you are unfamiliar with pure functional programming and ZIO, you can find some useful links at the intro of my previous post
All the following pitfalls are again from my first-hand experience with writing ZIO code, stumbling a bit here and there, and making a note of it. …
For someone like me, who has spent most of his professional career with OOP (C++, C#, and lastly Scala for the last few years), the pure functional programming style in Scala felt enticing.
Because you work only with total functions, you gain a lot more confidence in your code. No more unexpected thrown exceptions. With referential transparency you gain easy refactoring and no surprising side-effects. Plus, you have local reasoning (just by looking at a function you know exactly what it does without knowledge of what happens before it).