In this post we will talk about how we were able to reduce the amount of time it takes to build client libraries for our Java HTTP services. We will introduce the connection library we built, the motivation behind it and cover some of its main features.
Some benefits of a microservice oriented architecture are how the smaller codebases make microservices easier to scale, debug, and understand, which also makes refactors and component upgrades easier. One disadvantage to this type of architecture is the upfront cost it takes to spin up a new service. Prioritizing architectural changes and refactors at…
Giannis Neokleous, Paul Sastrasinh
Maintaining a large distributed service-oriented platform is hard. As both the organization and the architecture grows, and more features are added, it becomes increasingly harder to keep a mental picture of all the services and pockets of knowledge that start forming across the organization. Existing, manually drawn architecture diagrams become stale quickly, which makes it difficult for new employees to spin up on all the services. It also makes auditing service endpoints for security purposes impossible since our security engineers are unfamiliar with the codebases. To address these issues, we at Knewton have been building Kizceral…
Written by Giannis Neokleous
The previous blog post talked about why Knewton needed a distributed tracing system and the value it can add to a company. This section will go into more technical detail as to how we implemented our distributed tracing solution.
General Architecture and Tracing Data Management
Our solution has two main parts: the tracing library that all services integrate with, and a place to store and visualize the tracing data. We chose Zipkin, a scalable open-source tracing framework developed at Twitter, for storing and visualizing the tracing data. Zipkin is usually paired with Finagle, but as mentioned in Part I, we ruled it…
Written by Giannis Neokleous
Here at Knewton we use Cassandra for storing a variety of data. Since we follow a service-oriented architecture, many of our internal services are backed by their own data store. Some of the types of data we store include student events, recommendations, course graphs, and parameters for models. Many of these services and clusters are often deployed in more than two environments, increasing the total number of Cassandra clusters deployed.
On the Analytics team at Knewton we are constantly working on improving a lot of the inferential models that go into our platform, while at the…
