Technical Vision — Part 1

What has worked really well for us so far, is giving engineers ownership over their applications. It means that they become responsible for creating technical specs for new features, manage the planning of their team, do the implementation, make sure everything is working as expected when they deploy. We don’t have a QA team, which means that our engineers are responsible for the stability of what they push to production.

This level of decentralization implies that we have a flat engineering organisation. Our engineering teams consist of a team lead and 0–5 additional engineers. Each team is responsible for a clearly defined set of features of our stack. These features are driven by multiple microservices, owned by the team.

This blog is divided into three parts:

• Scaling communication between engineering teams
• Scaling our architecture
• Scaling consistency between services

Challenge 1: Scaling Communication between engineering teams

Given the number of teams we have, communication is becoming increasingly more important. Looking at some of the questions new and existing engineers ask:

• What API can I use to get the course info?
• How can I use the Multiplexer to execute R code?
• How can I authenticate a user in my new service?
• Wait we have a projector app?
• How can we use it for playing videos?
• We need to make this small change to this app of your team… Can you explain to me how it works?
• I’m new on the Learn Engineering team, where can I find out how the “IMB” works?
• I don’t understand the database structure of our payment system?

What we get is something like this:

Without a structured process, every team has to communicate with all other teams for information. This means that if we have n teams, the number of inter-team communication flows become n*(n-1)/2. For example, 6 teams gives us 15 flows. This becomes hard to scale.

Let’s take a step back and look at what communication flows seem to work well. At DataCamp we use some of the most popular open-source libraries and frameworks like React, React Native, Express, Koa, Ruby on Rails, etc. What’s striking, is that we are able to use them, without needing to have contact with the maintainers.

Why is this? Imagine trying to use React, when there is no documentation. You probably wouldn’t consider using this in production. Let’s keep this in mind and try to come up with a list of criteria that are needed to scale a library or service:

• Documentation is needed on how the project works, how to run it locally, how to run the tests, how to contribute, etc.
• Documentation on the API. Imagine if you want to use a service like Stripe, without there being any API docs.
• Have decent test coverage. This shows that the library or service will work as expected and protects against unintentional breaking changes.
• The project solves a problem in a way that is generic and extendable. If the library is too tightly coupled to solving one domain specific problem, it can work in your service, but other teams might not be able to use it to solve their specific problem.

In order for us to scale communication between engineering teams, the goal should be, that when an engineer wants to use an internal service or library, all they need to do, is read the documentation.

The next part will detail how we plan on scaling our technical architecture.