Technical Vision — Part 2

At DataCamp we have more than doubled our engineering team in the last 12 months. The first part of this series goes in detail on which technical changes are required for us to scale our internal communication. This part goes more in depth on how we plan to change our technical architecture.

Challenge 2: Scaling our architecture

As mentioned in part 1, we want engineers to take ownership over their services. This requires a significant amount of granularity in our architecture. (We don’t like having two teams both owning a service)

The fact that we have a “main app”, shows that we have too much logic in one place. This application is responsible for DataCamp for Groups, integrations with other learning platforms, user authentication, course info, tracking user progress, etc. This has some implications:

• When the main app goes down, most of the other microservices go down.
• If the Learn Engineering team wants to make a change to the API that the learning interface app uses, they need to make changes to the main app, which is the maintained by a different team.
• There is no uniform system for authenticating users on different applications.
• The Enterprise Engineering team depends too much on the capacity of the team maintaining the main app (Growth Engineering).

Additionally, some services like the Projector app (responsible for video playback on DataCamp) are highly coupled to the main app. We want to have a clear separation of concerns between our services.

Splitting the monolith

Our flow for extracting the campus-backend (an API that will manage all data related to taking courses) from the main app would look something like this:

1. Create a new campus-backend API in the main app. This API will interface exclusively with the database models we are extracting (Course, Chapter, Exercise, etc).
2. Split up all the logic in the main app, so it uses this internal campus-backend API, instead of calling the models directly.
3. Once all logic in the main app uses the new API the controller logic is split.
4. The new campus-backend should be well documented and end-to-end tested, so it can be refactored.
5. We duplicate the main app, deploy it, create a new DNS, but still let it connect to the same DB.
6. Update the endpoint in the main app to call the external DNS for accessing the campus-backend.
7. During a maintenance window, we split the database by promoting a read replica.
8. The Learn Engineering team will now become responsible for the new campus-backend service. They can choose if they want to refactor or rewrite it.

Separation of concerns

In part 1, we talked about the benefits of looking at our internal services as if they were open source, with a focus on improving communication among teams. An additional benefit that we did not discuss, is that this mindset imposes questions like “If we were to open source this service, would anyone use it?”, “How coupled is this library to our specific problem?”, “How hard would it be for other people to integrate with it?”, etc.

Assuming that other teams will interact with the newly created service, as if it were to be an open source library, automatically puts you in a mindset of high decoupling.

The next part will go over how we want to enforce some consistency among our services, without limiting ownership in our engineering teams.