GDPR, how it affected FidMe’s back-end
A journey into our service-oriented architecture
These last few months GDPR imposed many changes to most of the apps we daily use. Hopefully, one of these app is FidMe.
If you use FidMe, you probably noticed the most obvious change : a large popup, asking for your consent regarding data usage. Beside this change, we have decided to take a deeper action on privacy by providing many tweaks and features for users to control their data usage and more.
We will focus here on one of the most -programmatically- interesting features we have added since GDPR.
The <Delete my account> feature
Now you must be wondering how the hell we managed to live without having a “delete account” button since then…
I also wondered that when I started working on it. First off, let’s remind ourselves that most websites and apps only make you believe that you can actually delete your account.
In reality, for many reasons (historical, financial, etc) this feature is often limited to removing the ability for the user to login.
This is called “soft delete”, and in code it often translates to something like that :
Only users for which deleted_at property is nil have the ability to login.
As you might have guessed, this does not actually delete any user’s data.
With FidMe, our goal was to provide the user with the ability to actually remove its data from our servers. Meaning :
- Any field containing private information has to be deleted
- Any traces of his actions must be deleted
- Loyalty cards, photos of receipts, user information, emails, everything must be deleted
Here comes the challenge…
To give you an insight of our back-end architecture, here is the list of our main services:
Some of these services even have sub-services.
That makes a list of approximately 30 services working together to make FidMe app work.
Having a large number of services is useful in many ways. But some trivial tasks in appearance, such as deleting a user account, can become a real challenge in a decentralized architecture.
Instead of querying a single server to delete a user account, here is the flow we had to introduce :
- The Consent service (the one in charge of privacy) is asked to delete a user and it will follow this demand to all the other services, here is the schema
- In detail, a DeletionRequest is created on the Consent Service in a “pending” status. Meanwhile and during a 1 month period, the user receives a few emails reminding him that he can still cancel his request.
These reminder emails are triggered by a background task, we then send them synchronously :
- The 1 month period expires. A deletion background job is created for every expired and not canceled request (we call them “due for deletion”) :
- The job is triggered, cleaning process starts.
In order to remove data from each service, we introduced the notion of Cleaners.
A Cleaner, is a class that is supposed to hold the behavior related to the “cleaning” job of a single service.
Here is what we do then :
- We call every cleaners, here is the most simple example, the one in charge of deleting user’s receipt photos
- They dispatch any associated HTTP request required to complete the deletion
- If any cleaner fails, we retry a few times, otherwise we store the reason in an upper layer
This main Runner class, the one that is responsible for calling each independent Cleaner is called by a Rails ActiveJob that is in charge of handling retries and failures.
By moving our code away from the active job itself, we add more abstraction to our manager. It becomes easier to test, easier to call again in case of failure, and gives us the freedom to move away from Rails if we feel like it.
That’s it for the job of the Consent service. Or is it?
Well, it has called every service, it gave them the responsibility to handle the deletion themselves. Hopefully they have done their job. Everything is fine.
Except one thing. Considering that the Consent service is responsible for the user’s deletion, it would just feel natural if it has a bit of control over it.
Of course, it is up to each service to know the behavior and logic required to delete a user account. But this entire feature felt very close to the main Consent service.
So, we split our services in modules…
Adding the deletion behavior to each service is now pretty straightforward.
Each service must implement more or less logic to delete user’s data, but in general it can be as simple as deleting things in cascade
But when there is more behavior required, for example when we have to delete images, logs, complex destructured things in our database, we don’t want to “pollute” our service’s code with deletion related code.
So we thought, let’s make a module !
Our stack being very rails oriented, we have what is called “engines”. An engine is a sort of Rails plugin that adds/override any behavior of a Rails monolith.
The plugin being a Gem, all you have to do is add it to your Gemfile.
At this point, we needed to add methods and behaviors to each main Rails application in order to handle Consent HTTP request.
To follow up on the above general schema, here is the detailed redundant process for each service.
And in this process our newly created engine adds the entire logic to the initial service :
- It adds the correct route :
- Which leads to a new controller with its own Authorization strategy
- And of course, the service holds the logic that deletes everything
All of the database’s related deletion is done transactionnaly in order to avoid and catch any potential error.
In the below example, we call our “Anonymizers” which are in charge of… guess what… anonymizing the data that can’t be completely destroyed. The whole thing is processed in a transaction.
Our special insight into our service oriented architecture is over !
Even though it is not the point of this post, I feel that I should write a little about my feelings regarding our architecture.
So far, even if service oriented adds complexity, we reach a level of decoupling that is extremely high and pleasant.
I think it is the most important thing here. Given that our server load has increased over the years, we managed to keep our heads clear of all the complexity by working on very small and decoupled fragments of our application.
This is extremely important for us, and for our new developers. We are really hoping that they will be able to quickly grasp the problematics of the projects we make them work on.