Authors: Li Wha Chen, Abhimanyu Saha, Tommy Zhang, Kavita Kattimani, Douglas Paton
In this blog, we take a look at how e has evolved to handle the increased number of requests across Adobe Experience Platform. We examine the challenges that arose from this increase and how our API-based solution helped everyone manage the situation.
Managing privacy-related services has become increasingly complex over the last few years. Regulations like GDPR mean that greater care needs to be taken with regards to how information is stored, transported, and used. These regulations meant that people have more control over their data. If they don’t want a company to store it or use it, or if they’re in an industry where privacy is critical, they can issue a request to know what data is being stored and delete the data being stored. For the Adobe Experience Platform Privacy Service team, these new and changing regulations meant we had to adapt to how we processed this data. And that we had to come up with a system for handling the large volume of requests that we would get from customers who store their data with us every time a new privacy regulation was passed or an old one was updated.
This large volume of requests we saw would inevitably lead to a bottleneck that would slow everything down. At first, we weren’t ready for these spikes. But, as these regulations became more common and complex, we needed to get a system in place to help us manage the load, streamline the process, and reduce bottlenecks.
Getting around the roadblocks
One of the challenges we faced was that we work with a variety of different services and solutions. Each platform had different requirements when it came to how we dealt with it and it handled requests. This meant we had to come up with a way to manage requests across all of these different systems that would be compatible, without having to come up with a different solution for each one.
We also needed a system that was stable enough to allow us to focus on being compliant and getting ahead of upcoming regulations. The previous setup often led to bottlenecks and other performance issues that required a lot of manual intervention on our part.
Creating a more stable, self-serve solution
To help us handle the spikes in requests and deal with latency issues, our first step was to upgrade to a more performing database. We decided that Azure would serve us best.
As part of continuous refactoring and code quality improvement, we normalized the data persistence layer and modularized code to help adopting future enhancement with little code changes.
As a result of these changes, we have significantly improved the API response time 87% faster in the best scenario from 80.5ms to 10.4ms.
The next step was to come up with a solution for managing privacy requests across the different solutions our customers used. Solution teams were spending considerable time and resources to manually verify that all jobs were received. To make the system self-serving and resilient, we developed a job status API that allows solutions to cross verify the jobs received by the Privacy API. Best of all, if the system got stuck, solutions would be able to retrieve and respawn the job, without our intervention.
To further drive the self-service aspect of Adobe Experience Platform Privacy Service we updated the UI to be more user-friendly. The new interface also provides users with the ability to track metrics and monitor performance on their end. Users can now track the total number of jobs in progress, keep an eye on the total number of requests they have made, and see how many errors they have encountered. Users can also take advantage of the Job Builder functionality to create job requests using the new interface.
Initially, notifications for the system were sent via email. This proved to be cumbersome for customers with thousands of submissions each day. They could do the job GET status API to check on results, but the deluge of email notifications was neither scalable nor sustainable. To manage this, we switched to Adobe I/O Events workflow. This lets us leverage Adobe Experience Platform Pipeline and Adobe I/O Console to deliver push notifications based on subscriptions and configuration. There are four notification messages that are possible Job Complete, Job Error, Product Complete, and Product Error.
One of the points to highlight here is Adobe I/O events which are in JSON follow CloudEvents standard format.
- CloudEvents is a specification for describing event data in common formats to provide interoperability across services, platforms, and systems.
- Adobe is participating in the CloudEvents working group and Privacy events are the first event provider published using this standard.
We have a system that is now more resilient, as well. User escalation is down by roughly 90% to the point where we may have one instance a month. This more stable system has also reduced CSO. We used to hit once a week during the first couple months of the GDPR 1.0, at this point, it’s been over seven months since it has happened.
Staying ahead of the curve
Stabilizing the backend and removing the roadblocks was just the start for us.
New regulations are coming out every year and current regulations are prime for updating. We have to be ready and compliant for these updates as they come out.
Looking forward to the next little while we have the California Consumer Privacy Act coming into effect in January of 2020, with a goal of being compliant by November 2019. There is the LGPD coming in 2020. And we are planning for HIPAA regulation as well. All of which will be incorporated into Adobe Experience Platform Privacy Service, with the ultimate goal of making managing privacy requests as simple and self-service oriented as possible. We know that privacy is a complicated thing. Removing as much of the manual aspects of these requests as possible and creating a more stable, user-friendly UI helps increase compliance for everyone.
As part of the changes we’ve and to help people understand that we’re focused on all aspects of privacy, we’re rebranding from being a GDPR-specific to a more general Adobe Experience Privacy Service. This way, it’s clear that we’re tackling the one to two new privacy regulations that are being put into place each year.
We currently have 10+ Adobe services onboard as part of our privacy platform. Our goal is to continue adding services, so we can offer customers full compliance across as much of the Adobe family as possible.
- Adobe Experience Platform Privacy Service — https://www.adobe.com/experience-platform/privacy-service.html
- Adobe Experience Platform — https://www.adobe.com/experience-platform.html
- Azure — https://azure.microsoft.com
- Adobe I/O Events — https://www.adobe.io/apis/experienceplatform/events.html
- Adobe I/O Console — https://www.adobe.io/apis/experienceplatform/console.html
- CloudEvents — https://github.com/cloudevents