Ireland’s Data Processing Commission (DPC) has recently provided another example of a familiar problem: legacy systems may serve a critical function, but the difficulty of integrating it with newer applications limits the utility of the data they store and process.
In this case, Ireland’s DPC has a problem with Lotus Notes.
A quest to replace Lotus Notes applications
As the lead data supervisor for tech giants like Apple, Facebook, and Google, the DPC is the EU’s busiest GDPR regulator. According to their 2020 Annual Report they handled more than 10,000 cases and received 4,476 GDPR-related complaints and 6,628 data breach reports. Remarkably, they manage all of this with Lotus Notes, an application platform first released over 30 years ago.
They had planned to prepare for GDPR enforcement by implementing a new website and case management system which should have replaced the Lotus Notes applications by April 2017, but the IT upgrade is overdue and over budget. Three years after the GDPR went into effect, the regulator monitoring some of the most advanced tech companies in the world is facing criticism for slow enforcement as it struggles to manage a huge case backlog.
The Irish Council for Civil Liberties recently released a report suggesting that the use of Lotus Notes was one factor limiting the DPCs ability to enforce the GDPR efficiently. The DPC reportedly responded that it had a fit and functional case management system, but conceded that the system was “dated” and “limited” in terms of how much it can be adapted for integration with a new DPC website and the shared platform used between EU data protection authorities. In other words, it works well on its own, but has trouble sharing data with the DPCs EU counterparts and working with the website that receives most of the complaints and breach reports.
Lotus Notes end of life: applications and data
Lotus Notes is a perfect example of the problems of legacy systems. It was cutting edge software when it was introduced in the late 80’s and wildly successful in the 90’s. It rapidly evolved from the original three applications — Group Mail, Group Discussion and Group Phone Book — into a workflow application managing email, discussion groups, websites, document libraries, workflows and custom applications.
Many of those custom Lotus Notes applications, like the DPCs case management system, still perform critical functions today, in spite of the cost of their maintenance and the difficulty of integrating them with newer systems.
A “document” stored in a Lotus Notes database (NSF file) contains not only data fields, but also rich content, embedded files, embedded views, links to other documents and databases, and parent-child relationships with other documents. These data structures have to be completely preserved in order to retain the documents as business records.
Integrating this Lotus Notes application data with newer systems can be quite difficult. It is also difficult to use the data for analytic purposes or archive it for compliance purposes, both of which should concern the DPC. The DPC’s 2020 Annual Report noted that it is aware of some troubling trends in GDPR enforcement. It is receiving a growing number of complaints that “disclose no identifiable data protection issue” and complaints that appear to be from individuals or organizations “attempting to misuse the GDPR to obfuscate or pursue other agendas.”
Given the volume of complaints, reports, and enforcement actions the DPC must manage, and the limited resources the DPC has to work with, analyzing historical data to help focus their efforts on the most critical complaints and issues ought to be a high priority. The data stored and processed by the legacy case management system must be preserved in a format suitable for analytics use.
Similarly, the DPC will be required to meet legal requirements to preserve information regarding enforcement actions. Given the slow pace of resolution, data regarding current actions will need to be readily available for several years, and archives will need to be preserved longer to meet public records requirements.
The issues are not unique to the DPC. In 2016, Forrester reported that on average between 60% and 73% of all data within an enterprise is not used for analytic purposes. Those numbers will decrease, as data analytics becomes a core business function, but organizations will have to address the issue of legacy data to get the most from their analytics efforts.
Compliance is also becoming an issue for more and more organizations, as new laws like the California Privacy Rights Act require more businesses to comply with requests to disclose, amend, export or delete personal data at the data subject’s request.
Historical data buried in proprietary databases is difficult to access, which means it increases the effort and expense of complying with data subject access requests. As advances in analytics raise the value of data, and new regulations require access to all personal data, organizations will have to migrate data from legacy databases into an accessible form.
Data migration
Fortunately, it is possible to extract data from Lotus Notes .nsf files. When archiving Notes-based data or migrating it to a new system, organizations must take care to preserve not just the plain data, but also the full information context.
Document hierarchies, links, attachments and metadata must be preserved along with the documents in a searchable form. Additionally, data that must be preserved for long terms, such as public records, will likely outlive not just the systems in operation today, but at least a few generations of future systems. To make future migration easier, it is best to convert Lotus Notes application data to open, standardized formats.
With proper planning, the data stored in your legacy systems does not have to go dark; it can be preserved for analytics and compliance purposes for years to come.
Originally published at https://www.swingsoftware.com.
