Hard Lessons from Hardware Failure: Data Loss and Recovery in the La MaMa Archive
by Kylie Goetz
Digital & Special Projects Manager, La MaMa Archive
Flood. Fire. Mold. These are words that conjure up terror in the heart of every archivist. For those of us who work with digital collections, you can add the words “RAID Array Failure” to the list of nightmare scenarios.
Yet, because we work with technology every day, we understand its fragility. We build in redundancies and plan for contingencies. We know the adage that “lots of copies keeps stuff safe,” as Stanford University articulated in their LOCKSS framework for distributed preservation. But what happens when our best-laid plans go awry?
In July 2022, the La MaMa Archive, the living archive of La MaMa Experimental Theatre Club (ETC), experienced that dreaded hardware failure. Our server crashed and new drives had to be purchased and installed. A previous issue in 2021 had been resolved without incident, so Archive staff assumed data protections were working as they should. We discovered, to our dismay, that the automated processes had not been re-established after an earlier failure. The line of code meant to automatically create nightly backups of La MaMa’s digital catalog was faulty. The IT contractors had not been verifying the backups and never noticed that the system stopped creating mirrored files. Delays, poor communication, and failed attempts to reload the digital collections site left us offline and in limbo for more than a month.
And still, further problems arose. The contractors did not confirm whether the backup was in place before attempting to restore the failed server array. In what was meant to be a “cost-saving” recommendation, only failed drives were replaced and not the full array. So when a server recovery was attempted, the reused disks in the RAID were overwritten. This made any further attempts to recoup missing files from those disks impossible. Not only was our public-facing catalog inaccessible, but eight years of digitization, metadata creation, and cataloging work were at risk. Our entire digital collections were in peril. The nightmare was happening to us.
Whirl-i-gig, the developer of CollectiveAccess, had taken a backup from late 2020 of our collections management software (CMS) to support a redevelopment of the design on the public-facing archives web site. Luckily, they still retained that backup. It included all of the digital surrogates shared on the Archive’s public collection site in access file size as well as all of the backend cataloging work. From this, Whirl-i-gig was able to restore our online catalog, complete with metadata and access level digital surrogates, up to the time of their 2020 backup. Without this assistance, we would have lost almost everything from our digital collections. Even with this assistance, more than a year’s worth of cataloging and digitization work was gone.
In order to move forward, first, and unsurprisingly, we terminated our relationship with the IT contractors who had overseen the data loss. La MaMa ETC then hired a dedicated IT Consultant, Thomas Lineham of Volcate, a New York-based technology consultancy. Volcate, together with Whirl-i-gig and the Archive, developed an action plan to quickly recreate and share the records and images that had been lost.
The details of that process — the nuts and bolts of how we rebuilt the catalog records and strengthened our data infrastructure — are laid out in a white paper, written to document our data recovery and share our experiences with other archivists. The paper is available here: Recovery after catastrophic hardware failure and data loss in the La MaMa Archive.
The broader lesson, and one which applies to every type of collection, is this: relationships matter. We believed we were following best practices regarding data protection, but relied on third parties who did not have the same buy-in that we did. All of the remediation work required extra hands and extra funds. We reached out to various partners for help. Whirl-i-gig offered their backup and helped relaunch the site, despite this work being outside of their contract with the Archive. An emergency grant from the Auchincloss Foundation made it possible to support extra cataloging hours for the Archive’s Metadata/Digitization Specialist. Reviewing and processing the import mappings and metadata for such a large batch import required additional Whirl-i-gig support hours, funded as part of a larger cataloging grant from the National Historical Publications and Records Commission (NHPRC) to preserve and make accessible our paper and photographic records. Without the help of our staff, consultants and partners, or the generous funders of our archival work, we would have been at an impasse.
Our best advice is that whenever catastrophic events occur to collections, whether they are caused by environmental issues, technological failures, or human error, you have to acknowledge them to be able to move forward deliberately. Allow staff the space to vent frustrations and grieve the loss of their previous efforts or collections work, and include them in planning the recovery. The care of all archival collections, including digital ones, is an ongoing process that requires ongoing vigilance from everyone involved.
Robust systems are those which are reviewed, tested, and revised throughout their lifecycle. For small non-profit archives and arts organizations, there will always be a negotiation between best practices and what is feasible within limited budgets, shared organizational infrastructure, and uncertain resources. How best to navigate those decisions will be different for every archive, and therefore needs to be a regular part of collections management discussions at all levels.
Despite all of the headaches and heartaches, we know that we have limited the possibility of such a data loss occurring again. We can continue building our catalog, confident that access to the rich history held in the Archive will be accessible and discoverable to La MaMa artists as well as theatre researchers, students, and interested members of the public for years to come.
Through this undertaking, we’ve learned of other institutions with similar but unpublicized situations, and we found comfort in their recoveries. So we are sharing our experience here not only as a cautionary tale or case study, but as consolation for any institution that finds itself at the beginning of a recovery process. With patience and perseverance, you may find your collections safer in the long term.
Visit our digital collections at catalog.lamama.org
