When we make a cup of tea, take a photo, or record our daily steps, we leave behind part of ourselves. As described in Strategies for Engaging Data Tracks, user generated data holds significant value for organizations, governing bodies, but, most appropriately, for the user themselves. The ability for people to share their lives and identities through digital technologies has created the possibility for experience to persist beyond the lifetime of its user. However, collecting memories through digital technology becomes a problem when it lacks structure and does not mirror the way people process memories. This conceptual prototype explores the infrastructure and implications of a digital legacy experience, its data process, and the potential outcomes.
How can digital memory-making and legacy creation be more intentional?
Approach
The concept was developed through three design approaches. The first approach was to learn by building. Assembling existing technology for new applications raised questions of how systems might adapt to memory documentation. The second approach was to diagram the data infrastructure as it relates to user experience over time. The diagrams helped to develop key processes from a perspective of longevity, both on the front and back ends of the system. The third approach consisted of materializing speculative artifacts which communicate the dynamic quality of the experience. The ephemerality of memory could be suggested through slow-reacting photographic printing (cyanotype), AI-generated images (DALL·E), and digital collage to propose data physicalization (Photoshop).
Process
Learning by Building
The concept was initially constrained by a scenario in which a user suffers memory loss and wants to capture daily experiences for themselves or their loved ones. With the belief that identity is embedded in daily actions and behaviors, passive memory collection was explored through smart home technology and automation. Through integration of Apple HomeKit, Apple Shortcuts, Philips Hue bridge and lights, an Arlo wireless smart home camera, a generic motion sensor and an Aura digital picture frame, the prototype constructed a space in which memories could be collected and displayed automatically when triggered by physical presence. The result was a third-person perspective of one’s experience, where images reveal motion and activity different from a composed picture in which the user is aware of the person on the other end of the camera.
The second iteration of passive collection for memory was through wearing a body camera. Body cameras, or body cams, are often associated with accountability policies for law enforcement officers, or GoPro-style action videos. In this experiment I wore a body cam while performing everyday activities such as driving, grocery shopping, and walking around my home. Despite brief clips of my face in a mirror, the outcome resulted in video clips of context without identity, primarily unfocused movement which made sense only to the wearer. A few stills from the body cam were used to create cyanotype prints to remove color information (other than blue) and convey the image as a moment for remembering rather than observing as part of the present.
The prototypes above may share characteristics with surveillance, however surveillance lacks consent. A network of memory recording devices as described would be configurable by the user. Privacy controls and provisions, such as face blurring, should be considered. The output need not be visual, other media formats such as sound or (in the future) smell could open the range for non-representational memory triggering.
Diagramming Legacy Infrastructure
The previous design approach surfaced team discussions regarding user experience, data modeling, and data privacy. In order to understand each problematic area, the digital legacy infrastructure as a whole was diagrammed to clarify the minimum interactions between user, data, and time domains. The infrastructure is composed of five key phases: Memory Capture, Memory Recollection, Artifact Sharing/Inheritance, Legacy Compression, and Legacy Externalization. These phases occur on two time-bound layers: Conscious Experience — which encompass user decisions and preferences, and Legacy System — the computational processes applied to data on the back end.
Before explaining each phase, it is worth defining memory data and legacy data. Memory data encompasses any data which can be tied to an individual’s perception of an event, emotion, relation, or information designated for future recollection or retrieval. Memory data is most likely media such as image, audio, video, or text, but can also include data tracks which embody the self. Legacy data is similar to memory data, however it reflects the aggregate experience through data visualizations or summaries, media compressed or synthesized into a single artifact, or information communicated through an accumulation. Consider memory data as the single photo within the legacy data photo album.
The first phase is Memory Capture which encompasses both active and passive collection of memory data. Active collection would include the user taking a photo, writing a journal entry, or directly documenting. Whereas passive collection incorporates the user’s biodata, environmental data, or other data which contextualize the memory. In this process the user may set permissions such that only images with people are saved as memories, while others might require an approval process. Settings for passive collection could include significant spikes in biodata, collection during seasonal events, or events based in significant locations. From the collected information, the memory object will be composed of information about the user, the technology and context of capture, the media type and details, the context of the memory including location or recognized content, sentiment based on information derived from the data and user input, and how the memory correlates to other memory objects. The memory object could be formatted to JSON like the example below. Similar memories are clustered together throughout the memory capture process.
The next phase of Memory Recollection is the process by which memories are periodically resurfaced. The process causes the user to recollect that specific memory and adjust its significance, recollection frequency, and classification. Perception changes over time and as personal objects trim down, so might memories. The system can ask the user to reflect on a memory through active or passive means such as a likert scale of most significant to least, or through gaze tracking to understand where their gaze falls more frequently. The process may reveal that the images of a user’s prior significant other are less important, or that the user become more interested in memories of a family member as time passes. The frequency by which memories are bounced to the user could mimic spaced repetition systems (SRS), used by language learning systems, to help people remember their most valued memories. The frequency may also help with users working through traumatic memories that may benefit from memory reconsolidation as directed by a mental health professional. Through the process of memory recollection, memory objects will update such that the data reflects whether they are deemed more significant or more valuable, thereby appearing more or less in a legacy artifact. The legacy artifacts are composed of several memory objects that represent an aggregate experience.
Following Memory Recollection is the Artifact Sharing and Inheritance phase. The system should support the ability to share memories and legacy artifacts just as people share photos. Given that the system stores more information in a memory object than the media file alone, users may share a reference to only pertinent information which may be revoked based on the owner’s preferences. Legacy artifacts describe an experience through several memories, like a collage or a short animation. Visual stylization may be applied as a means to share partial information, such as a blur, decolorization, or composite. Likewise, users may inherit legacy artifacts similar to heirlooms. Significant inherited artifacts may blend with the user’s own legacy data, just as hearing a story secondhand might be remembered as if it occurred firsthand.
The phase that is working in the background throughout the user’s lifetime is Legacy Compression. The memory recollection process serves to incubate what is considered valuable or prioritized. When memories are surfaced during the recollection phase, the cumulative legacy becomes crystallized through the compression of the multiple memory objects into one. In other words, what might be a digital equivalent of a personal time capsule? What objects, experiences, and information communicates the whole of a life? How might that time capsule be adapted for different intents, such as documenting ancestral heritage or sharing insight to others with similar medical conditions? It is from the compression process that legacy can be easily internalized by the user and externalized to other groups.
Legacy Externalization is the final phase in which legacy is distributed to external parties such as healthcare providers to support medical research, estate planning to distribute assets among beneficiaries, organizations to commemorate culture or archive institutional knowledge, and the shared network of loved ones who inherit intimate information. The legacy infrastructure must consider the type of information and security access based on user preferences and how those may change after death. Below is a table of access variation across possible data types such as identification data, legacy artifacts, biometric data, etc., for each external party.
The infrastructure seeks to mirror the natural processes humans exhibit when considering their memories and legacy. People choose to remember moments which hold significance to their life as a whole. They modify or rearrange those chosen moments as their perspective changes through life. Memories become blurred, compressed, and abbreviated based on time passed, who the memory is shared with, and how embodied mediums age. Lastly, identity is obfuscated to varying degrees when legacy information is shared with external parties such as estate planning attorneys, medical professionals, or philanthropic organizations. The infrastructure described above seeks to adapt human behavior to digital information in order to preserve privacy, provenance, and, ultimately, to reduce the cognitive burden of defining a legacy.
Materializing Memory
Prototypes of legacy artifacts from the system were produced to shape the system’s design language and suggest how the phases of the legacy infrastructure may materialize.
Through the prototyping process DALL·E and images under creative common license were used to represent relatable and realistic, yet anonymous depictions of memorable events. The four below each represent different levels of abstraction, stylization, and context information. The variation resulting from a phrase led to idea that the user could collaborate in creating synthetic artifacts of their memories by contributing language or media.
With inspiration from the Sensory Moving Image Archive and Jessica Charlesworth’s MeMo, the next step was to understand how the complexity of memory data could be visualized in a unified design language that echoes the ephemeral themes. The Ginkgo was chosen as a visual representation of memory both for its reputation as a memory-enhancing supplement as well as Ginkgo biloba being an ancient tree species. The irregular yet recognizable outer edge of the ginkgo leaf would embody a data visualization of the user’s engagement with recollected memories. The radial veins of the leaf encompass the various data components of a memory. The face of the leaf can serve as a projection surface to capture the visual impression of a memory.
In combination with the earlier concept of smart home technology to create a memorial space, the ginkgo branch could be a living projection in the home to convey the growth and pruning of a user’s legacy. The intent of a projection is to bring data visualization in a living space, similar to how a child’s growth is marked on a door jamb.
Exploration of a dataset is mirrored in the hierarchy of trees, branches, and leaves. The user could explore their memory by filtering for different parameters, enabling various data visualization and generative modeling techniques to curate the forest of legacy.
Conclusion and Next Steps
The concept has evolved within the research-through-design process, resulting in a comprehensive infrastructure which processes memory objects into a final legacy dataset. The data which represents memories are essentially data tracks driven by longitudinal perception. Despite the concept’s focus on personal memory, the system has application in organizational settings where institutional knowledge or revisioning can be archived.
The next steps would be to consider the implications of the system, prototype with first-person datasets, solicit feedback from a diverse population, and develop higher fidelity assets.
