TEN steps FWD — E3— Multi-modal NFTs
Welcome to TEN Steps FWD, a series exploring the evolving landscape of Decentralized Confidential Compute (DeCC) through the lens of TEN and web3 development. This series will showcase a variety of fascinating use cases and real-world applications, pulling back the curtain on some of the most innovative decentralized applications (DaPPs) in the realm of web3, AI, and DeCC.
Read Part 2: Horizon IT & Postmasters Strike Back
Reevaluating Value with Multi-modal NFTs (mNFT)
Beauty is in the Eye of the Beholder
In this segment, I’ll discuss an active area of research at TEN and one of the most exciting innovations in digital spaces: multi-modal NFTs (mNFT). Before diving in, let’s briefly explore how traditional value representation and interaction paradigms shape our current understanding.
Historically, we have used practical methods to represent value due to limitations in time and cognition. Consider language as a system of communication; we use words to convey thoughts, money to communicate value and trust, and art to express intrinsic and extrinsic values. These mediums illustrate the unity between language, economic systems, and asset classes, highlighting the importance of underlying semantics and shared experiences.
An Augmented Intelligence
What does an augmented world look like as AI, Blockchain, and private personal agents extend memory and comprehension, providing improved analytical insights and intent automation?
With AI, we can develop specialized communication methods for more accurate and efficient interactions. Imagine having the ability to remember and process an infinite amount of information. This would allow us to plan and anticipate our actions with the same ease and intuitiveness as exchanging small gestures with close friends, such as a knowing glance or a nod. Similarly, generative AI can bridge the gap between our ideas and their expression, enabling everyone to create and communicate with the skill and fluency of a master artist.
Subjective Experience Enablers (SubEE)
Three critical components of disruptive innovation in the era of web3, blockchain, and AI are:
- Decentralized escrowed data availability (contextually relevant, zero-knowledge operations)
- Deep-learning and intent paths (high-dimensional compute with summarization)
- Generative multi-modal views (translation functions from semantics to multiple channels)
These components are critical infrastructure enabled by DeCC platforms like TEN to reduce friction beyond just financial transaction but to exchange of cognitive senses, enhancing our fully realized experiences.
Understanding Multi-modality
Multi-modality enhances communication by adding information and context or amplifying the original message. For example, the concept of “firetruck” can be enriched with visuals, sounds, and sensations, providing a deeper understanding and experience.
The Multi-modal NFT
A basic NFT (non-fungible token) represents something unique and is suitable for asset classes focusing on rarity or collectivity. An mNFT extends this concept by acting as a subjective reference to a base experience. It captures multi-modal experiences, providing a richer, more personalized connection to digital assets.
Example: A Title Fight in Vegas
Imagine you’re in Vegas watching a title fight. A knockout punch happens, and the moment is captured as an NFT. Your personal experience of joy and excitement and your friend’s contrasting experience of anxiety and despair are both mintable as mNFTs. These moments, enriched with contextual and sensory data, create a more comprehensive and shared digital experience.
Practical Use Case: Subjective Views of Fireworks
To illustrate the concept of multi-modal NFTs (mNFTs) and Subjective Experience Enablers (SubEE), let’s explore how two individuals, Alice and Bob, experience a fireworks display differently. We’ll capture, analyze, and augment their experiences using GPT models and user profiles.
User Profiles
Subject 1: Alice
- Heart Rate: 65 bpm
- Emotion: Calm, composed
- Cognitive Load: Low
- Common Sentiments: Contentment, mindfulness
Subject 2: Bob
- Heart Rate: 72 bpm
- Emotion: Energetic, curious
- Cognitive Load: Moderate
- Common Sentiments: Inquisitiveness, anticipation
Capturing Initial Perceptions
When Alice and Bob watch the fireworks, their unique baseline vectors influence their perceptions. The system captures their real-time biometric data and emotional states, creating differential instantaneous models of their experiences.
Example Fireworks Perceptions
Alice’s Perception: Alice sees the fireworks as a soothing display, appreciating the calm beauty and structured patterns. Her calm demeanor and contentment make her focus on the harmonious and serene aspects of the show.
Bob’s Perception: Bob experiences the fireworks with excitement and curiosity. He focuses on the vibrant colors, loud bangs, and the unpredictability of the explosions. His energetic and inquisitive nature drives him to seek out the dynamic and thrilling parts of the display.
Modifying the Fireworks Images
These are actual generative AI images using diffusion models on data matching the presented simple profiles.
Using the data from Alice and Bob’s profiles, we can tweak the images to match their subjective experiences more closely. The goal is to minimize the semantic difference in meaning for each individual.
Base Image:
- This is the original image of the fireworks display, capturing the event as it objectively appears. It serves as the starting point before any subjective augmentations are applied.
Alice’s Augmented Image:
- Baseline: Calm, composed
- Enhancements: Softer colors, smoother transitions, and a more structured and symmetrical pattern.
- Augmented Image: The image is modified to emphasize the calm and serene aspects of the fireworks, making the display appear more harmonious and soothing.
Bob’s Augmented Image:
- Baseline: Energetic, curious
- Enhancements: Brighter colors, more contrast, and dynamic, chaotic patterns.
- Augmented Image: The image is modified to highlight the energetic and thrilling aspects of the fireworks, making the display appear more vibrant and exciting.
Summary and Extention
By leveraging GPT models and user profiles, we can establish baselines for Alice and Bob’s perceptions, calculate the vectors between their experiences, and augment images to reduce subjective semantic differences. This approach not only enhances individual experiences but also provides a richer, more personalized connection to digital assets. It showcases the potential of multi-modal NFTs in creating immersive and meaningful interactions.
It is important to note that we have primarily focused on one modality here — visual perception of fireworks — but the true power of this technology lies in its ability to extend across multiple expressions. This same process can be scaled to include additional modalities such as sound, tactile feedback, accelerometer data, augmented reality (AR), virtual reality (VR), and more. By integrating these diverse sensory inputs, we can create even more comprehensive and deeply personalized multi-modal experiences.
The Importance of DeCC Blockchain and mNFT Protocols for Experiences
Incorporating blockchain and mNFT protocols for experiences and subjective perspectives offers significant technological advantages:
Privacy and Authenticity
By signing over an experience, the holder controls the privacy layer. This ensures that only the holder has access to the experience data, while a reference to enclave-protected data is available for trustless inference. This approach maintains the privacy and authenticity of the experience. For example, a user’s private experience of a concert can be securely stored and shared only with trusted parties.
Market and Exchange Medium
mNFTs create value through the rarity and uniqueness of subjective experiences. Derivatives can be obtained from other assets, whether NFTs or other mNFTs. The value of an mNFT can be influenced by factors such as the identity of the person who had the experience (e.g., a famous person) and the quality of the experience (e.g., high amplitude or rarity), enhancing their collective measure. Imagine owning an mNFT of a memorable event experienced by a celebrity, which could be highly valued in the market.
Indexing, Querying, and Aggregation
A standard specification for linking semantic data references and subjects of perception allows for interoperability and generalization across various use cases. This makes it easier to index, query, and aggregate data, enabling broader applications and more seamless integration. For instance, mNFTs can be indexed across different platforms to provide a cohesive experience for users.
Seamless Integration into Token Platforms and Cross-chain Exchange/Bridges
mNFTs can be easily integrated into existing token platforms and support cross-chain exchanges and bridges. This facilitates broader adoption and utility, allowing mNFTs to be exchanged and utilized across different blockchain networks. For example, an mNFT created on one blockchain can be transferred and used on another, enhancing its versatility and reach.
Conclusion
Multi-modal NFTs and SubEE DeCC technologies promise to revolutionize how we capture, share, and interact with digital experiences. By bridging subjective experiences and leveraging advanced AI and blockchain technologies, we can create richer, more meaningful connections in digital spaces.
Explore more functionality of mNFTs:
Read more from the “TEN steps FWD” series:
Alch3mist, (aka Anthony Nixon) is a web3 engineer with a passion for cognitive science, AI, and information theory. Currently contributing to TEN.
