Virtual Beings: A New Internet-Native Medium for Shared Storytelling

Published in

1kxnetwork

17 min readMay 19, 2022

The increasingly virtual nature of our shared social experience will exponentiate the population of virtual beings over the next few decades.

In short, virtual beings are entities or characters that exist entirely in the digital domain, crossing over to our world via social media interaction, shared mediums like games and Internet-native spaces, and hologram technology. They range in scope from human-controlled avatars manipulated with motion capture technology to fully-sentient anthropomorphic AI.

Virtual beings exist at the intersection of multiple technological trends (e.g. VR/AR, artificial intelligence, high-fidelity CGI, gaming, social media) and human social psychology (e.g. parasocial relationships, desire for companionship, cultural animism, anthropomorphic design).

Web3 has the potential to further accelerate the emergence of these digital consciousnesses through new creator revenue models, Internet-native organizational structures, and crypto-enabled digital ownership.

In this article, I will:

Define and briefly discuss the history and characteristics of different types of virtual beings
Review some of the existing technological stack used in their creation
Identify areas in which crypto-economics and web3 technologies can improve upon the creation, management, and distribution of virtual beings
Highlight existing projects working in this domain
Speculate on the future of web3 social experiences in the age of artificial intelligence and synthetic media (AI-generated media, generative media, personalized media)

By reviewing multiple aspects of virtual beings, this piece hopes to elucidate the concept as a trend here to stay that can significantly benefit from the use of the web3 technology stack.

Types of Virtual Beings

Virtual beings (VBs) come in many forms and vary in complexity.

VTubers

VTubers are virtual beings who stream their life on popular video platforms. VTubers are humans using a spectrum of consumer-ready motion capture technologies and character creation softwares, commonly in the anime aesthetic and streamed on Twitch or YouTube. — virtualhumans.org

VTubers originated in Japan in the mid-2010s and grew into an international phenomenon by the early 2020s. Using real-time motion capture technology to manipulate a virtual avatar, VTubers stream and post videos across a range of platforms, including YouTube, Twitch, Niconico, and Bilibili, to a fanbase enthralled by the characters they’ve created.

Early iterations of the VTuber go as far back as February 2010 when visual novel maker Nitroplus began uploading content to their YouTube channel featuring their mascot Super Sonico talking about their upcoming releases. The phenomenon reached a tipping point, however, in late 2016 after Kizuna AI made her debut on YouTube:

【自己紹介】はじめまして！キズナアイですლ(´ڡ`ლ)

Kizuna AI first coined ‘virtual YouTuber’, shortened to VTuber, to describe herself. Drawing on her ability to create an intimate experience for her fans by responding to their questions and comments, she was able to grow her fanbase to two million subscribers within ten months.

Following the success of Kizuna AI, the VTuber trend exploded in popularity in Japan, and later expanded internationally via the appeal of anime and manga fandom. The dramatic increase in Twitch viewership during the 2020 COVID-19 pandemic further heightened familiarity with the concept outside of Japan, leading to the creation of a VTuber tag on Twitch and for YouTube to note it as a dominant trend in its 2020 Culture and Trends report, emphasizing its growth to 1.5 billion views per month by October of that year.

VTubers rely on an ability to tap into the power of fan interaction, parasocial relationships, pseudonymity, and storytelling, creating vibrant characters that tend to be more interesting than their human streamer counterparts.

Today, there are more than 16,000 VTubers (up from just 1,000 in March of 2018) with millions of fans across social media platforms. Popular VTubers include Gawr Gura, Inugami Korone, Nyanners, Ironmouse, Kiryu Coco, Code Miko, and Ai Angelica, to name a few.

Within web3, NFTs have similarly been noted for creating rabid fanbases, community interaction, and a sense of belonging, so they are well-suited to capture VTuber mindshare. Some VTubers have begun to experiment with the technology, with Kizuna AI being the primary example after announcing a hiatus beginning in February 2022 to take time to further explore NFTs.

The most obvious use case for VTuber NFTs is the creation of token-gated content and Discords, along with digital collectibles. But two other possibilities stand out: rights management and avatar markets. Fully rigged character models could be created by third-parties and distributed in the form of NFTs. This is an upgrade to the current marketplace and commissioning system in place since creators could better track the use of their models while receiving secondary sale revenue flows. Anata is one such project experimenting with this, launching 1,000 male and 1,000 female Live2D VTuber avatar NFTs that are fully functional in streaming apps.

Virtual Influencers

A virtual influencer is a digital character created in computer graphics software, then given a personality defined by a first-person view of the world, and made accessible on media platforms for the sake of influence. — virtualhumans.org

Virtual influencers are not dissimilar to their traditional counterparts. Their lifestyles are depicted on social media, they interact with their fans across many mediums, and are often used by brands as spokespeople for their products. However, unlike your average celebrity, virtual influencers are CGI-generated personas that can be anyone, do anything, and go anywhere in the digital world. They are not limited by physical constraints, so can be in two places at once, never tire, and never go off-script.

One of the most well-known virtual influencers is Lil Miquela, a 19-year old self-proclaimed robot pop star living in Los Angeles. She has modeled for Prada, starred in Calvin Klein ads, gave interviews at Coachella, and topped Spotify charts. In 2020, Lil Miquela made approximately $12M for her parent studio, Brud.

Virtual influencers like Lil Miquela have almost three times the engagement rate of their IRL counterparts, and brands love them for the control they enable over their identity. There are plenty examples of celebrity sponsors making costly mistakes for brands, something that could never happen with a virtual influencer whose entire life is scripted.

In addition to control, virtual influencers offer brands the ability to tell stories across a myriad of platforms and formats in a way not possible by their human counterparts in what is known as transmedia storytelling. This form of storytelling uses multiple media to provide different access points to a story or character. To get a sense of this power, consider the Travis Scott concert in Fortnite. In order to perform in-game, an avatar had to be created for Travis Scott which breaks the illusion — you know that it isn’t actually him but instead him playing as an avatar in-game. A virtual influencer, alternatively, looks the same across all platforms, which creates a narrative spell not possible by the flesh-to-digital transcription of real life celebrities.

The most prominent way brands have interacted with virtual influencers to date is in the realm of fashion. Virtual influencers are oftentimes decked out in the latest wear as they pose across social media. Their scripted lives allow them to craft a specific lifestyle element and personality for their brands. Lil Miquela attended the Prada FW18/19 show in Milan, Shudu Gram became the digital face of Balmain’s models, and Noonoouri graced Dior ads as part of their “Rouge” campaign that originally starred Natalie Portman. Major fashion houses such as Balenciaga, Alexander McQueen, Valentino, H&M, Nike, Salvatore Ferragamo, and more have all embraced virtual influencers to showcase their items. Some, such as Prada, have even doubled-down on the idea to create their own in-house characters.

Music has also been a prominent area for virtual influencers:

Yameii Online’s track “Baby My Phone” inspired multiple Tik-Tok trends and mash-ups reaching millions of views across hundreds of thousands of videos
Kai, a 16-year old virtual influencer within Roblox known for creating the game Splash (who recently raised over $20M from investors) performed to over 100,000 fans in 7 different Roblox venues simultaneously for over 48 hours
Virtual K-Pop girl group K/DA created by Riot Games to promote new game characters ended up raking in 508M views on Youtube with a viral music video and song ‘Pop/Stars’
FN Meka raps on social media while showcasing ultra rare items like Starbucks-branded PS5s and AR sneakers

Virtual influencer culture has recently begun merging with web3: Dapper Labs acquired Brud — the studio behind Lil Miquela, multiple virtual influencers have launched NFTs including Lil Miquela, LV4, Rae, Shudu, FNMeka, and Yameii, and major web3 studios such as RTFKT have collaborated with virtual influencers to promote their NFTs. It is especially exciting to think about the future of web3 and virtual influencers, such as rental contracts to use in events or ads and fractionalization of the influencer to form DAOs around them that curate their stories and receive revenue flows to a community-owned treasury.

Distributed Personas

A distributed persona is a digital character that is managed by an open and permissionless community. Fans create content such as music, animations, videos, lore, and more using the character. Distributed personas may be autonomous in some form such as AI artists but are ultimately moderated by the community.

Separate from virtual influencers although similar in nature is the category of distributed personas. These are characters who are broadly managed by an open and permissionless community, as opposed to virtual influencers which have one authority or studio managing them.

One of the best examples of this today is Hatsune Miku, a Japanese Vocaloid software voicebank released by Crypton Future Media in 2007. Vocaloid software allows users to synthesize the sound of singing by typing lyrics and inputting a melody. Because each vocaloid has its own characteristics and are effectively separate ‘singers’, they are typically released with moe anthropomorphic characters, the most successful of which is Hatsune Miku herself.

Hatsune Miku was originally released under a “remix-free” license. Since her debut, she has starred in hundreds of thousands of songs and music videos made by creators and fans from all over the world. A freeware program, MikuMikuDance, spawned a boom in user-generated animations which were even included in live holographic performances attended by fans around the world. The distributed nature of Hatsune Miku and the remixing culture at her core prompted Crypton to adopt a non-commercial with attribution creative commons license for the character.

Hatsune Miku is a highly successful distributed software that has a decentralized community of creators making new songs and animations for the virtual being, earning over $120M in total revenue through music, merchandise, hologram concerts, and shareable media in the first five years alone. This model lends itself well to DAOs — human creators release new music and dances, vote on content canon, and collect revenues to a community-owned treasury. Some distributed personas have begun experimenting with this idea.

For example, Botto is a decentralized autonomous AI artist controlled by holders of $BOTTO, the native governance token. Token holders stake $BOTTO to earn non-transferable Voting Power (VP), which is used to vote on 350 AI-generated art works each week. At the end of each period, the piece with the most votes is auctioned on SuperRare as an NFT with proceeds used to buyback and burn $BOTTO tokens. To date, Botto is one of the highest grossing artists of all time on the marketplace with over $2M in total sales.

Since creative AIs have the ability to infinitely create new works, human curation will play an important role in the new economy. DAOs that manage the curation, distribution, promotion, and governance of distributed personas while benefiting from value creation will likely spawn a new age of AI-enabled and/or human-generated digital character productions.

Autonomous Entities

The simulation of human intelligence in machines that are programmed to think like humans and mimic their actions. The term may also be applied to any machine that exhibits traits associated with a human mind such as learning and problem-solving. — virtualhumans.org

The first three types of virtual beings were broadly human-controlled in one fashion or another. Our final type, however, is not. Autonomous entities use artificial intelligence and other tools to simulate life. They can run the gamut from chatbots and virtual world non-playable characters to AI companions and digital reincarnations of the deceased.

AI Companionship

One of the simplest forms of autonomous entities are AI companions. The rise in trust and embrace of AI among younger generations seems to entail a future where humans form close bonds with AI entities. Entire business models have formed around this idea, with companies like Replika providing friendly chatbots that interact with their humans throughout the day and Hybri providing custom 3D model bots to simulate romantic partnership. The growth of the AI companion market is evident already: Xiaoice, an AI companion spinoff from Microsoft, was recently valued at $1b, and the conversational AI market is expected to grow to $13.9b by 2024 on the back of the Loneliness Pandemic — a noted uptick in loneliness experienced by 61% of polled adults.

One project in web3 focused on AI companions is Petaverse Network, which draws on a long tradition of virtual pets, from Tamagotchi to Nintendogs. Petaverse Network seeks to create an interoperable digital standard for virtual pets, hashing their personality on-chain so that it exists forever. This personality represents a data layer for the pet that updates over time as you interact with it across games, AR, and VR. Other projects will be able to build on this data layer, reinterpreting the pet for their own virtual worlds and games.

Digital Reincarnation

Extending AI companionship further, an AI could be trained on the digital footprints of an individual to replicate their behaviors after they die in what is known as digital reincarnation. With enough text messages, social media posts, and videos used as training data, a neural network could learn to simulate the speech and mannerisms of the deceased while presenting themselves in a photorealistic CGI model of their likeness or as a chatbot. In fact, it was recently reported that Joshua Barbeau used a software to create a hyper-realistic chatbot backed by GPT-3 to recreate the experience of talking with his deceased fiance by training it on her text messages. Because of the sensitive nature of digital reincarnation, rights management will be crucial, and encrypted NFTs that require staking to access might be the best solution to ensure these digital replicas are not used elsewhere on the Internet.

Autonomous Entities

Autonomous entities are not limited to companionship or digital reincarnation though. They could also be virtual world agents. A glaring issue with many virtual worlds today is the emptiness felt when exploring. Populating virtual worlds with highly intelligent NPCs is one method of giving life to these environments. Imagine:

Decentral Games’ virtual casinos stocked with highly sophisticated chatbots
Cryptovoxels stores inhabited with AI NPCs selling digital goods 24/7
Anthropomorphised trading bots with distinct personalities
On-chain creatures that live and adapt to external variables using oracle data feeds
Game characters that morally evolve over time based on the actions of a user across multiple games

The possibilities of virtual beings with artificial intelligence in digital domains borders on the endless.

The VB Stack

Virtual beings run on a variety of technical and creative infrastructures in order to operate.

Artificial Intelligence

Virtual beings vary in their degree of autonomy, ranging from fully human-controlled (e.g. vtubers, most virtual influencers) to increasingly intelligent AI enabled by Natural Language Processing (NLP), Computer Vision, and a cognitive development layer for memory storage, emotion, motivation, and more. Some of the most common tools in this domain is OpenAI’s GPT-3, a large language model that is scarily accurate in depicting conversation with an actual entity, or machine learning algorithms that update base datasets over time as interactions occur.

While many autonomous virtual beings today run on centralized AI infrastructure, two prominent projects within web3 have taken different approaches to recreating AI on-chain.

Alethea AI, originators of the “iNFT standard”, made waves last year when they sold Alice at Sotheby’s for $478,800. iNFTs hash a GPT-3 personality prompt into the metadata of the NFT that is subsequently decrypted on the Noah Ark platform and used to generate a GPT-3 response to user inputs. These iNFTs can be trained over multiple engagements with the user, progressing through various levels of intelligence over time. Users can create iNFT versions of their favorite digital assets by pairing any NFT with a personality pod and training it using a combination of user interaction and $ALI token staking.

Altered State Machine takes a different approach to accomplish a similar goal. Agents in ASM begin with a form (the image or model of the agent) and a brain (a base NFT with a ‘genome’ matrix — a randomized set of base values — and a memory tree which logs state changes). GPU cloud compute providers use their hardware to run training algorithms for specific ASM applications (e.g. a popular game or a DeFi trading bot), similar to how mining works with PoW blockchains. The outputs from these training sessions are called memories and are logged in the memory tree in ONNX format, modifying the value matrix to improve the skill set in that specific application.

The first proof of concept for ASM is AIFA, an AI Football game, which uses a combination of Unity, IPFS, and a sample Training Gym. ASM hopes to extend the applications of their agents beyond this, however. Some conceptualized ideas in the whitepaper include DeFi trading agents, game character agents, chatbots, and digital assistants.

The design space for artificially intelligent agent creation on-chain remains largely unexplored, so it will be exciting to see this field develop. Importantly, the shift from proof of work to proof of stake means large amounts of GPUs will be looking for alternative uses, so decentralized AI models using GPU mining could be the next frontier.

3D Modeling and Animation

Central to all iterations of virtual beings is the use of 3D modeling, animation, or motion capture technology. While virtual beings take a variety of graphical aesthetics and forms, all require base models that are rigged for animation or movement. Marketplaces and repositories for these models already exist, and NFTs could provide an additional avenue for financialization.

The most common applications used in the modeling and animation of virtual beings and VTubers today include Unity, Unreal Engine, Blender, iClone, and FaceRig to name a few.

While some virtual beings do not attempt to look photo-realistic, the ones that do require extensive work. Designers often struggle with overcoming the uncanny valley, the innate revulsion humans feel when observing imperfectly rendered human faces. It’s possible the uncanny valley effect decreases in prevalence over time as newer generations become more accustomed to interacting with CGI and humanoid robots, but for now, the rendering of humanoid characters must account for this physical aversion.

The key limitation to realistic rendering is computer processing power. For example, a high fidelity facial render requires a mesh 3D model with sufficient detail and deformation control, texture imagery, and specular maps, all of which are intensive.

Clothing models are similarly intensive, primarily due to subsurface scattering (e.g. silk reflects light differently than cotton). This has led to strong demand from both CGI and gaming studios as well as the fashion industry for computer-aided garment design. Digital fashion houses like Artisant, The Fabricant, The Dematerialized, DIGITALAX, Tribute Brand, and RTFKT that specialize in NFT models and digital design are well-suited to capture market share of the virtual being space if they continue adoption of web3 technologies. As virtual beings continue to grow in popularity, it is likely that digital fashion houses grow in tandem.

The difficulty of avatar creation contributes to the creation of vibrant marketplaces for 3D models where creators sell the rights to their models to those looking to use them in digital art, games, movies, and more. Web3 unlocks additional opportunity here: NFT 3D models that can be resold with secondary sale fees trickling back to the original creator, superfluid payment streams that give control over a model while renting it for use in a virtual world, rights control via encrypted files that must be staked to unlock, and more.

On the animation side of things, 3D meshes are rigged to skeletons that can then be animated via code or user action. Experiments with combining animation and gestures with NFTs have already begun with HEAT representing the forefront of this movement, hoping to craft a marketplace for animations that can be applied to 3D models. By retargeting .glb and .gltf files to the bone structure of a 3D model, HEAT is able to create interoperable movement files for any standardized avatar file in virtual worlds.

VB Studios

The size of the team required to build a virtual being depends heavily on its complexity. Anyone can spin up a simple VTuber persona using VRoid Studio, FaceRig, and a streamer setup, but a virtual influencer or autonomous entity can require expertise in AI, animation, speech synthesis, modeling, content creation, and more.

VB studios often have teams dedicated to CGI art, photography, animators, storytellers, social media planners, fashion strategy, business development, PR, financial management, merchandising, and more. These studios can be dedicated to the management of a single character or they could be focused on creating a slew of characters. For example, The Diigitals Agency behind Shudu Gram manage a total of seven virtual influencers, Lu of Magalu is backed by one of Brazil’s largest e-commerce businesses, and SuperPlastic manages Janky & Guggimon, Dayzee & Staxx, Kranky, ShüDog, and more.

The DAO model works very effectively as a means of organizing content, creating fan experiences, and engineering beings (Hatsune Miku, although not currently a DAO, being a prime example of the power of decentralized community management). I anticipate that entire decentralized studios dedicated to virtual beings will form over time, whether that entails DAOs managing individual and collective personas or operating as service DAOs for those wishing to integrate virtual beings into their projects.

CULTUR DAO is the most prominent decentralized studio for virtual being creation, billing itself as the Web3 Pixar. Via a token-gated Discord, a group of artists, technicians, and evangelists use shared financial incentives to drive innovation in the field. Committees are formed around avatars, artificial intelligence, speech synthesis, story creation, and experimentation with existing AI NFT infrastructure such as ASM’s brains and Alethea AI’s personality pods. Their recent hackathon produced a myriad of interesting projects including Decentraland AI Chatbots, a VB ballet, talking Cryptopunks, AI character memory tests, and Kittygotchi — a tamagotchi-like virtual cat.

The Synthetic Media Social Layer

Synthetic media — AI-generated, personalized media content — is proliferating, and virtual beings are a further extension of this social media layer. While virtual beings may feel foreign now, it isn’t hard to fathom that within a generation or two they will be commonplace and socially accepted. Deepfakes, social media bots, and anthropomorphic smart devices all already contribute to a significant smearing of the borders between the human and the synthetic. This will continue to evolve as younger generations interact with virtual beings more regularly. Today, almost 15% of virtual influencer audiences are younger people between 13–17 years old — double the average of regular influencers, whose young audience averages about 7%.

The initial stages of this synthetic virtual future is already here, captured by news of humans marrying holographic representations of virtual beings, dating applications like Hybri and dedicated virtual companionship platforms, virtual influencers holding more influential prowess than human counterparts, growing trust in AI, etc.

Web3 technology provides an additional access point for the creation and management of these synthetic media experiences. Ownership layers enabled by NFTs, decentralized governance across a myriad of stakeholders over the Internet, revenue models not possible in previous iterations of the web, the permanence of the blockchain, and more all open up a new frontier in synthetic media.

Crypto technologies, decentralized internet organizations, and virtual worlds are the foundation on which a new synthetic media social layer for reality will be built.

If you are building anything in web3 at the intersection of artificial intelligence, digital fashion, avatar infrastructure, computational layers for rendering, or virtual beings, please reach out. I’d love to speak with you.