DESIGN PRACTICE INNOVATION — kalmdesigns.com
Filling the Metaverse
To make and then keep the Metaverses interesting, development and creative opportunities abound, and Gen AI is making inroads.
The metaverse concept has its ups and downs and is either near…or close-ish…or maybe so far in the future that it won’t happen in your lifetime.
Let’s explore how content creation and Gen AI intersect.
So, metaverses are coming (or “Are Metaverses coming?”). Either way, let’s talk about the opportunities metaverses create for developers and creators because one of the things that will determine whether metaverses are successful will be content, as the following quotes indicate…
Companies like Meta will try to define presence — What can be seen and what can be interacted with, but individuals, groups, and companies should and will create their own platforms that provide the level of interactivity that makes their audience comfortable.
For instance, while Zuck is trying to dominate the idea of the metaverse (renamed Facebook to Meta after the concept of the Metaverse), one of the people with the strongest vision of the metaverse, Neal Stephenson, is working on creating a more inclusive platform for it.
Additionally, experiences such as Minecraft and Roblox are metaverses — with Robolox already experimenting with Gen AI to help users create content via text description.
So…for the foreseeable future and possibly forever, there will be multiple metaverses.
Usability drives adoption
One of the things holding back adoption is the mixed-reality headset hardware issues — narrow field of view, perception issues, view quality, lag, etc. However, with ARKit from Apple and ARCore from Google, the distribution channels for mixed-reality are broadening from headsets to phones and tablets. While mobile phone and tablet screens are not as engaging as headsets, they are ubiquitous — and with more universal engagement, more content is needed for the masses expected to be using them and then the headsets expected to follow.
Once hardware viewing platform quality is high enough, metaverse adoption will be driven by whether there is enough content to feed audiences.
Content creation pipelines
Currently and for the immediate future, companies are generally creating targeted content for specific purposes — movie studios creating extensions of stories, marketing firms creating experiences to sell products, museums creating educational experiences, and companies creating learning experiences for complex machinery and situations.
Schroepfer replied that there are basically three problems facing VR adoption: cost, friction and content.
…
Content is still thin on the ground, though. There have been some hits, like Superhot and Beat Saber, but nothing to really draw crowds to the platform (if it can be called that). — Coldewey 2019
To ensure the content is there as interest grows, content must be easy to create and import.
Creating content, whether from capture, creative platforms like Unity, Unreal, Blender, etc., or CAD, or via computationally created content, needs easy-to-use pathways into the system. Going forward, technologies that enable tying content to reality by starting with reality and overlaying artistic direction can be another pathway to enable engaging mixed realities. Generative AI is playing a role in many of these pathways. While much of the content is created for gaming and advertising, other avenues like walk-in-others-shoes experiences, meditation venues, and patient stress management show additional experiences or destinations that virtual reality enables.
A large amount of content must be continuously created to create a large-scale mixed reality that can provide enjoyment and compelling experiences that draw users.
Right now, most consumers probably don’t feel like they are missing out — and that’s unlikely to change until there’s decent content, games or entertainment that offer a signficantly better experience than the existing options. — simplywall.st
For successful adoption, the content creation pipeline must be kept full.
Considering Content Creation
Creating simple objects in a scene that doesn’t interact with reality is relatively easy. Creating a realistic or cinematic experience that does interact with reality requires effort.
Capture
Creating objects for 3D applications via cameras or LIDAR represents a way to capture our current reality in context. LIDAR and camera-based tools create point clouds that must be turned into polygons or other geometry that 3D programs use.
It would be nice if, when you wanted the Leaning Tower of Pisa, you could SLAM (Simultaneous Localization and Mapping) and put it in your scene, but a lot of manual effort and other work is still needed to get a representation that is the right level of detail.
While there are many tools for converting the results of these tools into usable content, there are still technological hurdles to common use. As with other spaces where cognitive complexity is being tamed by Generative AI, the pipeline from capture to usable 3D objects is quickly transformed.
CAD
A huge resource for content is companies that create physical products. They have computer-aided design (CAD) files for their products — buildings, furniture, appliances, cars, luggage, watches — everything manufactured that can be turned into content. The CAD connection for companies with physical products should allow companies to integrate versions of their products into environments.
Tooling is needed to adjust the CAD files for commercial use — reducing the level of detail (LOD) and taking other actions to maintain intellectual property (IP) security.
Once pipelines can be created that companies can easily plug into and feel safe pushing their content to creating spaces filled with products we know or want to learn about, it can become much easier. This will help engage companies to make it easy for creators to use their products by creating environments where their products are highlighted.
3D Creation
Often, content creation will fall to the content authors as CAD files only exist for manufactured goods, and scanning is still difficult.
Simple online tools like Spline provide an introduction to creating content.
There are many tools for creating sophisticated 3D content, from free tools like Blender to paid tools from Autodesk and many between and beyond.
Authoring content through these tools requires dedication. Creating great content requires artistic ability and dedication.
Objects & Environments
In addition to letting users create objects and environments edit-by-edit — For example, starting with a cube, duplicating, and manipulating to some complex result — Blender has Geometry Nodes, Rhino has Grasshopper, Houdini’s is a procedural system, and so forth. These tools let creators build systems instead of just objects or environments. Unlike hand-built objects, systems can react to changes, so when layered with the user's view, they could adapt to the current experience instead of being a static one.
As a creator, you need to watch how AI and various geometry creation tools mentioned above change your work from creating objects and environments to creating styles and systems that interact and integrate within platforms like Unity, Unreal, and other systems used to distribute metaverse content.
Animation & Interactivity
In addition to creating environments, making it easy for non-technical people to assign actions and non-pre-canned-looking animations to their creations needs to be addressed.
But assume that GenAI will be trained on the movies, shows, and other content to see how and when objects and people react and be used to assign “base” interactions — like people walking around, chatting in a cafe, etc. Generative tools can map real-world motion to generated figures, faces, and sci-fiction or fantasy creatures.
Gen AI will take care of the details while things where story and emotion are conveyed, like creating stories and the logic of what will happen when will likely still be something humans will continue to do.
Special FX
Nebulae, explosions, destruction, magical construction, and other less-than-unusual effects are other places where human imagination can shine. While GenAI may soon be taught how to do basic effects, creating new ones that impress and engage audiences will be a desired ability to keep the pipeline full of content that audiences want to experience.
Universal Scene Description (USD)
A schema format for 3D, open-sourced by Pixar, that is used to drive interchange between different 3D abstractions. Learning about its abilities to bring together different workflows and technologies will teach you a lot about 3D pipelines and how and where you might contribute.
With Gen AI touching each process and across processes
As mentioned above, Generative AI is changing all aspects of 3D content creation, from making the transition from capture to usable content easier to automating animation and content creation.
Augmented reality and beyond
Augmented reality will use the same tools as virtual reality but requires a somewhat different mindset since the results need to work with the real world instead of in place of it.
“More importantly, in the context of the disconnect between digital and physical worlds, cognitive load also depends significantly on ‘cognitive distance’, that is, the gap between the form in which information is presented, and the context in which it is applied. For instance, when following navigation instructions from a dash-mounted in-vehicle satellite navigation system, the driver must consume the information presented on the screen, retain that information in working memory, translate the instructions onto the physical world in front of them, and then finally act on the instructions, all while still maintaining control of the vehicle. The resulting cognitive journey creates a significant cognitive load. This is why we are seeing Head-Up-Displays (HUDs) now being used to overlay the relevant data and guidance where it is needed — by placing information in context and superimposing that information onto the physical world by way of a projection on the vehicle’s windshield, this reduces cognitive distance and therefore minimizes cognitive load.”
In the future, we can see creating cinematic alternate realities that require the effort of styling not just scenes but all of the content that can match objects in an environment the user might physically be in — i.e., if there is a chair in the user’s physical environment, the mixed-reality needs to know how to replace it.
AI tools like Scenimefy can make this easier by taking footage and giving it a specific style. This reduces the need to create 3D content.
Intellectual Property Management
Companies need a way to protect their intellectual property (IP) but make it available for use in appropriate venues.
- How do manufacturers quickly and safely create content that lets authors showcase their products but maintain rights to the forms and functions with their logo on them?
- If a creator wants to make a house as part of their section of the metaverse, would companies want to control the choice to put a Wolf range in the kitchen over a Thermador?
- If the creator’s story is horror-themed, would a company not want to let their products participate?
LOD
A great metaverse experience will have parts of the pipeline from content to user that can adjust the level of detail to match the capabilities of the user’s environment and equipment. Creating and transporting high-definition models will be an issue in low-band-width environments.
Content creators may want to decide if they want control of LOD or whether they completely abrogate it to the platform. Metaverse platforms could charge people for LOD, like streaming services charge for higher-resolution feeds.
Payment & Ownership
NFT-like (Non-fungible Token) tools might be able to make companies’ assets available with certain restrictions. This would mean plumbing the greater metaverse architecture to understand how assets are tagged…and to respect that tagging.
End Notes
Content pipelines and how content flows through them will become more important as metaverses grow. This article has touched on a few technical and creative areas that need to be created or enhanced to get content from authors to viewers.
Generative AI will make it easier for more people to participate in content creation. There will likely be a future where artists can (somewhat) easily combine their style with almost any supported experience — shopping, cinematic, meditative, etc. creating new avenues and opportunities to sell or share their style.
People in different roles will migrate their work and skills to the medium, and people with different backgrounds will create great diversity in how and why content is created. Providing services that help people from different backgrounds to create content will be a series of opportunities.
Companies are working to tie all of the technologies together, and there are many opportunities for new companies to create content, manage a part of the pipeline, or provide services that fill, manage, or extend it (or them…again…multiple metaverses).
The opportunities across the pipeline for individuals and companies to create, manage, and distribute content to metaverses when and where they become available can be yours…or someone else’s.
Introductory Articles
Resources
7 things to know about the augmented reality landscape. Dean Takahashi. 2019.
How Oculus squeezed sophisticated tracking into pipsqueak hardware.
Devin Coldewey. 2019.
AR/VR: Beyond Gaming to Enterprise Adoption. Ethan Harden. 2018.
The Creative Pipeline for VR Developers and Artists: An Introduction. Bob Duffy. 2018.
The Venture Reality Fund. 2017.
Marketing in the Metaverse and the Need for Consumer Protections. Louis Rosenberg. 2022.
