Digital Fashion, Art, NFT, AR, VR, WebXR, 3D Deep Learning, BCI, and the future native digital economy.
We interview Emma the CEO of Digitalax, to learn why NFT underpins Digital Fashion, Art, and how this ties into WebXR, Virtual Reality, 3D Volumetric Capture, Lidar RGB Fusion, affects future Neuroscience research, defines future digital economies, the tools, the news, and some research.
Digitalax X SVGN.io News
Written by Micah Blumberg with major contributions from Cecile Tamura and Emma-Jane MacKinnon-Lee from Digitalax. https://youtu.be/c_CgElxhwaA
Introduction
I follow many artists on social media, and one day on of my favorite artists, a Virtual Reality Artist named Anna Zhilyaeva (who is world famous) posted that she had sold her first ever NFT for $24,593. For the VR communities on social media this was big news. You could make a painting in Tilt Brush and auction it as an NFT for potentially thousands of dollars!
Across VR social media groups where I re-shared Anna’s news the response -seemed to be enormously positive, with many people congratulation Anna Zhilyaeva on her excellent work, commented on how well deserved it was for her to have had such a big success with her first non fungible token sale.
A brief history
The first time I heard about this coming convergence between NFTs and Virtual Reality was at GDC the Game Developer Convention in April of 2019 in a talk called Blockchain Game Development and ERC-1155. I sat in the audience as Enjin introduced this new protocol to the world.
Shortly after that point in time I started to have discussions with people in San Francisco, about how we could use WebXR libraries like Aframe and make it possible for people to bring in game items with ERC-1155 (a non-fungible token protocol)
What is an NFT?
The big idea is that you could receive an item from a game and you would get to keep that item outside the game. In the game Legend of Zelda for example you may get a sword, but you don’t get to keep it, when the game is over the sword is gone, trapped in the game. On the other hand if you were to get that item with NFT that would mean that you owned it and you could keep it even after you left the game.
NFT answers the question of how do you attach authenticity, verification, and digital ownership to an item. That could be a digital item, or a physical/digital hybrid item and how can you attach ownership to that?
Emma explained that in the past eight months the NFT market has exploded. A lot of artists have recently moved into the space to learn about NFTs and how they can use them to monetize their own work.
About Emma-Jane MacKinnon-Lee and Digitalax
Emma (age 22), the CEO (and in my mind a genius) at Digitalax, has always sat within the hybrid of math & science on one side, with a lot of art & creativity on the other side. She was really good at math, science, and biology, but also greatly moved by art and fashion. She has spent time just looking at the future, trying to imagine future industries, what does the world look like in 10 years, 50 years, or even in the year 2100 she has contemplated. One of her big answers is that Gaming, VR, AR and 3D content are going to play a big role in earth’s future that is leading to native digital economies.
“If this is the future”, she asks, “what are the details of that, what would that look like? What are the interesting market segments that will be really big and generate a lot of value for a lot of people?”
For Emma the answer that rang out was Digital Fashion. “Digital Fashion is undervalued at the moment if 3D VR AR is the future. People will start expecting that the digital fashion industry exists.”
“We started focusing on the designer side, how to bridge the NFT market, where is the value coming from and how do you make that sustainable? A lot of it is focused on cosmetic side, [but] digital fashion has to have application utility for using this kind of asset. How can these digital fashion assets be integrated into game and VR experiences? How about indie e-sports or mod e-sports? How can you bring identity to the players?
Digital Fashion
What we are talking about with digital fashion, and what we are talking about in Emma’s case includes 3D models of dresses & textures, an artist or fashion designer can create something that is really sort of a costume for your virtual character in one sense.
Today NFT underpins Digital Fashion, buyers can actually own these items in the digital world, knowing that what they purchased is backed on the blockchain and secure. Then both the buyer and the wearer can really have a new emotional experience with their item(s), with a sense of connection and a sense of ownership from the digital fashion items that are now truly their own because the proof of that ownership is backed by the blockchain.
Digitalax has been alive in this Non-fungible Token / Digital Fashion market for two and a half to three months, taking the perspective of bringing digital fashion into the space, before that it was just artwork dominated. Digital Fashion is going to play a huge role in 3D content industries in the future, gaming, VR AR identity and self expression.
The Player-Creator Economy began in Japan long before NFTs arrived.
Cecile Tamura pointed out that the player creator community has been going on in Japan for a long time, and that goes back long before NFT’s and blockchain arrived on the scene.
Cecile shared with me that the Player-Creator economy is a billion dollar industry and growing, its been a big part of Japanese culture for a long time and it has been exported to other countries and keeps growing.
Fate/Grand Order is a great example of how big this economy is, it is a free to play RPG game, in Japan, that features in app purchases and has made billions.
“Fate/Grand Order from Sony’s Aniplex continues to be one of the world’s most lucrative mobile games, having just surpassed $4 billion in lifetime player spending according to Sensor Tower Store Intelligence estimates.”
Cecile also wrote saying “There’s a cascade of news coming out yesterday about Genies partnering with Unity to build its virtual avatar world. It also includes their fashion. And they are starting to talk about NFTS. With traditional retail sales down during the global pandemic, fashion brands like Gucci and The North Face have recognized this new market and entered the space. The North Face offers in-game gear for Pokemon Go while Gucci has partnered with Genies to offer apparel and accessories for its celebrity avatar clientele — all of which can be purchased by the consumer to accessorize their own alt-persona.
The North Face offers in-game gear for Pokemon Go
Snap Partners with Ralph Lauren to create Bitmoji Wardrobe
Genies partnering with Unity
Cecile also wrote: “To track the evolution of fashion and avatars, look at the concept of the digital runway show. Initially a novelty to help brands promote their lines while under Covid-19, we saw elite fashion brands use avatars to showcase their real world designs. Then the avatar and digital goods market began to grow. Now, we’re seeing an influx of apparel brands presenting similar digital collections as before, BUT the apparel on display can be purchased both for human and avatar.”
Rise of the Avatars: Their Future in Gaming, Fashion and Music
Genies will let consumers create their own 3D avatars with Giphy and Gucci
CryptoGenies? Digital Avatars Are Coming to Dapper’s Flow Blockchain
So what sets Digitalax apart?
My argument is that Digitalax is going above and beyond these other companies with a multi-layered approach that focuses on the end goal of bringing application utility to these 3D NFT Digital Fashion assets, and that has to include building the technology to not only bridge artists work with 3D applications, but also the infrastructure that leads to the native digital economy.
The three layers of the Digitalax strategy:
Layer 1
I would describe the first layer of Digitalax as being about the Content Supply Chain, from creation to distribution, with ERC protocol standards, this is in part about Digitalax’s partnerships with Fashion designers, to create really unique digital fashion garments. It’s about digital ownership, full IP assignment and rights, to these 3D files, to the FBX, to the 3D asset, to the gif, digital fashion. How can you streamline the idea of a digital fashion? Digitalax has answered that and gone beyond answering that to actually working with designers who minted their textures on chain with Digitalax’s help in a concept Emma created called Fractional Garment Ownership.
Within Layer 1 we’ve got blockchain smart contracts, working with artists to making sure they can sell their stuff, everything from their 3D models to their textures is tracked, if someone contributed a texture then when it goes to auction everyone can get paid because everything is tracked.
To start with an artist or designer could use a 3D model creation program, plausible examples might include Blender, Maya, Cinema 4D, 3DSmax, Houdini, you might get a 3D model out of that.
A 3D artist may want to also think about rendering it really nicely with products like Renderman or Octane. With a render farm like Octane an artist can ray trace all the light rays to create the most beautiful realistic art.
RNDR or Render Token, 3D artists and 3D animators can think of Octane as a company that provides a render farm, with Render Token people can loan out their unused GPU time to that render farm, and get rewarded for that with RNDR tokens.
“RNDR gives GPU owners an opportunity to monetize otherwise idle GPUs by offering their compute power to creators in search of rendering resources. In short, RNDR allows GPU owners to loan out their GPU power to creators in need of additional power.”
Otoy also created a file format for 3D files called ORBX, which is the foundational technology that led to the ITMF specification and the Immersive Digital Experiences Alliance.
“The ITMF specification is based on ORBX® scene graph technology, a format created by OTOY and now supported by dozens of software systems used in 3D animation and game development. Starting with the ORBX scene graph format, ITMF is designed as an interchange and distribution format for conveying high-quality, complex image scenes to immersive media displays over commercial networks. Applications range from passive viewing, gaming, and telepresence, including six degrees-of-freedom (6DoF), to next-generation light field displays. Recognized for its potential to deliver an immersive, true-to-life experience, light field is now considered the richest and most dense form of visual media, thereby setting the highest bar for features that the ITMF will need to support and the new media-aware processing capabilities that commercial networks must deliver.”
Which I mentioned because for me that is another connection back to Emma’s company Digitalax, and the second layer that Emma is working on which is about her development of the open source 3D file format specification called DASH.
Layer 2: Dash File Format Specification and File Intercommunication Architecture.
Dash or the second layer of Emma’‘s strategy’ is what interests me the most out of everything that Emma has been doing (and this article doesn’t cover all of it) because it enables the interoperability of the Player Creator system with the Artist-Fashion Community, by solving a long standing issue with 3D file formats. Dash aims to solve a problem facing 3D artists, game & web developers alike with a novel approach that uses opinionated logic, and Fourier transforms to create an accurate & compressed representation of a 3D model in it’s original environment (such as a 3D program) and transfers it’s appearance accurately to another environment (such as a game engine).
Read about Dash:
Revision 1.5: DASH File Format Specification and File Intercommunication Architecture.
In the video I referred to the Fourier Projection Slice theorem also known as the Fourier slice theorem.
Layer 2 is meant to answer the question: what is the technical innovation needed to bridge these 3D assets, how can you bring them in applications, where they are rendered correctly, and are usable.
Emma is creating Dash to answer the second part, it’s a new file format that solves the problem of 3D file interoperability in a smart way.
Emma created the Dash File Format to be an open source standard for enabling the interoperability of digital fashion assets, and other 3D items, maintaining the accuracy of their appearance with opinionated logic, and maximizing their compression size and speed with the fourier transform.
3. Layer 3: The Player Creator Economy
This is leading to third layer of Emma’s strategy, creating the player-creator economy, building the metaverse from Snow Crash, the Oasis from Ready Player One, the Matrix (film 1999), Sword Art Online, Gun Gale Online, Accel World, and so on.
Layer 3 is about The Application utility: building an industry: Sustainable value generation. An example is PODE. PODE is a Perpetural Ownership and Digital Economy based on NFT. That you can access to for personalized experiences across gaming, VR, and other live 3D content applications.
You can begin to develop your own applications that connect to the Pode economy, and if you are interested in that it’s time to get familiar with game engines and webxr scripting frameworks.
Player creator developer tools: Game Engines & WebXR
With the aid of game engines like Unity, Unreal, Godot, and WebXR tools like Aframe, Threejs, Babylonjs, Playcanvas, React three Fiber you can help build the future digital economy.
Games made with game engines like Unity, Unreal Engine, and Godot can either be streamed to the browser, or imported into a webpage via wasm to run on webgl with the WebXR standard.
WebXR https://immersive-web.github.io/webxr/
Unity
Aframe could be the fastest way to create a super simple AR or VR application on the web, it wraps around threejs.
Threejs is high level scripting that interfaces with WebGL
Babylonjs is intended to be more of a game engine, it’s supported by Microsoft.
Playcanvas, is a different approach to WebXR because it has its own built in hosting solution.
Exokitweb integrates multiple platforms, so every WebXR page can interface with every other WebXR page, regardless of the platform, library, or framework that they were created with.
React Three Fiber (r3f) is an astonishing new tool that makes it so easy for React developers to start integrating threejs graphics into their web pages that I can easily see companies like Facebook and Twitter using r3f (React Three Fiber) to make 3D versions of the most popular websites (like Facebook, Twitter)
Janusweb is a different approach to WebXR in that its more like a world that you help create.
Mozilla Hubs
NFT Based WebXR Platforms
WebAverse is also a world but it’s different for a few reasons.
“In Webaverse, you can package and load in an existing 3D website written using THREE.js, A-Frame, Babylon, or any other WebXR compatible JavaScript framework”.
“Since Webaverse is built with NFTs, any creation in Webaverse can be bought, sold, and traded on open and decentralized marketplaces like OpenSea.”
Cryptovoxels is a virtual world powered by the Ethereum blockchain. Players can buy land and build stores and art galleries. Editing tools, avatars and text chat are built in. It’s built with Babylonjs.
Decentraland’s virtual world was first built with Aframe, then it was rebuilt with Babylonjs, and then it was rebuilt with Unity. Each time they claimed that the change was for performance reasons. The reality however is that all the game engines and webgl webxr apps seem to face the same kinds of challenges for rendering performance. (outside shaders, but webgpu fixes that.) While Unity offers a nice toolset for both beginning developers and professionals I believe that all of these other tools have the capability to be as performant as Unity is in the hands of an experienced software engineer with ninja level coding skills.
There are several WebXR service providers offering something like a proprietary closed source subscription based pricing model like @MetaVRse @the8thwall @ZapperApp and in my opinion the pricing models seem to be thinking about huge corporate customers. WebAR and WebVR applicatoins are in demand by Fortune 500, and Fortune 100 brands.
Beyond these three layers interest in NFT, and crypto currency is growing in part thanks to recent news.
For example the other day the Price of Bitcoin reached a new all time high $48,226.25 shortly after the news that Elon Musk’s company Tesla had bought $1.5 billion in bitcoin, and that Tesla has plans to accept it as a form of payment.
Ethereum, what NFT protocols are based on also all reached a new peak high at 1786.28.
Conclusion
One of the things I like the most about Emma’s company Digitalax is the Dash file format specification. Interestingly she appears to have coined the term Material Instance Segmentation, in a paper I read that explains part of how Dash works
One research topic that this reminded me of is the topic of 3D semantic segmentation, are you familiar. I first learned about it via a talk by Or Litani in San Francisco years back https://orlitany.github.io/
Like Material Instance Segmentation (which is accomplished with opinionated logic instead of neural networks) 3D semantic segmentation is a niche inside the topic semantic segmention, and a niche inside the topic of 3d object segmention, and those are niches inside the idea of using neural networks for capture or rendering, and those topics are niches inside the topics of capture, rendering, and deep learning respectively, and deep learning is a niche inside machine learning, rendering is a niche topic inside the computer graphics industry, and so on.
I suspect that very few people have climbed this far atop this information technology stack, and that’s just part of what Emma is building.
In a previous article Cecile Tamura and I interviewed Urho from Varjo about their Mixed Reality headset that integrates Lidar & RGB in a fusion algorithm to capture the real world and turn it into a cinematic object at 90 frames per second.
The best news is that the science behind the technology under discussion is real, the research out there that you can find backs up the science behind the concepts that Emma is talking about and the concepts behind what Varjo is talking about. Here are a few references to science articles that I dug up.
“Deep neural networks have made good progress in 3D shape reconstruction owing to their powerful ability to extract priors from big data [18, 15, 21, 9, 6, 27, 12]. However, high-resolution 3D shape reconstruction is still challenging due to the cubic growth of computational cost.” (…) “The thickness map is the 2D spatial projection of the 3D shape, which is easily predicted from the input image by a general convolutional neural network.” (…) “We have exploited the Fourier projection slice theorem and introduced the 2D thickness map which can reduce the domain gap between the input image and 2D slices. A deep network was built to predict the thickness map from the input image by exploiting the edge and silhouette constraints. This network allows us to predict fine details (edges) and global shape (silhouettes) of thickness map separately from the input image, which allows a more accuracy reconstruction result.” https://openaccess.thecvf.com/content_CVPR_2019/papers/Shen_3D_Shape_Reconstruction_From_Images_in_the_Frequency_Domain_CVPR_2019_paper.pdf?fbclid=IwAR32vXasRpZ3yWNs4TkKW5nxuL_FDMx90iho3bUFJXIvXldJtkCSt5dNHuU
“(…)can also be used for the accurate semantic segmentation of a 3D LiDAR point cloud and how it represents a valid bridge between image processing and 3D point cloud processing(…) Finally, we demonstrate that this architecture is able to operate at 90fps on a single GPU, which enables deployment for real-time segmentation.” https://arxiv.org/abs/1905.08748?fbclid=IwAR2bd4FB2hjhBJUsc6sPQhPKfCnmHS5wvX4FltFm6ZJdnqUZ3LcrGUPoPn0
“PointSeg: Real-Time Semantic Segmentation Based On 3D LiDAR Point Cloud
YUAN WANG et. al.
+
Related Papers” https://www.paperdigest.org/topic/?paper_id=arxiv-1807.06288&fbclid=IwAR1fOAdBo9ml7X3YpZ1ngx-pSk199GK5GKWGudB1idLzSi7zl_3N75eYspQ
A preview of that future can be seen in this article which is about using two Multi-modal-neural-network-models for 3D Semantic Segmentation, one for capture and the other for rendering.
We should expect that 3D Semantic Segmentation (Deep Neural Networks) will be integrated into Lidar RGB fusion algorithms on your iPhone, so anyone can easily create fantastic 3D models with something as easy as a phone because of the convergence of technologies.
Being able to store these new 3D models in a compressed way is where Dash can play an important role.
This also affects the future of neuroscience in terms of the study of neural correlations, because when you have semantic segmentations of the real world, you can use that data with deep neural networks and affective computing combined with biosensors and new medical imaging technologies to take the research of studying neural correlations to a new level.
Join Emma’s discord
Join Cecile and I in some groups we admin to learn more, share your ideas, and participate in the conversation:
The Neurohaxor WebXR discord
