Breaking New Ground: The Innovation Behind Masterpiece Studio
Lead R&D Po Kong Lai dives into deep learning and unlocking 3D creator potential
Imagine a world where anyone can create whatever they want. A world where creatives can effortlessly design their dreams in 3D without the traditional barriers of training or expensive tech. Sounds like an amazing vision, right?
Well, at Masterpiece Studio, that vision is becoming a reality. Thanks to the innovation from the R&D team and the leadership of computer science expert Po Kong Lai, this group is constantly experimenting on the cutting edge of 3D content creation.
Keep reading for an insider look into Po’s groundbreaking work and how it’s about to unlock a whole new world of creator potential in the metaverse.
Meet Po!
Hi, my name is Po Kong Lai and I lead Research & Development at Masterpiece Studio! That means I develop a lot of prototypes related to 3D asset creation. From geometry processing to machine learning, we design things to make that 3D asset creation portion easier and more intuitive.
From a business perspective, our R&D team de-risks the technologies that could give us an edge; from the tech perspective, we explore new methods and determine which ones we should focus on.
Can you provide some background on the work you’re doing with geometry processing and deep learning prototypes?
Geometry processing relates to anything that deals with 3D models– often represented as meshes– which are 3D structures composed of points, lines, and polygons. Combined, these elements work together to make a 3D surface, and geometry processing gives us the language and frameworks needed to manipulate those surfaces. For instance, at Masterpiece Studio we might ask, “how do we mold a sphere into a sculpture of a face?”
Deep learning is now being used in mesh processing as well, but sometimes we take in matches as input and want to do something that’s not as well defined.
To apply deep learning for meshes, the trained model can take in the entire mesh (or just elements such as vertices) as input. For example, if you’re using a smart brush and you want to grab a region of a mesh to deform, traditional geometry processing would provide an equation that describes how to move the elements in that region. With learning-based methods, you provide a bunch of data and then from the data you would train a model that learns what type of motion the user intends to do. With the traditional approach, you will need domain knowledge as well complex specialized equations. With a learning-based approach, you need a dataset but the output can be much more natural since it’s not confined to a set of handcrafted equations.
However, deep learning — at least for our field — doesn’t only stop at meshes. One of the ways to get really detailed color on them is to use texture maps, which are basically just 2D images. If you imagine the surface of your 3D model as being composed of a very thin paper-like material, and if you have a way to cut it into pieces, you can flatten those pieces onto a table or plane — whatever you want. Once it’s flattened, that’s an image; that’s why we can essentially “paper mache” high-resolution images on top of the 3D surface. By applying image-based deep learning methods on those flattened pieces, we can obtain new pieces which can have highly stylized color patterns. This, in turn, stylizes the color appearance of the associated mesh. So for us, deep learning is a bit more open-ended that way. In general, if we can get some data that might be related to the 3D asset, we process it and output some more data that might be more relevant to the artist.
Why is your work so innovative and exciting?
In terms of innovation, this is the first time in human history that powerful VR is accessible to the general public. VR isn’t really a novel thing in the sense that we’ve had this for many years, but it was limited to research labs like the CAVE system. In it, you’d wear these 3D goggles and interact with large projector screens — but that’s a $2 million room. Your average person isn’t able to access those types of facilities, so it’s exciting that this powerful VR is now available to the consumer at a fairly low price point.
It’s also the first time we have easy access to VR with controllers instead of only headsets. With the advanced image sensors we have now, we can even track our hands in 3D space without the use of manually placed markers. It’s a totally new concept to have people manipulate data in 3D space this way, especially without years of training.
Traditionally, you would work on a 2D interface, learn how to think in 2D and then put it back into 3D — but now, it’s much more natural to work with 3D from the get-go. People can visualize their ideas in 3D, create it, see it and share it like they’ve never been able to before.
How does your work help unlock creator potential in the metaverse?
More people having access to this tech means more people creating, so I think that’s a very straightforward answer. Our challenge is to focus on the UX side to make 3D creation very natural, because we have the capability — but making it usable for people is a big challenge. For the R&D team, this means focusing more on algorithms that reduce time and complexity for any tasks that the user wants to do.
Current 2D interfaces on computer screens have a lot of real estate. People have been trained to select very small fonts and buttons there, but in VR, it’s much more difficult because it’s hard to predict where people will look. For example, you can’t put buttons too high because people might not look up to find it. It’s hard, but it’s interesting work.
Our goal at Masterpiece Studio is to make the user experience as natural as possible. This will unlock tons of possibilities in the metaverse and beyond, as creators will be able to go in with their hands, design what they want and use that asset in games or wherever else.
Do you have any predictions for the future given the new ground you’re breaking right now?
I predict we’ll have much more machine learning. At the moment, machine learning for 3D is in its infancy. If you think of machine learning now — your phone detecting faces and so on — that technology took many years to mature.
Plus, once deep learning started to get more popular, we just started paying people to label data for us. It’s much easier to label data when it’s images because you’ve got the natural interface and drag-and-drop region of interests. You simply type in what you think is inside that box, and you’ve labeled an image. However, labeling data in 3D is much more challenging because it’s a bit of a chicken-egg problem.
We don’t have natural 3D interfaces that enable anybody to label 3D data, which means it’s a very specialized task. It takes quite a bit of money to hire someone to create 3D objects and label them. The other challenge for 3D and machine learning is that it’s unstructured. For images, it’s fast because it’s in a grid system — at whatever point in the image, there’s going to be an up, down, left and right. However, with meshes and 3D data, one point on a mesh can be connected to two triangles or a hundred triangles — we don’t have that same known size, so it’s a lot more difficult to save on space and processing time.
I think for the future, machine learning is going to take on a much bigger role as people get more data and join the metaverse. A lot of algorithms nowadays have lots of custom parameters, knobs and levers to tweak — so as we learn about what people like about those features, we will have more UX and machine learning that learns which parameters users will typically like given a particular use case.
Any closing thoughts for big dreamers who might want to join the R&D team?
The work that we do is innovative and unique. If you join Masterpiece Studio, you’ll have quite a bit of freedom to explore what you would like to do, as long as it’s in the general realm of making 3D asset creation easier and more intuitive.
This job isn’t about being stuck in specific roles — our team is always doing open-ended research, which means we’re trying to discover the best strategies. If you have new and interesting ideas, we want to hear them.
***
Take your career to a new dimension with our rapidly growing team!
Working for Masterpiece Studio means building the future — the projects you will work on here will be ones you won’t want to stop talking about, researching, and exploring every chance you get.
If you’re interested in living on the cutting edge of tech, art, and business theory, joining our Talent Network is a great way to learn more about the new digital world, to get to know our brilliant team and to stay on the pulse of Masterpiece Studio news and job opportunities.
