Introducing Beeble: Mobile Virtual Production Platform for Content Creators
Hello, I’m Hoon Kim from Beeble.
Beeble creates a mobile virtual production platform for content creators. In this article, I would like to introduce Beeble with 10 questions and answers.
Q1. What is Virtual Production?
Virtual production is a technique of compositing real objects into CG backgrounds in real time.
What do the movie “Avatar”, “The Lord of the Rings”, and the sci-fi series “The Mandalorian” have in common? The answer is that they were all filmed using virtual production.
Virtual production (VP) is a technique of compositing real objects and computer graphics in real time. For filmmakers who had to go through labor-intensive CG work every time after filming in front of a green screen, VP, which enables real-time viewing of CG-composited images, has been a dream for a long time. Film directors such as George Lucas (Star Wars), Steven Spielberg (A.I.), Peter Jackson (Lord of the Rings), and James Cameron (Avatar) have been using VPs since the early 2000s.
VP, which enables filmmakers to shoot without time and space constraints, began to spread rapidly during Covid-19 when filming at overseas locations became impossible. In particular, ever since Disney filmed The Mandalorian with VP using LED Walls and Unreal Engine, it is reshaping the film/TV industry.
Q2. LED wall? Unreal engine? How does virtual production work?
The principle behind this technique is to display a real-time virtual background on an LED wall and place a real subject in front of it to shoot together.
VP using LED walls is an integration of cutting-edge technology that was not possible 5 years ago. The principle of this technique is to display a real-time virtual background on a large LED wall and place a real subject in front of it to shoot. Okay, then how do LED walls create the effect of actually being there?
1. Photorealistic background: If the background isn’t realistic, the immersion will drop drastically, right? VP utilizes a 3D game engine and powerful GPU servers to render a photorealistic background. This real-time rendered background is displayed on a special high-resolution LED wall.
2. Believable Lighting: Even with a realistic background, if the lighting on the real subject doesn’t match the background, it wouldn’t look natural. The powerful LED wall provides precise lighting that illuminates subjects as if they were actually in virtual space. Just like the stormtrooper’s armor glowing bluish in the gif above.
3. A background that matches the camera’s viewpoint: Wouldn’t it be awkward if the camera moved closer to the subject, but the background remained the same? To match the viewpoints of the camera and the background, the camera viewpoint is recorded using a tracker, and then the background viewpoint is updated accordingly.
Q3. What are the advantages of virtual production?
VPs offer unique advantages of efficiency, real-time, and creativity.
- Efficiency: VP saves time and money spent on travel, transportation, and finding a location. As long as you have a premade background, you can shoot anywhere in the world. The time and weather can also be changed with just a few clicks.
- Real-time: With VP, you can see the final result while you shoot. Actors can immerse themselves in a virtual environment rendered in real-time, and the production crew can be creative without worrying about post-production.
- Creativity: VP enables creative shooting techniques that were previously impossible or too expensive. With VP, you can shoot the Earth from a space point of view or make a quick sweep of the Himalayas without an astronomical budget.
Q4. I am a YouTuber. Can I use Virtual Production?
Unfortunately, virtual production is out of reach for general content creators.
To successfully run a VP studio, you will need large LED screens, camera trackers, and rendering PCs, as well as a space to build your studio and operational staff. Thus, building one costs millions of dollars, for example, Disney spent more than $100 million building virtual production. Unfortunately, VP is out of reach for general content creators.
Q5. YouTubers also do real-time composition, how do they do that?
Individual creators rely on green screen and chroma key techniques.
Individual creators rely on long-established green screen and chroma key techniques. Chroma key is a technique to change the green part to a different background when an actor performs in front of a green screen. Recently, many of you can easily change the background through Zoom’s AI background. However, the chroma key technique is not a virtual production because it does not provide believable lighting and a consistent viewpoint.
Q6. If there is a chroma key technique, why is virtual production necessary?
For immersive storytelling, you need a VP with realistic backgrounds, believable lighting, and accurate viewpoints.
If the lighting and perspective don’t match the background, it is very difficult to tell an immersive story. Like the YouTube video above.
- Awkward lighting: The background is a soft light from above, but the subject is illuminated with a strong white light in front of his face.
- Awkward point of view: The viewpoint of the YouTuber is at the height of the face, but the background’s viewpoint is lower than that.
Some content (such as informational content) can be filmed using the chroma key technique. But for immersive storytelling, content creators need a VP. And of course, it must be affordable and accessible.
Q7. Is there an affordable and accessible virtual production?
Beeble develops mobile virtual productions that anyone can use easily.
Beeble develops mobile virtual production so that anyone with a smartphone can do realistic compositing. The goal of our product is:
- Realistic real-time compositing: Provides realistic backgrounds and believable lighting with accurate camera perspective. You can shoot as if you were really there, transcending time and space.
- No physical constraints: Doesn’t require any physical devices: LED walls, green screens, camera trackers, GPU servers. It can be used anywhere such as in a classroom or a living room.
- Easy to use: Anyone can use it easily without any CG expertise. With just a few clicks, you can select the desired background and press the shutter to shoot.
Q8. Is mobile virtual production possible?
Beeble utilizes AI to implement VPs that do not require any physical equipment.
What makes all of this possible is AI. Beeble utilizes AI to implement VPs that do not require any physical equipment such as LED walls or camera trackers. We’ll cover more detail in later posts, but the key components are:
- Matting AI: Given an image, remove the existing background to synthesize the desired background
- Relighting AI: Adjust the lighting on the subject to match the virtual space
- Lightfield AI: Efficiently capture and render a specific space
- Built-in Tracker: Reproduce real-world camera movement in the virtual space
Q9. What are your plans?
Beeble is developing a selfie app that allows you to freely change the background, lighting and viewpoint.
Beeble is developing a selfie app that allows you to freely change the background, lighting, and viewpoint as an MVP (Minimum Viable Product). Are you curious about how we started developing a VP-based selfie app as our MVP? We’ll talk more about that in the next post.
Q10. What kind of team is Beeble?
- Beeble is a startup co-founded by AI experts. We all have more than 5 years of AI research experience, and have published 14 papers at top AI conferences such as NeurIPS, ICLR, ICML, CVPR, and AAAI.
- Beeble is an experienced AI product maker. We have launched several AI products that people actually use, from cancer diagnosis AI to speech synthesis AI. This was done while we worked as AI researchers at Krafton and Lunit.
- Beeble has strong teamwork. We’ve worked together over the past few years and have built strong teamwork.
If you are interested in mobile virtual production, please stay tuned! We will share the details of our technology and product development on this blog, one by one. For more inquiries, please contact info@beeble.ai. 🙂
