The story behind the production of “Magical Forest”
Using generative AI for WebAR development
Content
- Introduction
- → Selected into Niantic Innovation Labs Program
- → About Magical Forest
- → Using Generative AI
- Brainstormed with ChatGPT
- → Think about Experience Content
- → Think about the Background of the Story
- → Summary
- Materials Generated by Midjourney
- → 3D Textures
- → Spell Cards
- → Key Visual Design
- → Title Logo Design
- Audio Generated with Eleven Labs
- Conclusion
- Editorial Note
Introduction
Selected into Niantic Innovation Labs Program✨
Designium was selected for the Niantic Innovation Labs Program, with Niantic offical as a mentor. The program is to use 8th Wall to develop and produce WebAR content compatible with mobile phones and VR / MR headsets in about 10 weeks.
The result of this program was “Magical Forest”, where you can experience a mixed reality world full of fantasy. This MR creation is compatible with mobile devices and Meta Quest 3, so you can easily experience AR on your web browser.
About Magical Forest
Players will play with a dwarf character “Kobbito” who lives in the magic book to complete the task of searching for insects that escaped from the magic book in the “Magical Forest” that appears in the real world. In the magical forest, mysterious plants will grow wherever Kobbito walks, and spell cards will appear allowing players to use magical powers. Players enter the fantasy world through AR in their daily life environment.
Using Generative AI
At Designium, members are actively trying out various AI tools. In this “Magical Forest” project, we used AI tools to conduct project production on an experimental basis, from planning to design to materials.
In this article, designer Kosu summarizes the behind-the-scenes story of using generative AI for production. First, we brainstormed through chatting with “ChatGPT” and finalized the plan. We then used a popular text-to-image generative AI tool called “Midjourney” to assist in creating the material from a designer’s perspective. Finally, we created the narration using the AI voice generator called “Eleven Labs”. The following content will introduce these processes through examples!
Brainstormed with ChatGPT
Because the Desginium team creates prototypes very quickly, in the initial stages we started by testing AR/MR technology for catching insects and walking in the forest. After the core (part of the experience) of “Magical Forest” was realized through the prototype, we used ChatGPT (GPT-4) to generate ideas to flesh out the story settings and experience content.
Think about Experience Content
First, we provide experience content to ChatGPT. Then, to expand the content of the project, ChatGPT came up with interesting elements that could be added besides catching insects.
[Prompt]
Please come up with ideas for planning AR game apps for smartphones and
headset devices.
AR technical specifications
・The camera can detect the flat surface of the floor, but cannot detect
walls or obstacles.
・The camera can recognize some gestures.
・Characters can be controlled via interface or controller.
Experience Flow
・A dwarf named "Kobbito" will appear.
・When the player taps on the book, Kobbito and the magical forest will
appear in AR
・Players walk with Kobbito in the magical forest and search for insects.
・Caught insects will be returned to the book.
・Does the game end when all insects are caught?
・There are single player mode and multiplayer mode.ChatGPT proposes some ideas as follows.
[Response]
Story Development:
Kobbito is the guardian in the picture book, and while he is temporarily
unconscious, the insects escape. These insects are key to driving the story
of the picture book, and without them, the world of the picture book would
stop.
Insect Character Setting:
Each insect has its own name, personality, and role in the book. For example,
the firefly "Lumino" has the role of lighting up the night page.
Tricks and Challenges:
Some insects can only be found through specific gestures or in specific
locations. For example: When the user makes a gesture pointing to the sky,
"Flutter" will appear.
Elements of Competition and Cooperation:
[Single player] A mode where you collect all the bugs by yourself.
[Multiplayer] Teams or individuals compete to see who can collect all the bugs
first.
Magical Moment:
When you catch a specific insect, a short story related to that insect from
the picture book will be released.
Progress Rewards:
Collect a certain number of bugs to receive Kobbito's special gifts and new
magic gestures.
Expansion of the Magical Forest:
Every time you play the game, the magical forest grows, with new areas and
new insects appearing.
Community function:
Users can chat with each other and trade the insects they catch.
Magic casting settings:
Certain insects can only be caught by performing certain actions or steps.
For example, some insects appear by singing magical songs or assuming certain
postures.
Customize your picture book:
You can design your own picture book page based on the insects you collect
and share it with other users.Next, ask ChatGPT to provide specific examples of responses that interest us.
[Prompt]
Could you give me concrete examples about the "Magic casting settings"?I will review the responses to the conversation, add more prompts if necessary, and keep asking questions.
[Prompt]
Can you give me 10 ideas for AR apps that take advantage of the
"Magic casting settings"?After asking this question several times, ChatGPT came up with the following thoughts. Of course, not everything can be implemented, so we choose what seems feasible to guide the conversation.
[Response]
Magic Ring:
Follow the magic ring that appears on the screen with your finger, tracking
it accurately to summon specific insects.
Make AR In-game Items:
Catch specific insects by combining 3D objects to create new items.
Wave a Magic Wand:
Wave your smartphone like a wand to capture insects that respond to
specific movements.
Time Magic:
Capture insects that only appear at night or insects that only appear
during the day based on the actual time.
Magic Spell:
Say specific words or phrases into your phone's microphone to summon insects.
Twin Magic:
In multiplayer mode, two players can catch special insects by making the
same gesture at the same time.
Magic Potion:
Find specific ingredients and combine them to create a magic potion.
When you use magic potions, specific insects will appear.The idea of “Combination of AR Items”, using items to catch insects, seems to be able to be integrated into the experience. And it also fits the setting of the magical world. So I narrowed down the idea and came up with more specific examples as shown below.
[Response]
Light Wand:
Item: magic stone + wooden stick
Result: Create a glowing wand that illuminates insects hiding in the dark and
makes them appear.
Wind Whistle:
Item: wooden pipe + bird feather
Result: Play the flute summons wind, which will cause flying insects to fall.
Magnetic pendant:
Item: metal chain+magnet
Result: Create an insect-attracting pendant that makes it easier to catch
nearby insects.
Water Orb:
Item: water beads + transparent bag
Result: When the Orb is dropped to the ground, a small pond will temporarily
appear, attracting insects in the water.
Fire Trap :
Item : fire seed + dried grass
Result : Light a fire and chase away insects that are afraid of fire.
Dirt Boots :
Item : old shoes + clod of soil
Result : Boots that can detect insects hiding underground.
Talisman of Lightning :
Item : copper coin + charged crystal
Result : Shake the talisman to create lightning and attract insects that
are attracted to lightning.
Moon Mirror :
Item : silver plate + night dew
Result : Use the mirror to find insects that only appear at night.
Sun Clock :
Item : gold gear + sun fragment
Result : Obtain information on insects that are only active during the day
and the time of their appearance.
Ice Trap :
Item : ice crystal + snow powder
Result : Create a temporary area of cold air to gather cold-sensitive
insects in one place.Once we come up with specific items, we consult with engineers to see which ones are technically possible to implement. Taking working hours into consideration, we decided on the following specifications based on ChatGPT’s proposal.
[Specification]
Players can use magic items through "spell cards" during the game experience.
Magical powers can help catch insects.
Possible items that can be implemented：
・Magic Magnifying Glass: Allows you to see transparent insects.
・Time-stopping Hourglass: Temporarily stops time, making it easier to catch
moving insects.
・Magic Mushroom: Plant it to attract nearby insects to come closer.
・Dirt Boot: Detect insects hiding underground.
・Sound clue: Find out the location of insects by selecting the sounds they
make from their calls or movements.
・Magic Flute: Insects hiding nearby will come out.Think about the Background of the Story
Next, consider the background story with ChatGPT.
[Prompt]
Could you please give me 10 ideas for the story of "Magical Forest" based on
the conversations we've had so far?[Response]
The 10 story ideas proposed are as follows.
Dream of Picture Books: The picture book was drawn by the player himself when he
was a child. Now players are embarking on a journey as an adult in search of
a forgotten piece of imagination.
Time Fragment: The dwarf is a messenger from the future. The future without
insects in picture books is chaotic. Players from the past hold the key to
saving the future.
Memories of Picture Books: Insects are the memory lost from picture books.
As players collect insects, the story of the picture book is gradually
pieced together.
Guardians of the Magical Forest: Insects are the spirits that protect the
forest. Without them, the vitality of the Magical Forest would diminish.
By bringing back insects, light returns to the forest.
Poet of the Forest: The dwarf is the poet of the forest. Insects are a
source of inspiration for his poetry. If we don't bring back the insects,
the poetry and songs of the forest will be lost.
Picture Book Restorers: Picture books were created by the great magicians of
the past. However, the magician's apprentice, a dwarf, releases the insects
by mistake. The player takes on the role of a restorer, searching for insects
to restore the world of picture books to normal.
Treasures of Insects: Each insect has its own treasure. By finding it and
returning it to the insects, it will return to the picture book.
Change of Seasons: Time passed faster than usual in the forest. You have to
find the insects within a limited time. In the forest, the seasons change in
a very short time. Some insects only appear during certain seasons, so look
for them as the seasons change.
...(omitted)Ask ChatGPT to provide more specific examples in interesting directions. So I sent the proposed content back to ChatGPT and asked them how they could make the story more engaging. I received the following advice.
[Response]
The relationship between Kobbito and picture books can be explored in a
little more depth. The characters the Kobbito play in picture books and
the problems and emotions they face in the real world are also factors
that can make the story more engaging.When I asked for specific examples, I received the following response.
[Response]
Guardians of Picture Books: Kobbito and his friends take on the role of
protecting picture books and strive to maintain peace and balance within
picture books.
Protagonist of the Story: Kobbito is the protagonist of the story in the
picture book and has various adventures and experiences throughout the story.
Creator of Picture Books: Kobbito himself is actually the creator of the
story and characters of the picture book, and spends his time with the
characters born in the story.
Daily Life as a Picture Book Resident: Kobbito lives a normal life in the
picture book, interacting and developing friendships with other characters
and creatures.
Picture Book Wizard: Kobbito has the ability to manipulate magic in picture
books and use it to experience various events and adventures in the story.Furthermore, we received specific ideas by changing the questions, and we also received suggestions for the setting of the magical forest.
[Response]
Magical Protection : The Magical Forest was originally a special place in
picture books, protected by Kobbito. When the insects trapped in the forest
escape, the forest suddenly appears in the real world.
Secret Seed: One day, a child plants a mysterious seed drawn in the corner
of a picture book page in the real world. This was the origin of the Magical
Forest.
Space-time Distortion: Because Kobbito used magic incorrectly, the time and
space in the picture book was distorted, causing the real world and the world
of the picture book to overlap.
Magic Door: In the picture book, there is a door leading to a magical forest.
This door suddenly appears in the real world, causing insects to escape into
the real world.
Continuation of the Dream: When the player dreams, Kobbito is looking for the
insects in the picture book in the player's dream. When the player wakes up,
this dream is connected to the real world.
Kobbito's Experiment: Kobbito tried a new magic, but it failed, causing the
insects in the picture book to escape into the real world.
Magical Melody: One day, a child discovered an ancient musical score drawn in
the corner of a picture book, and when he played it, a magical forest appeared.
Influence of the Moon: On a full moon night, when the magic in the picture book
becomes most powerful, its influence spreads the magical forest into the real
world, causing the insects to escape.
Memories of picture books: Picture books were once loved by people, but now
they have been forgotten. Characters from picture books appear in the real
world in order to be loved by people once again.
Kobbito's wish: Kobbito used magic to seek knowledge and experience about the
world outside the picture book. As a result, the world in the picture book
overlapped with the real world.To put the question another way, what is interesting about the world of the Magical Forest? When I asked this question I received the following response.
[Response]
Magical Forest = Past World
Players travel back to the magical forest of the past world to look for
insects and creatures from the past era.
Magical Forest = Beyond the Mirror
Through an ancient mirror, users enter a paralleled world called the Magical
Forest. The world in the mirror is the opposite.
Magical Forest = Planets in Space
The forest actually exists on a distant planet, and players explore that
planet in a spaceship.
Magical Forest = Inside the Music
A magical forest that appears when a specific melody or song is played.
Users can look for insects according to the rhythm and melody of the music.
Magical Forest = Inside the Painting
The scenery in the painting becomes real and there is a magical forest inside.
Hidden within are the mechanisms and mysteries related to color and
brushstrokes.
Magical Forest = Inside People's Memories
Visit a magical forest that exists in people's memories and explore past
scenarios and scenes.
Magical Forest = Above the Clouds
A magical forest floating on the clouds in the sky. Players fly through the
sky and search for insects.
Magical Forest = Land of Shadow
A magical forest hides in the shadows on the other side of the light. Users
can search for insects in the dark by the light.
Magical Forest = World of Letters
A magical forest formed by the letters and words of books. Users search for
insects while solving riddles related to the meaning of words and the rhythm
of sentences.
Magical Forest = Inside the Snow Crystals
A small magical forest exists inside each snow crystal. Users search for and
collect insects along with the winter scenery.It’s interesting to come up with ideas that I wouldn’t have thought of on my own. Once we confirmed what the story would look like, we could manually modify the setting and story to fit the specification and incorporate previously discussed ideas. Finally, share it with ChatGPT and let ChatGPT tidy it up. This is the backstory we created and polished through ChatGPT.
[Response]
A picture book created by a legendary magician hides a door connecting
the real world and the magical world. In that magical world, there lives
Kobbito, small creatures full of curiosity. One day, driven by the spirit
of adventure, Kobbito opened the door to the picture book world and stepped
into the real world. However, because they forgot to close the door, the
insects from the magical world followed them to the real world. With the
power possessed by these insects, a magical forest begins to spread in the
real world. In order to maintain the balance between the real world and the
magical world, Kobbito and the players embark on a grand adventure to find
insects and bring them back to the picture book.Summary
We developed the app based on the experience and backstory. We made modifications and adjustments based on feedback from Designium and Niantic offical to refine the final experience. By brainstorming with ChatGPT, I was able to come up with ideas that I wouldn’t have thought of on my own, and I was also able to organize my thoughts and summarize my final plan. From the next chapter, we will move on to the production process of the “Magical Forest”.
Materials Generated by Midjourney
The following will introduce how to use images generated by Midjourney in a project from a designer’s perspective.
3D Textures
In the game, you can meet Kobbito and insects with mysterious patterns in the magical forest. The patterns for Kobbito costume and insects are generated using Midjourney as textures for the 3D models.
STEP 1. Image generation
- Insect pattern
At first, I tried to generate a pattern by entering the word “Polka dots” as a prompt, but the pattern was too thin and too regular and beautiful. It didn’t feel right to use it as a pattern for insects.
So I looked for a new direction by changing the words. I used “insect surface” in the prompt.
You can also use the prompt “- -no” to specify content that you do not want to be included in the generated image. Maybe because I used “insect surface pattern” in the prompt, the resulting image sometimes looks like the insect limbs included in the pattern, but I can avoid this by using “- -no insect body”.
In the end, after adding the prompt “strong distortion dots”, it is closer to the image I want. It is interesting to be able to create unexpected patterns by using generative AI.
- Kobbito Costume
I wanted to create a marble texture for the Kobbito costume, but if I just used the word “Marble” the color would be too light and not stand out. I tried different ways to re-express the marble pattern and in the end “like an amoeba” worked out well. Just like this, the generated results will be affected by how the words are worded, so I think it is important to get creative with your wordinig.
STEP 2. Apply to 3D models
We create the visual appearance of various insects by applying the generated images as textures to the insect model. Although I ended up not using the image generated with it this time, you can also create seamless textures by using the prompt “- -tile”.
For Kobbito costume, it cannot use the generated image directly, so the image needs to be edited based on UV mapping. However, this is still more efficient than drawing the design from zero.
Spell Cards
I designed the spell cards using images generated by Midjourney, based on ideas I picked during the ChatGPT phase. The following two materials are required.
- Magic Mushrooms: mushrooms that attract insects.
- Magical Light: The light that allows users to see transparent insects (the light in the game looks like a ball of light floating behind Kobbito).
STEP 1. Image generation
- Magic Mushrooms
Let’s start with mushrooms, this seems easy. Initially, the mushrooms looked poisonous, but after trying various prompts over and over, I was finally able to generate pop mushrooms that looked like game items.
The wait time to generate an image was quite tight, so I used “Set Items” and “Game Assets” which would generate an image with multiple samples. ↑ As you can see in the image above, it is efficient because you can choose as many mushroom designs as you want in one generation.
Once you decide on the image you want to use, increase the resolution based on your expected size. In Midjourney, you can increase the resolution to 4096 x 4096 px by pressing the “Upscale” button.
- Magical Light
For “Magic Light”, it was hard to create the image I wanted. I want it to look like a floating sphere so that it wouldn’t deviate from the image of the game item. But even if I use the word “floating”, the resulting image of the item doesn’t look like it’s floating.
I also decided to avoid using glass or light expressions as they are difficult to edit when cutting out the image and turning it into material. I tried words related to light balls and ended up choosing an item that looked like a light ball from the images generated using the word “crystal.”
STEP 2. Retouch with Photoshop
When I put the two generated images together, I noticed a difference in their appearance. The mushroom is 3D-like and the light is 2D. So I applied brushstrokes and some effects in Photoshop to soften the visual differences. Additionally, I adjusted color and brightness so that it looks more natural even when the two items are placed side by side. Manually editing small adjustments like this in Photoshop is much faster than changing the prompts in Midjourney.
STEP 3. Design the Spell Cards
After adding the background and text, the spell card is complete. In the final version of the design, the color of the illustration was further adjusted to match the background color.
Key Visual Design
The image below was created by using Photoshop to combine 3D elements with images generated in Midjourney.
STEP 1. Image generation
First, I create a rough image in my head. The image I wanted to create was like Kobbito exploring the magical forest. I wanted to use the designs of the Kobbito and plants directly from the game, so I decided to have Midjourney generate the forest background and composite images rendered from the 3D model onto it.
At first, I added flowers to the image, but it seemed difficult to composite, so I decided to generate a simple forest without unnecessary elements. I’ve also added a prompt to generate a 16:9 ratio image, since landscape is easier to use later.
We chose the image below to have enough space to compose the central elements and to have a slightly unrealistic atmosphere.
STEP 2. Composite with Photoshop
Next, we will add elements to the forest background in Photoshop. I will add images rendered from the 3D models of Kobbito and plants used in the game while adjusting the balance.
Because the individual materials don’t blend together and stand out, I use Photoshop to adjust the brightness and color of each material to get balance throughout the image. To integrate the composite elements with the background, I added a stroke filter on top of the layer to blend it all together.
By the way, I later realized that I needed a wider image than ↑ the image above. This time I used a vertically stretched image, but I could also use Midjourney’s Zoom function to further expand the resulting image up, down, left, and right, in which case I can add more forest part to the background. The strength of AI tools is that they can respond to such needs immediately.
Title Logo Design
The title logo for “Magical Forest” was created using vectorized parts of images generated by Midjourney.
STEP 1. Image generation
Midjourney is not good at generating text, so I created the text in Illustrator as shown below.
I wanted to add motifs of books and plants to the text, which are key to the experience, so I decided to use Midjourney to generate these materials.
An illustration-style logo was initially generated, but by adding “flat”, “simple”, “clean” and other prompts, the logo was refined into a flat one. When thinking about vectorization, it seems easier to use monochrome. Since I will only be using part of the image this time, my judgment will be based on ease of use of the subject rather than overall completion.
Now that I have generated some images that can be used as materials, I would like to convert these images into vectors and use them for the title.
STEP 2. Vectorize the generated image
This time I used this website to vectorize the generated images. You can easily convert images to vectors by simply dragging and dropping them to upload them.
After downloading and opening it in Illustrator, you can see that it has been converted into a path as shown below. Then you can edit the path as needed.
Through this process, I vectorize multiple generated images and extract motifs that can be used.
STEP 3. Combine the motif with the text
Add the motif extracted from the generated image to the text path and adjust the overall balance. I also added a leaf motif to the text using leaves extracted from images generated by Midjourney.
STEP 4. Finished with Photoshop
Texturing and coloring were done in Photoshop. The completed logo is shown below.
Audio Generated with Eleven Labs
Because there are scenes at the beginning and ending of the experience where Kobbito talks to the player, we thought it would be a good idea to add audio to increase the immersion of the experience. Therefore, we used “Eleven Labs” to generate the voice, an AI voice generator popular for its ability to create natural AI voices. You can try it for free, but if you want to use it for commercial purposes, you’ll need to subscribe to the pricing plan.
The “Magical Forest” has a need for English narration, so we chose an English demo voice based on accent and tone. It is generated based on the selected demo voice, which you can also adjust the voice model in the “Voice Settings”. In addition to the default voice models, you can also use voice models generated by other users from the “Voice Library” in the menu.
Even if the settings are the same, the tone will change each time a new voice is generated. So we recommend producing multiple voices of the same sentence for comparison. It’s easier to adapt and use if they are generated sentence by sentence rather than all at once.
In addition, sometimes the sentence breaks and intonation of AI reading are unnatural, but if punctuation or spaces are added between words and regenerated, it may become more natural. When you feel uncomfortable with the way it starts speaking, using a space or line break at the first character may make it sound smoother.
You can check past generated samples and their settings from the “History” list, which is useful if you later want to check the settings of a previously created sound.
Please turn on the audio when experiencing the “Magical Forest” to enjoy the final result.
Conclusion
Advantages of using generative AI
Through this project, I was able to use AI tools to independently produce materials that would normally be outside my scope or require outsourcing. For example, I am not good at creating illustrations, but by using Midjourney to generate images, I can be in charge of everything from illustration production to final design by myself, as in this example of the spell card illustration. Another point is that you can give direct instructions (prompts) so that you can try out various ideas without hesitation.
Assignments
AI-generated content may lack consistency in style and subject matter. For example you cannot generate the same content again even with the same prompt. Therefore, it is difficult to fine-tune works generated by AI. And it will take a long time to continuously pay attention to the generation of small details. I think relying on design tools such as Photoshop to make such corrections can make more effective use of AI.
Also, it is difficult for AI tools to fully understand the overall context of a project, including its background, objectives, cultural elements, and emotional nuances. So it’s essential to tailor AI recommendations to your specific situation and guide you in the right direction. When using generative AI as design materials, it is necessary to carefully consider whether they are in line with the purpose and message of the project.
Demos
Finally, I would like to introduce to you some scenes from the final version of “Magical Forest”. The illustration for the Magic Mushroom card, the Kobbito and insect textures were created using Midjourney. If you have a Meta Quest, we recommend using headsets to experience it!
EDITORIAL NOTE
I am Mary Chin (Chi-Ping Chin), the writer and designer of the PR team at Designium. Learning to use AI and how to work effectively with AI is very important. This is also an issue that I will continue to study in the future. Let’s work together to find the answer! If you have any ideas for cooperation, please feel free to contact us. 😉
