How to create a game using the power of 30 neural networks
We have been following the AI development for the past few years and for 4 years have been using the concept of AI co-creation in our company. Previously we have written two big overview articles (1, 2) concerning the use of AI in graphics and creative IA concept. Since then we have collected all possible technological solutions, products, tools and neural networks in graphics and made Generative AI Library with more than 1300 segmented tools.
As the result based on our experience we have decided to make Phygital+ product, that allows creators artists and game designers to use the potential of the most developed and up-to-date tools for content generation. Right now the product has more than 30+ neural networks that are aimed at solving particular task, for example, panorama or texture generation. Among other tools there’s also powerful AI instrument that allows to train AI on your own style, character or location. The number of neural networks is constantly growing, and it takes us 1 day to add new AI tool. So, basically if a new tool comes out, we can quickly add it to our node-based product. With our product creators don’t have to know how to use Google Colab and they don’t need powerful GPU to experiment with the latest tools, we have it all in the cloud computing, accessible to everyone via browser.
When we started to use this approach, there was no such naming as Generative AI and the quality of generations was quite low. We used it only for texture generations and in our experiments in the art project. Now with the appearance of diffusion models the quality of generations increased significantly, making it possible to use it in the production pipeline.
We want to touch upon the potential of AI for game companies throughout the whole pipeline, and from 1300+ tools out there, at least 30+ neural networks can really be useful. To be more precise, we will talk about how to use AI in generating 2D and 3D content and how AI is used at the concept, pre-production, production and post-production stage of pipeline. In this article we focus on Concept and pre-production stage.
- Inspiration and moodboard
Usually when we start working on some project, the first step is to find relevant references. With AI you can turn one image into several references. In order to do that it’s better to use the neural network called CLIP Interrogator, which transfers any image into the text prompt. The prompt is a short text for neural network that is specifically structured to be understood by AI text2image tools. It means that it has specific keywords that represent camera angle, image style, pose of the character, etc. Text prompt is needed for generating concepts based on your reference.
At this stage you can also use ChatGPT or Lexica prompt search to modify it.
If you need to get a quick result, we recommend to try Artistic Mode feature. Whenever it’s on, it modifies your prompt to make the image more artistic. We call it PromptOps —optimization of your work with prompts, because we understand that prompt engineering is a skill to master. We also published Prompt Guide that helps to generate beautiful artworks easier and faster.
- Concepting
Moving forward, for quick concept creation we have a good number of pretrained custom checkpoints — Stable Diffusion models trained on the specific style. While using these custom models you’re almost guaranteed to have a great result. You can connect the result from Describe image (Image-to-text prompt) with Stable Diffusion nodes and choose any model you like.
Moreover, you have an opportunity to train neural network on your own style or object and generate what you want in the desired style.
Here the example of how we took 30 screenshots from the game Project Winter and under 30 minutes trained a brand new model, which now can generate images in that style. If you are training AI model in our web interface, it’s only available to you, so you don’t have to be worried about privacy.
Each model is about 3GB in size, and to many our clients they need dozens of model. Thanks to the cloud computing, there’s no problem in creating and training as many models as needed. You can create a model for each character, for the overall style of the game or only for the particular objects, for example, plants.
One of the most common task of game studios is generating batches of images with similar, but a bit different features. With us you can create a tool for generating assets, characters and locations in your style as a little factory for content generation.
If you like any of the character concepts that has been generated for you, you can edit and enhance it with ControlNet, InstructPix2Pix and other neural networks. For example, we liked the girl on the fourth image. We can change her hairstyle, make her robot or add glasses with simple text.
You can also use inpainting to work with the details on the image: you can create the mask right in the web editor, add it to the workspace, connect to the SD node and neural network will fill the blank white part with content. Don’t forget to use negative prompt — a small text that tells neural network what it should NOT generate.
On the left panel you can see all available neural networks for changing the image.
After we have made a portrait concept which we like, we can try to create the environment for this character or how they would look like in different clothes. In order to do that, we need to use outpainting: we need to add the character concept on the white background and ask AI to generate the srurrounding. You can repeat it several times to achieve the needed result.
If you want to put this character in different location, we need to look for a specific prompt, generate the location and change the background using Remove background feature.
- Storytelling (storyboards, animatics, comics)
On the concept production stage we have decided that this character fits our expectations, and now we want to generate draft frames and other poses of this character. In order to do that we need the neural network to remember this character. The DreamBooth node comes here into play. But we need more images of the same person to make the training effecient (it’s recommended to have at least 5 images of your character or person to get good results in DreamBooth). Portrait Animation can provide you with different face angles of person, so by simply using this node you can get variations of the same person’s face. Then we have to name the character (in this example we named the character Pinkymiel) and start training.
After the training you can create any storytelling with your character.
For your convenience we have collected all the templates in one place, so you can click on any of them and recreate a particular pipeline by yourself.
We are currently in the Open Alpha and would be glad to hear any feedback concerning the pipelines, usage of AI in your daily work and creative tasks. Soon we will publish more articles concerning tasks in game development on different stages of pipeline, stay tuned for more!
Follow us on socials: Twitter, Instagram, Discord, Telegram.
