Generate AI Rendering with Blender ComfyUI AddOn
In this written guide you will see how to generate ai rendering with Blender ComfyUI AddOn that allow you to make 3D AI render with the possibility to use viewport and also control your model.
What is ComfyUI?
ComfyUI is the most powerful and modular stable diffusion GUI as backend and the user interface will let you design and execute advanced stable diffusion pipelines using a graph/nodes/flowchart based interface.
A resume for ComfyUI:
Nodes/graph/flowchart interface to experiment and create complex Stable Diffusion workflows without needing to code anything.
Model supported by ComfyUI:
Fully supports SD1.x, SD2.x, SDXL, Stable Video Diffusion and Stable Cascade
What are the features that comes with ComfyUI Blender AddOn?
ComfyUI support even more feature, but the Blender AddOn support these three main features of ComfyUI that we will use as Blender AddOn are:
- text to image : basically with a prompt you can generate
- image to image: import an image inside Blender and generate a similar picture as final result
- control your character: ControlNET allow you to rig your main character by using openpose model that set every single movement that you can draw as empty with grease pencil
How to install ComfyUI
Install ComfyUI
Basically we will start by installing ComfyUI from the official website, you need to scroll until you find a link called “Installing ComfyUI”, I add the link for you so you don’t need to find it:
Then you need to click “Direct link to download” and it will start automatically to download, it’s around 1,5 GB with ComfyUI file
Here you can find the link to download on Windows (CPU and GPU are available for your AI rendering) :
Extract ComfyUI
After finish download you need to extract the file with 7zip or WinRAR to a specific path that you prefer with the folder “ComfyUI_windows_portable” and it will be display like this:
As you can see there are two options:
- “run_cpu” works with different cpu
- “run_nvidia_gpu” works only with Nvidia Gpus
But before to launch ComfyUI we need to install models in order for you to generate AI rendering images
Install ComfyUI models
I select for you three models to install, you must need to follow the same folder location after you install each AI rendering model.
1) Disney Pixar [Type A]
The first model that we will use is Disney Pixar AI model in particular the Type A that allow you to generate characters similar to Disney’s style.
Here you can find an example by the community:
To install this model you need to click download, it’s around 4,0 GB (3,95GB), but it’s worth it for the goal that we have in mind.
After you install you need to locate in the right path:
ComfyUI_windows_portable\ComfyUI\models\checkpoints
After you locate correctly we need to install another model that can undestand where your rig drawing will be pose as main character
2) OpenPose by ControlNET v1.1
The next model is ControlNET v1.1 that allow you to rig your character by drawing lines structure for the entire body.
You need to scroll until you will find openpose model
I added a link for you to simplify the process and it will automatically download by direct download:
After you download it you need to locate in the right path:
ComfyUI_windows_portable\ComfyUI\models\controlnet
Before to install the final Blender AddOn, we need to install a translator for ComfyUI to comunicate to ComfyUI Blender AddOn called Blender CUP.
3) Blender AI Bridge (Blender CUP)
Blender UI CUP works as connection between ComfyUI and ComfyUI Blender AddOn, without this module you can’t use it by starting the server
To download this model, you need to click on Code button and then you must use Download ZIP button
Once you finish download, make sure you have the same folder
ComfyUI-CUP-main
Then you need to copy this folder [ComfyUI-CUP-main] into ComfyUI_windows_portable\ComfyUI\custom_nodes
Install Blender ComfyUI AddOn
Before to launch ComfyUI we need to install Blender ComfyUI AddOn and also configure in the right way to make the connection possible bewteen the server that connect in only one.
Here you can see how the webpage display
Now go into Releases and select the last version:
In this case BlenderAI node 1.4.9(Compatible with Blender 4.1) and select BlenderAI_Node.zip file:
Now open Blender and install the AddOn as zip file, it will display like this:
You need to open the AddOn the arrow on the left side of the checkbox
Preference > General
Parameters to change:
IP : 127.0.0.1 (keep the same) change from 8189 to 8188 (right value)
Make sure the ComfyUI path corrispond to the actual path
After all these steps you can run server in both ComfyUI (on web and on Blender)
Run ComfyUI and ComfyUI Blender AddOn
Run ComfyUI on browser
So finally you can test both ComfyUI after setting all these parameters, first run ComfyUI, in my case I use CPU, but if you have Nvidia Graphic Card you can run GPU, then cmd will display like this
After that you will see on your browser the GUI of ComfyUI, here you can select on Load Checkpoint the Disney model
Run ComfyUI on Blender
Now you’re ready to launch ComfyUI also on Blender, so instead of using timeline select ComfyUI node in the menu
Now press New to add a new node tree for generate AI Rendering
Now on the right side go into the last panel and you can see Execute Node tree that we will user after we make a node
Node Tree
Now we will generate different node tree, for more precise information check my video on Youtube about it.
Text to Image
Viewport Image and ControlNET
Here is the final viewport image and control net node tree
