Trying Out Midjourney’s New Inpainting Function
This might sound weird: Ever since I’d heard inpainting was on the horizon, I decided to spend time exploring AI art in genres other than my main focus (e.g. here). But my logic was this: What’s the point in doing more in your main genre now, when in a few weeks you’ll be able to it so much better thanks to a game-changing tool? I figured I’d use the intervening time to round myself out as an AI artist, going for breadth rather than depth.
I also figured it was long overdue to brush up on my art history, so I read Gombrich’s The Story of Art, and finished it just in time for inpainting to roll out
Then it was back to a creation that I’d given up on pre-inpainting. I wanted 8 specific things:
- A Zen boulder garden, with the ground undulating in a spiral pattern, slightly departing from the traditional parallel lines and concentric circles
- Tree ferns incorporated as part of the garden
- Moss, like Kokedera
- For the Zen garden to be inside a cavern
- Reflective features such as mirrors on the cavern walls, and/or water on the cavern floor
- A cavern ceiling sculpted like mammatus clouds
- Foggy twilight outside the cavern entrance
- A view of peak shaped like the Matterhorn, towering out of a void, framed by the cavern entrance
- For the mountain to be vegetated without snow
In particular, Midjourney often fails when you try to frame one part of an image inside another. /blend only works some of the time. So before inpainting I’d struggled to get half of these in one image. But now my final result was:
I’d been forced to start with micro features and then work outwards to the macro level (as in The Roll Monitor and Wuhan Frankenstein). But now I figured I could get away with the reverse order, generating the main scene first and then inpainting in the details. So I prompted cavern interior, fog, A7R II — no stalactites, stalagmites — s 50 — style raw and chose this image as my starting point:
With high hopes, I clicked Vary (Region), selected the area inside the arch, changed the prompt to be a mountain with vegetation and no snow, and got …
Images with vegetation around the entrance but not on the mountain, which usually had snow even though I’d specifically told Midjourney not to put it in! No matter what I prompted, the result came out the same. Even when I gave up on the vegetation and just asked for a mountain without snow, it was still putting in the snow.
The bot had dropped many IQ points in inpainting mode. This was shaping up to be a disappointment.
As a last-ditch effort, I tried including an image of the type of mountain I wanted. It was the same one I used in my article comparing Midjourney to Jasper AI, generated with the prompt: expansive aerial photograph, focusing on an awe-inspiring glacial horn like the Matterhorn, whose steep ridges function as leading lines, cliffs with water cascading down into a cloud inversion that creates a dreamy feel, completely covered with lush subtropical jungle, foliage silhouetted against the fog, shot with Sony A7R II, 14mm lens, high HDR to enhance colour contrast and create a sense of intrigue, twilight, hot and humid air — style raw — no snow
I tried inpainting again, selecting the area under the cavern arch, and prompting: <URL of Image of Mountain> cloudforest in the distance, covering an awe-inspiring glacial horn like the Matterhorn, shrouded in fog, expansive aerial photograph, A7R II, twilight — style raw — no snow
This was more like it! Zooming in, I loved how even this scaled down version still kept the silhouettes of the vegetation against the fog.
Next was the mammatus cloud ceiling. My first attempt at inpainting had only succeeded with an image prompt, so I decided to try it again. First I needed to generate the picture to use as an image prompt. I used the text prompt: black sloping rock ceiling:: black mammatus_clouds — s 50 — style raw
If you don’t understand what I’m doing with the above prompt, there are two things to note:
- I used the double colon ::, so that, behind the scenes, Midjourney would generate separate images of a “black sloping rock ceiling” and of “black mammatus clouds” and then blend them together. I was hoping that the blackness and overhanging nature of both the ceiling and the clouds would be function as “linking elements” (discussed in Supercell Flying Saucer, The Roll Monitor and Rice Terrace Ziggurat), getting the bot to “glitch” and combine them into a single hybrid element, rather than create an image of a home with a rock ceiling and a view of mammatus clouds out the window.
- In Midjourney, you often get better results if you put an underscore between words in a phrase rather than a space, as long as the phrase is common enough. “Mammatus clouds” was common enough for this to apply. EDITED TO ADD: I go into more detail about that in this post.
One of the results was this image of a stylish interior:
I cropped it down to just be the ceiling, without the lights and walls.
For the inpainting, I selected the ceiling in my latest version of the cavern pic. I got best results if I really amped up the prompt weight for the mammatus clouds: <URL of above image>:: cavern interior, fog, A7R II:: black mammatus_clouds::3 — no stalactites, stalagmites — s 50 — style raw
One thing bothered me: About halfway up the right side of the entrance, a bit of rock stood out in an ugly way.
I tried using inpainting to remove it (using the prompt: cavern interior, fog A7R II — no stalactites, stalagmites — s 50, — style raw) but got the error message below:
This is a really annoying restriction. I didn’t want Midjourney to draw anything except fog, so it would hardly be a hard task for the bot even with such a miniscule area selected. I worked around it by selecting more sky around the mountain and running the same inpainting prompt. Midjourney gave me 4 images with the annoying knobby bit gone. Problem solved.
Now for the Zen garden. I selected a large swathe of the cave interior for inpainting, and tried about a dozen prompts, both with and without image prompts. This time, the best results came from a prompt with just the text: Zen garden tree_fern moss boulders Kodedera raked_gravel swirls whorls patterns of flowing lines — s 50 — style raw
There was some ugliness on the back right boulder, but I figured I could use inpainting to get rid of it. More seriously, none of the dozen prompts had yielded a single tree fern in any of the results. I wanted to fix this before going any further.
Inpainting prompts failed about 10 times to fix this. What worked was firstly to generate an image prompt using the following text: tree_ferns, cavern interior, moss, boulders, fog, A7RII — s 50 — style raw
Out of the results I selected the below image for further use:
Then finally to select a large portion of the walls of my Zen cavern (I also took advantage of this to get rid of the AI glitch on the right entrance boulder) for inpainting, and prompt: <URL of Above Image of Fern Cavern> tree_ferns — s 50 — style raw
I kept this image:
Now it was time for the mirrors. I decided to inpaint with an image prompt again. First I generated the image with: hall lined with mirrors, fog:: walls made of reflective black rocks:: humans, arches, windows, lights, candles, chandeliers, windows::-1 — s 50 — style raw
The result didn’t really match the prompt, but it was ideal for my purposes:
I wanted to remove elements that would confuse Midjourney, so I cropped it down to:
Then I selected portions of the walls in inpainting (this time, I couldn’t select too much or the AI would glitch), and prompted: <URL of Above Image of Mirrors>:: mirrors — s 50 — style raw
The best result was:
Midjourney gave me something more like a polished surface of obsidian, but I love how it blends in with the rock walls as a “blink and you’ll miss it” feature.
I was wondering whether to stop here, but I wanted to try putting water in the troughs of the spiral. I went into the inpainting tool and selected the troughs of the spiral, avoiding selecting the ridges except in the parts where the troughs were narrow in the picture. I tried a few prompts related to water, but got the best results with just one word: puddles
A few parts of the spiral now looked ugly to me, so I decided to inpaint them out like I’d done with the knobby rock at the entrance. I selected them, used the inpainting prompt: moss spiral in a Zen garden like Kokedera with swirls and whorls, puddles, A7R II and got my final image:
I also wanted a portrait version, but cropping the above image took out too much of the ferns for my liking. I needed to Zoom out. I did this with Custom Zoom, using the prompt: tree_ferns, puddles, moss spiral in a Zen garden like Kokedera with swirls and whorls, A7R II — ar 1:1 — zoom 2
Then I cropped the result to a 9:16 portrait ratio:
Four things I learned about Midjourney’s Vary (Region) function:
- You can get great results by first generating a “macro-image” of the scene you want, then inpainting in specific features (perhaps framed by elements in the “macro-image”).
- If you want to use it to introduce a feature, the bot usually does much better if you include an image in the inpainting prompt (perhaps generated separately in Midjourney).
- You can now use it to remove AI glitches and other small annoying things in the image. You could almost certainly use it to get rid of the infamous six-finger hands and the like.
- Sometimes the bot refuses your request for the above unless you select enough space. If you just want to vary one tiny thing, select negative space like sky in order to satisfy it.
This has inspired me to make heavy use of inpainting going forward, and I hope it inspires you as well.
