Stable Diffusion prompt engineering toolkit

7 min readNov 15, 2022

If you’re running Stable Diffusion on your local computer, you’re likely using Automatic1111’s excellent webui.

Even though the core application is packed with tons of features, there is also a growing library of user-contributed extensions. The most popular of these can be installed directly from the Extensions tab:

In this blog I will describe Dynamic Prompts, an extension that assists you to write more detailed and expression prompts. The extension provides custom syntax and a number of tools to add spice to your prompt.

Here is an overview of what’s available on version 0.24.0, the most recent at time of writing.

Dynamic Syntax

https://github.com/adieyal/sd-dynamic-prompts#combinations

Dynamic selection

A {house|apartment|lodge|cottage} in the {summer|winter|autumn|spring}

This syntax will randomly choose a value from each of the bracketed sections, here are some examples of the prompts it might generate:

A house in the winter
A lodge in the spring
A apartment in the autmun

Permutations

A {house|apartment|lodge|cottage} by {2$$artist1|artist2|artist3}

Using the $$ syntax, you can choose combination of elements, e.g.:

A house by artist2,artist1
A cottage by artist3,artist2

Custom delimiter

A {house|apartment|lodge|cottage} by {2$$ and $$artist1|artist2|artist3}

By default, combinations are delimited by a comma, you can customise it with the $$ delimiter $$ syntax, e.g. the example above would generate:

A house by artist2 and artist1
A cottage by artist3 and artist2

Permutation Ranges

You can choose a range of elements in your permutations, e.g.

A house by {1-3$$artist1|artist2|artist3}

This will create permutations of one, two, or three elements. Here is the complete list of posibilities:

A house by artist1
A house by artist2
A house by artist3
A house by artist1,artist2
A house by artist1,artist3
A house by artist2,artist1
A house by artist2,artist3
A house by artist3,artist1
A house by artist3,artist2
A house by artist1,artist2,artist3
A house by artist1,artist1,artist2
A house by artist2,artist1,artist3
A house by artist2,artist3,artist1
A house by artist3,artist1,artist2
A house by artist3,artist2,artist1

Wildcards

https://github.com/adieyal/sd-dynamic-prompts#wildcard-files

Sometimes there are too many items to add in the prompt box. In this case, you can create a wildcard file and place it in the wildcards directory. For instance, a file called artists.txt with one artist per line, can be used as follows:

A house by __artists__ and __artists__

This will pick two random artists from artists.txt

Nesting

https://github.com/adieyal/sd-dynamic-prompts#nesting

Wildcards can be nested inside of wildcard files, e.g.

-- artists.txt --

__expressionism__
__impressionism__

-- expressionism.txt --
Frank Lobdell
Frank Okada
Gene Beery
Grace Hartig

-- impressionism.txt --
Guillaume Vogels
Henry Luyten
Jef Dutilleux
Jenny Montigny
Juliette Wytsman

Then

A house by __artists__ and __artists__

Will choose artists from both expressionism.txt and impressionism.txt.

Wildcards can be nested arbitrarily in hierarchies to allow you to manage large wildcard libraries.

You can also nest inside dynamic selections, e.g.

A house by {__expressionism__|__impressionism__}

Note that this is subtly different to A house by __artists__ and __artists__. Using dynamic selections means that expressionism.txt or impressionism.txt and randomly chosen, then an artist is chosen randomly from the selected file. In the other case, expressionism.txt and impressionism.txt are merged, and an artist is randomly chosen from the combined list.

Globbing

https://github.com/adieyal/sd-dynamic-prompts#fuzzy-globrecursive-wildcard-filedirectory-matching

In more advanced use-cases, you can use globbing to select which wildcard files you want to use. For example, if you have the following wildcard directory structure:


wildcards/
├─ comics/
│  ├─ b&w/
│  │  ├─ heros.text
│  │  ├─ villans.txt
│  ├─ colour/
│  │  ├─ heros.txt
│  │  ├─ villans.txt
├─ movies/
│  ├─ b&w/
│  │  ├─ heros.txt
│  │  ├─ villans.txt
│  ├─ colour/
│  │  ├─ heros.txt
│  │  ├─ villans.txt

Choose a villan from all four villans.txt files

A painting of __villans__

Choose a b&w villan

A painting of __b&w/**/villans__

Choose a villan from a b&w and colour movie

A painting of __movies/**/*__

Combinatorial Generation

https://github.com/adieyal/sd-dynamic-prompts#combinatorial-generation

There are times when you want to generate all possible prompts. You can tick the combinatorial generation checkbox to enable this:

A {painting|sketch|drawing} by {artist1|artist2}

This will produce

A painting by artist1
A painting by artist2
A sketch by artist1
A sketch by artist2
A drawing by artist1
A drawing by artist2

If you are using long wildcard files, then your combinatorial generation could produce far more prompts than you intended. In other words, if colours.txt has 50 colours, shapes.txt has 20 shapes, and artist.txt has 100 artists, then

A __colours__ __shape__ by __artist__

will produce 50 * 20 * 100 = 10,000 prompts. In order to avoid this, Batch size and Batch count are used to calculate the maximum number of prompts generated. So if Batch size is 4 and Batch count is 4, only the first 16 prompts are used to generate images.

Magic Prompt

https://github.com/adieyal/sd-dynamic-prompts#magic-prompt

Writing a basic prompt is easy, but coming up with interesting modifiers can be challenging. Magic Prompt automatically runs your prompt through a neural network that updates your prompt to produce a more interesting image. For example, when you enter:

A woman on a windy hill

which produces something like this:

Magic Prompt might generated the following:

A woman on a windy hillside overlooking a magical forest, she has a pet wolf, delicate features finely detailed perfect art, gapmoe yandere grimdark, trending on pixiv fanbox, painted by greg rutkowski makoto shinkai takashi takeuchi studio ghibli
A woman on a windy hillside overlooking the city, rainy weather, close up shot, details, sharp focus, elegant, highly detailed, illustration, by Jordan Grimmer and greg rutkowski and PiNe(パイネ) and 薯子Imoko and 香川悠作 and wlop and maya takamura, intricate, beautiful, Trending artstation, pixiv, digital Art
A woman on a windy hillside overlooking a floating city made of water and ice, Fade Ninetails, highly detailed watercolor, trending on Artstation, highly detailed, incredible art, illustration, concept art, matte, art by WLOP and Artgerm and Greg Rutkowski and Alphonse Mucha, masterpiece
A woman on a windy hillside overlooking a magical forest, she has rainbow hair, flowing hair, wide hips, fantasy, bangs, symmetrical facial features, intricate, elegant, digital painting, concept art, smooth, sharp focus, illustration, from Valerian and the City of a Thousand Planets, from Star Trek 2021, illustration, by WLOP and Ruan Jia and Mandy Jurgens and William-Adolphe Bouguereau, Artgerm

and produce these images:

Magic Prompt + “A woman on a windy hill”

I’m Feeling Lucky

https://github.com/adieyal/sd-dynamic-prompts#im-feeling-lucky

If you’re lacking in inspiration you can use this option to automatically search through the Lexica.art database and return prompts that match your search string. A search for Mech warrior returns the following:

Searching for mech warrior using I’m feeling lucky

You can now tweak the prompts returned.

Jinja2 templates

https://github.com/adieyal/sd-dynamic-prompts#jinja2-templates

If you want a more extensive templating language, you can enable Jinja2 templates. This feature is only recommended for advanced users. Here are some basic examples of what you can do:

Conditionals

{% for colour in ["red", "blue", "green"] %}
    {% if colour == "red"}
        {% prompt %}I love {{ colour }} roses{% endprompt %}
    {% else %}
        {% prompt %}I hate {{ colour }} roses{% endprompt %}
    {% endif %}
{% endfor %}

Setting variables

{% with careers = ['doctor', 'lawyer', 'accountant'] %}
	{% for career1 in careers %}
		{% for career2 in careers %}
			{% if career1 != career2 %}
				{% prompt %}professional digital airbrush art of A {{ career1 }} and {{ career2 }}{% endprompt %}
			{% endif %}
		{% endfor %}
	{% endfor %}
{% endwith %}

Weighted sampling

My favourite colour is {{ weighted_choice(("pink", 0.2), ("yellow", 0.3), ("black", 0.4), ("purple", 0.1)) }}

Collections

Creating your own wildcard libraries is hard. We provide wildcard collections that you can use as is, or to customise for your own needs. Two collections are currently available:

Jumbo — 380 wildcard files with just under 16,000 wildcards, divided into categories such as aesthetics, appearance, artists, medium, style, and time.

Parrot Zone — 19 files and 818 wildcards. These were obtained from [Stable Diffusion Artist Style Studies](https://proximacentaurib.notion.site/e28a4f8d97724f14a784a538b8589e7d?v=42948fd8f45c4d47a0edfc4b78937474)

Complementary Extensions

Dynamic Prompts plays nicely with other great extensions such as:

X/Y plot to simultaneously tune parameters and prompts.
Randomize — an extension that randomly varies generation parameters such as seeds, steps, eta, samplers, and other variables. https://github.com/stysmmaker/stable-diffusion-webui-randomize

Conclusion

There are more features that I haven’t included here but hopefully this quick write-up gives you a sense of what the extension does. I find that developing prompts using this extension a much more enjoyable experience than staring at an empty text box trying to craft a new prompt from scratch.