Unity Muse: Testing the pre-release. Sprites, Texture, Chat

9 min readNov 23, 2023

Unity Muse logo, from https://muse.unity.com/

I have had the opportunity to test during some days a pre-release version (1.0.0-pre)of Unity Muse, and I would like to share with you the impressions. Are you excited as I am? Let’s go!

Features included

Muse adds to Unity IDE in the 1.0.0-pre version by Nov, 23, the following features:

Text-to-Sprite generation;
Text-to-Texture generation;

Also, as part of the release, they include separately from the IDE…

A Conversational chatbot called Unity Assistant, to guide you in your development process;

I’m in the waiting list for two other modules (I will write the chapter 2 after I got access to them), which are:

Muse Animate: A Text-to-Animation engine;
Muse Behaviour: A text-to-BehaviourTree engine;

What I figured out while subscribing for Animate and Behaviour is that they have planned also Muse Sketch.

If you subscribe to their newsletter & waiting list, they have a stealth Muse Sketch which seems to come soon

I could not find what Muse Sketch is about. I even asked the Muse Chat but what I got seems to be a hallucination based on Unity Muse Sprite:

Muse Chat, available outside Unity (see below for more details), when asked about Muse Sketch

My first guess was that it’s kind of a Text-to-Sketch but that seems to have been covered by the sprite modules for Muse we are covering in short in this article. So I will keep you updated as new details come up.

Unity Muse: Sprite module

The Sprite module of Unity muse is available from 2022.3 LTS on, and it allows you to generate custom sprites using Natural Language. You install it by following these steps:

Open the Package Manager: Window -> Package Manager.
In the Package Manager, click the + button and select “Install package by name…”
Enter: com.unity.muse.sprite

By adding that, you get the package installed in the Package Manager, all along with some common Muse library:

Pacakge Manager with Muse Sprite Tool installed

If you are testing it right now, make sure you update to the last version after installing it. As you see in the screenshot, I already have 1.0.0-pre.3, as previous versions for this and texture modules seems to be not correctly built and return errors in the interface:

Some errors arise after installing Muse Texture pre.1 versions. By updating to pre.3, they disappeared.

But after updating to latest versions, I got already a Toolbar section called Muse:

(At this point you will not have the New Texture Generator, only New Sprite and Style Trainer).

Style Trainer

Let’s first start with the Style Trainer. This feature allows you to:

Define a style name and a series of prompts you will use to test the style and decide which iteration in the training process of the style (yes, you train your own model!), is the best.

My style name is “Colouring book”, a common style used in other diffusion-based models to generate colourful book / card sprites

2. In Training Set, you add your own images (it asks for 5 initially) as a set with good representativity of your style.

Some images I have created with DALL·E 3 with that very same prompt, added as Training dataset.

In my case, I called it “Colouring Book”, as it was the style prompt I used in DALL·E 3 and Midjourney.

This is how the prompt works in DALL·E 3

Then, I aded some random prompts about the Lord of the Rings, as I (wrongly) presuposed the model will have some previously acquired knowledge about common fantasy worlds.

Testing prompts. They will help me to chose which is the best epoch

Up to this point, only clicking on Generate Style is necessary to trigger a 10-epochs training of 200 steps each.

The Generate Style button in Unity Muse Sprite

As the epochs go on, you will see the images generated for your prompts.

As the training went on, the outputs changed slightly. In some epochs the model was returning empty predictions for the first prompt:

But for the rest, it seemed to be creating better versions… or not (take a look at the 2 headed horse on the right).

No data from the internet was used!

At this step, an evident drawbacks became evident to me. Using known fantasy world, characters, and in general — proper nouns, does not seem to be very helpful for the prompt. It indeed knew what a Nazgul is (it created something similar), but it did not seem to heavily leverage previous experiences from the internet.

At this point I went to Unity’s documentation and understood the reason:

They claim no data from the internet were used. I don’t know how to feel about this - I evidently understand the ethics and privacy issues and it does not feel right towards the Art community. On the other hand, the tests I made were pretty disappointing — I’m not sure how Unity will try to catch up with other engines that will use scrapped data. The quality so far seems to be not very good, if we compare to what we can achieve with other diffusion models — What do you think?

These ones seemed as very early stable difussion or tiny / quantized models

Lessons I learnt

Several mistakes I may have commited:

Adding only a few examples of training data. I probably had to add more images (the interface was quite buggy only accepting PNG for drag and drop, and I had mainly JPEG, so I neede to select each of them manually.)
Using proper nouns and a known fantasy world, expecting the model would know.
I could have used better prompts.

Sprite Generator

After creating a style, you can use the Sprite Generator with that style.

Toolbar with the New Sprite Generator option

I liked some of the features I did not expect in the Sprite Generator:

The ability to remove background. This is a very cool feature I was expecting from DALL·E 3 and Midjourney., and makes a lot of sense for Sprite Generation. Good job, Unity!
Positives and negative prompts.
Weighting the impact of the style in your generated sprites.
Creating an sketch to guide your drawing.

These are REALLY COOL features. Unfortunately, my style was so bad that I could not really leverage them. So first conclusion: give some love to the first Style Creating step.

In general, the quality was so-so. I am very used to DALL·E 3 and Midjourney, so I had huge expectations, and this is what I got:

Again, the problem could have been the proper nouns about specific fantasy literature concepts. But the sprite on the right for example does not make any sense at all.

With more generic prompts the sprites were ok, but compared to the state of the art it felt underwhelming to me.

Prompt: a magic gold ring from a fantasy world

The coolest feature: as I’ve said, you can combine your style, your prompt AND a sketch. Again I blame myself for the quality of my sketch, which definitely impacted in the quality I’m complaining about.

Left: me not knowing how to draw. Right: the model not knowing how to draw.

However, I need to say that it was generating very cool landscape sprites, maybe due to the fact the training data I added was landscape related:

These I liked (the frames is probably my fault, my training images contained them)

Texture Generator

The Texture generator does not use the style. Instead, it asks for:

Obviously, a positive (and negative) prompt;
A color image;
A Canny-edged image;

I tried an Islamic-Kaleidoscope square texture, and some colors.

I’m not sure what I was doing wrong, but by using both the color and the shape, the results I was getting were pretty bad. Probably it’s meant to be used only one of them? (at this point there is no much documentation about it)

But removing the color and leaving just the shape, the results were better, but black&white:

Steel textures similar to fish scales? Looks good!

The only textures I could got working and I was proud of are those using the shapes by default included in Muse:

With that an a custom color, the textures started to look good:

Oh yeah, these already look professional!

As a DL person myself, I have the impression for both Sprite and Texture modules they don’t have much data. It works pretty decently for some inhouse datasets, but it does not seem to generalize well overall.

What was a good surprise for me is seeing how you can upscale, create more variations and transform to a PBR material.

Additional cool-looking features after creating textures

Muse: Chat (outside Unity)

Here you have a link to test the Unity Assistant, which can guide you throw your development process using the IDE. It seems to be trained using:

I tested the information about Tiles and Tilemaps, as it seems to me it’s not very popular. The result was satisfactory, including even additional resources and links:

A response about how to create Tilemaps with custom Tiles, by Unity Muse Chat

A question about how to create a Minimap with the ability to click on it and move the world camera to the clicked position in the minimap — it answered, although the response seemed like a bunch of pieces from different documents and not really a well-elaborated answered. Also, with the information provided, you can’t really create the minimap — it’s just some generic introduction:

A response about how to create a Minimap widget, by Unity Muse Chat

The first caveat I see is that they don’t seem to be using StackOverflow, Reddit or other channels where all the discussions are held. They stuck to their own forums, which again, as with the exampels of the Sprites and the Textures, seems very data-restricted to me.

Conclusions

Overall, I loved the ideas behind what Unity is building.

For the sprites, I loved the ability to train your own styles, to see how they included features from other libraries as the background removal. However, the quality of the results is not high, as they don’t leverage any internet data.
For the textures, I loved the upscaling and the ability to transform them into materials. However, it was pretty difficult for me to try to get a normal working texture outside their own suggestions.
The chat experience was just ok. It seems to be it’s a RAG system built on top of their Unity documentation, where the real power lies on the community threads outside their portals.

I will come back with more about the Animator and Behaviour Trees modules in Chapter 2. In the meantime, enjoy!

About me

My name is Juan Martinez, I’m a Sr. AI engineer who was been working in the field of NLP for about 15 years now. I’m now focused on Video Games Development and specially in the intersection between Video Games and AI. If you are in the video games arena or you just want to say hi, I’m happy to connect in LinkedIn !