Inside Scratch Lab: AI Image Generation
This series gives you a look inside Scratch Lab, where we help invent the future of creative coding for kids. See also: The Making of Animated Text
What if AI could help you create whatever you imagine, in Scratch? What if you could make any character or backdrop image you want, just by describing it?
We’ve been experimenting with AI image generation in Scratch Lab. In our prototype, you can type in a short description of whatever character or backdrop image you’d like. A few seconds later it’s ready to be included in your game or animation.
It feels amazing: your ideas become real in seconds. Each AI creation helps generate new ideas for more things to create. You can still draw your own sprites, or add them from the sprite library, and you can edit the AI images to your liking. It doesn’t replace your creativity — it amplifies it.
As we imagine the future of Scratch, we’re trying to understand where generative AI might fit in. Just like kids using Scratch, we’re tinkering and reflecting. Here’s what we’ve learned so far: pitfalls and possibilities.
Pitfalls
There are several reasons we aren’t sharing our prototype with the public, from bias, to cost, to credit, to weirdness.
Bias
If you ask our prototype for a picture of a computer programmer, it only produces images of white men. We want kids using Scratch to be able to imagine themselves in whatever role they choose, regardless of their own race, gender, culture or other identity. So we see this type of bias, reinforcing structural inequities, as harmful to kids. We used DALL·E from OpenAI to make our prototype — OpenAI has written about their efforts to mitigate bias, but the problem persists. This problem alone makes the technology inappropriate for use by children, especially without additional context to help them understand and critique it.
Inappropriate Content
OpenAI provides its own content filter for DALL·E to prevent people from creating images they consider inappropriate–but it’s imperfect, and their standards for things like nudity and violence are different from those used by the Scratch Foundation.
Credit, Compensation and Copyright
AI tools are created using millions of images, generally without the consent of the original authors of those images, and without credit or compensation. Their legal status is unclear, since most are created using copyrighted images. The Scratch community values remix culture — we celebrate sharing and creative reuse, as shown in our community guidelines. But we place a high value on giving credit to the original creators.
Cost, Access and Privacy
AI image generation, as of mid-2023, is not free– it’s generally a paid service. Scratch serves millions of kids around the world for free, so we’d want to find a way to provide free image generation. It’s also not generally available offline, so it would be inaccessible to the many people who use Scratch without connecting to the internet. And because it’s an online service, there could be privacy problems.
Getting the Image You Want
There’s an art to using AI image generators. Adding styles or artists can work well (like “cartoon” or “Van Gogh”). It helps to visualize and then describe in detail—you’ll get very different results for “turtle” vs. “top-down view of a cute green turtle in 8-bit video game style.” There are some limits on DALL·E’s understanding of syntax, so it can be tricky to get parts into a specific relationship, especially if it’s unusual (e.g. “a turtle riding on the back of a giant cat” might yield a cat riding a turtle instead). Learning how to write prompts is a learning experience, both about AI and about the creative use of language. Scratch ideally would include a system to help kids learn the skill of prompting.
Weirdness
AI image generation results, in 2023, can be weird. It’s both fun and frustrating. Human faces and hands may be distorted, and text in the image is usually gibberish. The image may be cropped at the edges. Getting a good image can take a few tries.
Possibilities
The world of AI image generation is changing fast, and it seems likely that some or all of the above challenges will get easier. Here are some reflections on what the creative opportunities are now, to help imagine what they might look like in the future.
Expressiveness
It takes just a few seconds to generate each AI image, meaning you can quickly experiment and try out ideas. You can make a picture of almost anything you can describe. In a Scratch editor with image generation built in, you can quickly go back and forth between generating images, editing them, and coding your projects. It could accelerate kids’ ability to bring their imagination to life.
Convenience
If you search the web for an image to use in your Scratch project, say, a penguin, you’ll need to carefully cut it out from its background in order to use it as a sprite. In my prototype, I realized we could ask the AI to help with that—I secretly add “on a white background” to each prompt for a sprite image, and then remove the white pixels. The result is a sprite on a transparent background that is easy to incorporate into a Scratch project.
AI image generation could also help kids make projects that reflect their own culture and interests. A kid in India, for example, may want to make a project with an auto-rickshaw sprite. The Scratch sprite library doesn’t have one (though we’re investigating ways to provide a wider variety of culturally relevant images within Scratch). AI has an ambiguous role here: it can generate incredibly diverse images, including ones that might otherwise be excluded, but its bias can reinforce harmful stereotypes.
Some Scratchers may find these kinds of convenience especially valuable, if they’re less interested in drawing and would rather put their creative energy into other aspects of Scratch, like storytelling, sound and music, or coding.
Creative material
AI image generation doesn’t replace the creative process — it becomes part of it. The images are a material for creativity, because they can be edited inside of Scratch, to add or remove or change them. You can even combine multiple generated images into one.
Creative feedback
AI image generation can be like a creative conversation, as you go back and forth between generating images and coding a project. Often, the AI will add a strange detail to an image, or interpret the prompt in an unexpected way. This can lead a new idea, which takes the project in a new direction.
In one case, for example, I wanted to make a project about an alien planet. The AI generated a wonderful alien backdrop, but it included something I didn’t mention in the prompt: a big weird spherical thing. My imagination turned this into an alien egg, and I used the AI to help me create an alien that I could animate hatching out of it — and that got me started making up the story for my project.
Another time, I set out to create a story about a dwarf character going for a walk. When the dwarf got outside, the generated backdrop included a strange object in the background. I prompted the AI to imagine this object up on the hill, whatever it was — the AI turned it into a mushroom. Then I decided the mushroom would grow bigger and bigger, which took the story in a fantastical direction. The ideas in the story are a result of a creative conversation between me and the AI.
Starting With Images
What if the process started not with a text prompt, but with an image, maybe even a drawing you make yourself? This could make the AI fit more intuitively into kids’ creative process, and invite more playful experimentation. We’ve played with a few techniques that work this way:
Variations
You can give the AI an image, and it will create alternatives with the same style but different details.
Inpainting
Start with an image, remove part of it, and have the AI fill in just that part using a prompt.
Filling Outlines
An AI technique called ControlNet takes the outlines from a drawing and fills them in using a prompt. You can sketch the outline of a creature, say, or a spaceship, and let the AI fill in all the details.
Of course, we don’t want people to think their original drawings aren’t good enough. But AI could add powerful new tools for those who want to experiment with them. Each of these techniques starting with images could add new expressive capabilities to Scratch that fit naturally into kids’ existing creative process.
What’s Next?
Because of the many challenges with AI image generation, we won’t be including it in Scratch any time soon, even on our Scratch Lab site. The Scratch Lab team will continue to keep an eye on progress in this area, as we tinker with it to help us understand the possibilities. We’re reflecting broadly on the role of AI in creative learning, guided by this essay from Mitch Resnick. And we’re exploring other uses of AI, beyond image generation (what if you could use AI to generate animations, music, or sound effects for your project? What if we had a block that could create new costumes from a prompt? What about generating chat responses, project ideas, or even Scratch code?). As always, we’ll be looking for ways we can improve Scratch that align with the vision of the Scratch Foundation: to spread creative, caring, collaborative, equitable approaches to coding and learning around the world.
Acknowledgements
Thanks for support and feedback from people at the Scratch Foundation and the Lifelong Kindergarten Group, especially (alphabetically): Karishma Chadha, Melodie Deisher, Eduard Muntaner Perich, Mitch Resnick, Eric Schilling, Ben Wheeler, Annie Whitehouse, Chris Willis-Ford, and Thaís Xisto.
Except where otherwise noted all images in this post were created using DALL•E
