How to use AI to build functional UX prototypes
Building functional prototypes without prior programming experience with OpenAI’s Codex Javascript Sandbox
Designing with Natural Language
Every design starts out as a collection of descriptions of an intended design, and UX designers and Visual Designers create those descriptions with 2D vector software to demonstrate, annotate and communicate the design for business to discuss and engineers to produce. Now with AI, it’s possible to produce code from natural language descriptions, meaning components can be contained in a well written plain text description!
By describing an application build piece by piece, you can build surprisingly functional prototypes with AI.
*March 2024 update*
I’ve since made an LLM powered software design and development assistant that helps prototype by generating custom code based on your designs!
It’s a free Figma plugin check it out at :
— —
*April 2023 Update*
Since this article came out six months ago there have been a tremendous amount of developments in the field of AI, largely driven by OpenAI. The DaVinci Codex model has been depricated (and then brought back for research purposes), Davinci Text 003 came out, followed by ChatGPT powered by the GPT-3.5 Turbo model, and finally GPT-4. Interestingly, and to some extent surprisingly, general models like GPT-3.5 and GPT-4 outperform models trained specifically on code like DaVinci. The popularity of GPT in general has exploded since then, and the views of this article have taken off. Surprisingly, much of this article is still relevant in terms of approaches and guidance on natural language powered instructional development of UX prototypes, but the Javascript Sandbox is quite outdated. It’s incredible how fast things are moving in AI. That said, I do want to update this for those reaching this article with regards to getting the same things done, but this time with GPT-4.
While it was helpful to see builds emerge in the Javascript Sandbox, GPT-4 doesn’t have that capability, but paired with VS Code, GPT-4 is very clearly more capable and usable by a large margin. GPT-4 is able to write better code and with less explicit instructions. You can ask it to write you a formatted HTML document with CSS, HTML, and Javascript all embedded, and after prompting with all the components and their intended design, ChatGPT 3.5 and moreso GPT-4 can produce all the code and link it together to be quite functional. Where with the DaVinci Codex model you had to spell everything out, GPT-4 can fill in a lot of gaps, but it still, like any developer, will need thorough instructions on what to produce.
So I went ahead and tried similar tests with OpenAI’s newest model (as of April 2023), GPT-4 to see how it stacks up against the Codex DaVinci 002 Model as is defined further below.
First, we can try the toggle with GPT-4. Here the prompt is to “create a toggle switch in a single HTML document”.
I sped up the video because right now GPT-4 talks a bit slower than GPT-3, but here is the response in action as viewed in VS Code:
Without even describing the toggle itself, it understands the concepts of the shapes, colors, and interactions of a toggle switch, and the keyword itself is enough to reproduce it.
What about a slightly more complex prompt, something like “make a button with 8px corner radius that has hover state and clickdown microinteractions in a single html document, use a colorful gradient on the button and white text, and make the design modern and feel like an Apple design”.
As you can see for yourself it has created a pretty contemporary design where the button appears to raise on hover state (through larger shadows) and gently pushes down (limited shadow) with a smooth 0.3 second transition.
Tips
“Continue” still works well, though for complex builds I’d recommend asking for specific adjustments and additions, alongside instructions on where to insert or adjust the new lines of code into your own document in VS Code. Because the entire conversation is contained as the context window, GPT-4 will be able to refer to the document and placements of new code, and understand the most recent or most up to date version of the document. You can ask questions for help and you can drop new pieces of code in based on your description of the design and its suggested instructions for changes to the document. Asking it to “write the code to add a button to a page, and tell me where to paste it into the document” is a good approach.
It’s worth reading the rest of the article because many of the conventions and approaches discussed here are based on interacting with AI in natural language to accomplish design and build tasks, and are not necessarily dependent on a specific model or interface, but I strongly recommend using GPT-4 over ChatGPT (3.5 turbo) due to reduced hallucinations, and more complex and properly formatted documents.
(October 2022 Original Article)
Language as Code
AI models are described as prediction machines for their ability to anticipate text following an input. OpenAI’s Codex model was trained specifically on code, which it will write when instructed to make something.
Github Copilot, which uses OpenAI’s model, performs a similar function as OpenAI’s beta Codex Javascript Sandbox, turning comments or instructions into code within VS Code, but this is for starting out and experimenting within OpenAI’s Javascript Sandbox itself. I have basically no experience with programming, and without a doubt there are better ways to implement these.
I don’t think AI can “replace” or obsolesce developers, it’s simply another tool that designers can use to better prototype internally, test externally, or communicate with developers. There’s an art to developing hardy applications and Codex can’t do what you don’t communicate to it. That said, it is possible to host these HMTL files on the web after obfuscating the code in order to interact with the builds online.
Prototyping
A solid, well researched product will go through testing phases, where a certain level of functionality is brought to the design in order to see how it performs in relation to user expectations and interactions. This makes for an important step in validating correct hypotheses and challenging any that prove to be incorrect, as well as eliciting unexpected insights.
One difficulty with prototyping is that most of the changes in the screen have to be designed for specifically with their own artboards, meaning a complex testing flow is going to look like a bird’s nest with tons of screens and even more connections, and will still only accommodate the exact flow specified in the prototype. Keyboards aren’t really functional, fields don’t really update, and calculations are not performed by the prototype but rather manually added to the design. This leads to a lot of dead ends in testing prototypes, where users can’t go forward or enter data that isn’t specified in the research protocol and test instructions.
By taking the concept of designing or prototyping outside of the conventional 2D image based approach, and into a natural language one, we can produce prototypes that allow for free-flowing user journeys or high-fidelity functionality if necessary.
These prototypes can be near fully functional, but the process has its own learning curve and a few drawbacks. In many cases it won’t replace designing prototypes on software like Figma or XD, which is faster and more to the point if examining a narrow experience flow or a simple interaction. But if your goals rely on a functional application to test, one with more free flowing user interaction than just guiding them on a list of predetermined, predesigned tasks, then this form of prototyping for internal or user testing can be valuable. Though it doesn’t replace the value of rapid prototyping with visual software, it does extend the abilities and range of the prototypes that can be made.
Users can enter any flow, interact with real platform keyboards and the front end can calculate any information and set basically any conditional logic for the UI or calculations. It can take a while to get the hang of it, but design in the future may involve working with AI to build better prototypes and demos for clients or devs.
Example of a simple toggle build
Vague prompt:
“Make a clickable toggle component”
Detailed instruction:
“Make artboard: a card 20em wide by 40em height, 0px 0px 15px background shadow. Center card on screen. Make border radius 1em. Make rowItem: a div 4em height, full width 100%, appended to artboard, 8em down from top. Make line1 and line2: 0.05em height divs, grey, 100% width, positioned at the top and bottom of rowItem.
Create toggleBack: a div with rounded corners, grey background, transition .4s, centered vertically. Append toggleBack to rowItem, aligned to the right side of rowitem, with right set to 2em. ToggleBack is 1.5em tall, 2.5em wide. Make a white circle diameter 1.3em named ‘toggleselection’, position it on the left side of toggleback set left to 0.1em. Center the toggleselection vertically in toggleBack.
Make toggleselection transition of .4s. Make artboard transition set to .4s. Onclick of toggleback, if toggleselection left is 0.1em, make toggleselection left 1.1em, and change toggleback background color to green, and change artboard background to black. Onclick of toggleback, if toggleselection left is 1.1em, make toggleselection left 0.1em, change toggleback color to grey, and change artboard to white.”
Of course this is a small component, but the opportunity is in layering these prompts and their outputs.
Building specific components and overarching functionalities
It’s hard to know what descriptions your instructions are missing until you see Codex’s interpretation of it. When entering requests with missing instructions, elements can be out of alignment, or types of placeholders for fields or attributes can pop up in the code.
OpenAI’s Codex Javascript Sandbox is currently in beta, so you can apply to join the waitlist: https://openai.com/blog/openai-codex/.
Recommendations
Design the wireframes & visuals, even if it’s for yourself
Copy is relatively easy to update post-build, but interactions and visuals should be fully thought through ahead of time and wires detailed, outlining responsiveness, names, components, opacity changes, and transitions. It’s a lot easier to build it properly the first time than it is to go and change things after! Tweaking finished designs isn’t too difficult, but reordering layout, page structure or flows can be relatively time consuming.
Use a text document for prompt management and version control
It helps to work with a prompt document instead of within Codex because it will save your prompts as the prototype is developed and updated. Consider saving these prompts for rebuilding lightweight versions of components so that you can layer on more functionality, view elements, test things, and paste the outputs into your file on VS Code.
Use VS Code in partnership with Codex Javascript Sandbox
You may want to download VS Code so you can take the code over to VS Code to save your progress in an HTML document. Codex can crash after a lot of requests so you’ll want to pull code out of Codex and into VS Code, but Codex is where you’ll be seeing what components you first build. It will allow you to save project files and build more complex prototypes than what the Sandbox viewer can handle. OpenAI codex will crash from time to time, so it’s easier to keep the builds in the Codex Javascript Sandbox light and representative of the specific related components in your app/website. You can export the code from the Sandbox to JSFiddle, where you can copy the whole document and paste into an HMTL doc on VS Code, adding snippets from there.
Name your components!
One way to handle complexity in a doc where a lot of elements or functions are present at once is to name the components logically as they’re built. Instead of writing the instructions alone, include a name of the element or function. Otherwise, Codex will come up with a name itself and sometimes in confusing ways that become difficult to remember or distinguish from eachother, causing problems when writing new instructions.
Refer to names
When elements or functions are created, they’re created in relation to other elements, in that a function is performed upon certain actions, or an element is anchored to a certain component. By referring to elements themselves rather than descriptions of them, it helps to increase the effectiveness of the responses.
i.e. referring to “incomeButton” instead of “the third button” helps Codex understand what exactly you’re referring to.
Keep in mind the order of elements and their functions
If you want to calculate something, your functions have to be referring to certain elements, meaning the elements have to be described before functionality can be. It’s straightforward but necessary to note that the order of descriptions is important. It helps to build the structure and visuals and then apply functionality to them after.
Build components with appendchild
When building something on Figma or XD, it’s common practice to group components together to be able to easily edit, move or reorder them. It’s useful here too, because it would take a lot less time to tell the program to move a component — and all that is appended to it — than it would be to move each element the same amount. “Appendchild” is a bit of language for attaching elements to each other.
So for example, when creating a component, you can make the div for the back or container, then append the icons, text and indicators to the back or container itself by adding ‘append to container’ to the end of descriptions of those elements.
Use commas, periods, and enter to distinguish sets of instructions
Commas help to continue a description of a certain element, whereas periods help to separate different elements if building multiple elements in one prompt entry. Enter creates a line break that separates chunks of instructions so that you can build the components, and then link them up with events or conditions. Sometimes separating entries doesn’t work well when sets of rules interact, meaning the logic is often written more effectively if Codex is aware of all rules or actions related to the same event.
i.e. in the case of the toggle, all three actions linked to the onclick event of the toggle (left/right position, toggle color, & artboard color) were in the same request rather than separate successive prompts.
Familiarize yourself with the basics to inform instructional language
Knowing the name of events, conditions, and attributes is like a shortcut when writing prompts. This is ultimately a programming tool and the AI has to know what you want, so you have to be able to communicate what it is exactly that you want in terms it can understand.
Elements, like divs and text, make up the objects in the program, defining the structural and visual aspects.
Events, like mouse hovers, onclick, keyups, keydowns, and timers, drive interaction. Terms like ‘on click of X’ or ‘on mousehover’ tend to improve response quality as opposed to something like “when you select __”.
Conditional if, else statements tell the program “on click of this, if condition is met, do this”, and else statements add the layer of “otherwise, do that”.
Search stackoverflow or google what you’re trying to do to find more accurate language.
By finding the names for functions or descriptions of what to do, you can more accurately tell Codex what you’re looking to see built. Phrases like “on click of X, do Y”, “append X to Y”, or “if this, then do that, else do that” help the AI connect it with the equivalent in programming language. (Those familiar with programming likely know stackoverflow, but it may be new to some designers.)
Use the edit feature on the right side panel
The edit panel on the right side of Codex Javascript Sandbox allows you to tweak attributes to adjust them to however you need them. You can paste in new attributes, edit values, alter responses to your liking and save by clicking back on the checkmark. Keep in mind that there’s no undo button for these changes.
Continue Prompt
When output exceeds the max tokens in the request, OpenAI will stop short of completion. If you’re following along with the output and you can tell that it is producing unique useful code instead of just repeating itself to infinity, then you can instruct it to continue. For example, when instructing Codex to create a set of particles with multiple layers of behavioral rules applied to them, if Codex stops short in its complex response, you can say something like “continue making a set of particles with the behaviors as instructed above”, or just “continue”, and it should carry on where it left off.
Limit Prompt
Sometimes codex will continue with a request, listing all possible attributes and filling them with guesses, or even just repeating itself in a loop. Codex doesn’t inherently know how long of a response you want, or that you don’t necessarily want it to add every possible attribute. In the case that we wanted to make particles move, but only the first rule of their behavior, you can request “set particles to attract toward the center of the group, and nothing else”.
Response errors
AI models don’t produce typical ‘error states’. If a response is incoherent, or broken, the AI can’t really tell, and it will display the same as a functional prompt. A prompt that doesn’t produce a design sometimes just needs improvement. A common error with AI is the infinite loop, where it begins to make up attributes and repeating them to presumably infinity.
Give as much detail as possible in terms of what you want.
Telling it to “build an app” or “make a website” will often result in a minimal response because it doesn’t actually know what to build, so it opts to fulfill the request with as few tokens as possible.
In this case, in telling it to “make a chicken sandwich”, there are a ton of ways to do this — it could be a visual depiction or it could pull up the URL of a picture of a chicken sandwich, but with such a short prompt without many details, Codex solves the problem with the least amount of tokens, by layering three bits of text, technically solving the problem, but in an unexpected way.
“Bread
Chicken
Bread”
Prototyping with AI
Components become descriptions when AI can translate instruction to code. What would be helpful is a natural language component library, a collection of natural language descriptions to help people build components faster.
It is interesting to think of UX design as involving prompt design in the future. Much of what UX design is about is communication of designs and interactions, something we do with a combination of language and image based wireframes. Much of this language is already familiar to a UX designer, making for a smoother transition to building with prompts.
Thoughts
What do you think about impact of prompt design on UX field? As the debate rages on about “should designers code”, this may add something to that fire by asking “should designers code with AI?”.
At the very least, if you’re interested in learning some of the basics around these programming languages and are more of a hands on learner, Codex / AI in general is a great way to learn!
Follow me on twitter @pagegrids for more thoughts!
