Using OpenAI to Create a UX Portfolio That Can Talk
Thoughts on using OpenAI’s API to create an app that answers questions about my UX case studies
I started a side project, Ask My AI on a lark. OpenAI released a tutorial on training its text complete API on specific web site content right at the time I was updating my UX design portfolio. That gave me an idea: What if I created an AI-powered chat based on my case studies? Could I create a portfolio that could be asked questions?
I got OpenAI’s crawler and tokenizer code up and running pretty quickly, then integrated everything into a Python Flask framework and created the UI. Here are some thoughts and observations I gathered during the process.
How it works
Ask My AI enables anyone to ask my portfolio questions about my experience and offers AI-generated answers based on my case study content. In addition, the AI detects what parts of my portfolio were used in generating answers and creates contextual links.
Dispelling the Mystique
The process of creating Ask My AI took some of the mystique out of AI. Part of that comes with being hands-on: You get content, format / tokenize it, pass it to an API, and it returns results. (The API itself uses natural language, which is cool. I’ll talk about that later.) But what I found revealing has to do with the nature of the content: I created every word of the source material.
I know what my AI should say
Every answer generated by Ask My AI is based on around 6,000 words, all of which I wrote for a specific purpose: to convey my experience. I know what questions prospective employers tend to ask, and the answers to those things are very much in my case studies.
Some AI-generated answers are right-on
The above question, “What is an example of a project you worked on?”, is a fairly common screening interview question. The answer based on my case studies and is totally AI-generated. It has a bit of a high school-essay vibe, but is otherwise a very good summary of one of my projects. What I really liked is that the AI created a link to a section of the case study that is nicely related to the answer. On top of that, it’s a topic (rapid prototyping) that I like to emphasize.
Some answers could be wrong in subtle, concerning ways
The answer to the question, “How do you know if a design is successful?” includes a lot of things I would say, but also a phrase I would not: “…getting feedback from stakeholders”.
That’s not just wrong, it’s an anti-pattern. Evidence-driven outcomes from customers ultimately should determine success. Stakeholders should be bought into that.
Part of the problem with the AI generating an answer with a design anti-pattern is in how “confident” it sounded. It slipped the bad information into an otherwise good answer, and made it sound conversational.
Answers could be way off the mark
The above AI generated answer to the question, “How do you work with engineers” is really bad*. The phrase, “make sure they know what is expected of them” makes me cringe and is indicative of an immature designer who has been working in dysfunctional product orgs. (My portfolio conveys a very different way of working with engineers. I also write about this subject.)
In this case, the “confidence” is projected through the conversational tone. A friendly-sounding chat belies a truly bad answer.
*I should note that when I asked the same question again, follow up answers were much better, especially when I adjusted some params (see “Experimenting with randomness” below).
Natural language API calls are weird but cool
In the past, API params might look something like this:
You let the API know what you want by passing specific variables in some predetermined format like JSON. OpenAI’s API follows that pattern, but a lot of what you want as a developer is contained in the prompt itself and is in natural language.
I was particularly amused when I wanted responses to list source material from my case studies, the API started returning completely made up sources (all of which sounded plausible). I added the command, “Do not make up sources”. It seems to have worked.
Experimenting with randomness
OpenAI also has params with a more traditional key:value format. One is called temperature, and it controls the “randomness” of answers. Low values will make answers focused and higher values more random.
I founds through experimenting that there isn’t a perfect temperature setting. If the temperature is low, answers tend to lack a conversation tone and seem more like snippets of content rather than a conversation. When the temperature is too high, answers are more natural-sounding, but can stray away from what is in my case studies.
In the end, sometimes low temperature works well for some questions, and higher temperature works well for others.
Closing thoughts
In building this prototype, it was interesting to entertain the idea that AI might work for individuals. Although imperfect, Ask My AI is designed to act as my agent: It’s trained around my intent (i.e., find a design role that benefits from my skills and experience), my relevant knowledge (i.e., case studies), and acts on my behalf (i.e., interface with recruiters), all-the-while remaining (mostly) under my control and supervision.
I think we’re still a ways off from getting to the point where this would work they way I imagined. What became clear in creating this app is that all this type of AI / LLM does is create plausible-sounding answers, which, don’t get me wrong, is impressive, and very useful in a lot of applications. The problem with Ask My AI is that about 10–20% of its answers are wrong, and some are very wrong . That means it would blow most screening interviews.
Despite its imperfections, Ask MY AI was still fun to build. What are your experiences working with LLM/AI APIs?
P.S., Here’s my prototype: https://kgates-portfolio-app.herokuapp.com/askmyai
