Final Project Documentation
Remembering Nostalgia
Link to Editor: https://editor.p5js.org/kimhy1031/sketches/UPzyGNqtM
One sentence description: ‘Remembering Nostalgia’ is an interactive piece where a user answers a prompt related to the user’s past by drawing on the p5 editor canvas and re-visualize their work by applying Monet’s style.
Project Summary: Many people live their everyday lives without reflecting on their childhood. When the present is difficult, I always found it cathartic to draw/scribble or look up favorite toys I liked as a child on the Internet (interestingly, I still search for Tamagotchi on the Internet and I like spending my break time doing that). That is why I came up with ‘Remembering Nostalgia’, an interactive piece that is meant to give a break to users busy with their routines. It’s meant to be easy, so all it requires is some drawing. On a p5.js sketch, the user is given a drawing prompt from the header that aims to help the user look back on their past and nostalgia. For example, the prompts “draw your favorite childhood toy” and “draw your favorite childhood pillow” force the user to pull something from what they remember and draw it on the given canvas. By clicking on the “change prompt” button, they can change the prompt to their liking and preference. As the user draws according to the prompt, they can change the default color of the pen (black) to a color on the palette (red, blue, green) to reinforce their drawings. As another form of visual effect, they can add a noise background to their drawing to add a vintage feeling, something that’s associated with remembering the past. After they finish answering the prompt with their drawing, they click on a button that reads, “Change my drawing!”. While this is going on, Runway’s Adaptive Style Transfer will be running in the background, with Monet as the style checkpoint. When the button is pressed, Monet’s painting style will be transferred and applied to the user’s drawing, adjusting the colors of the drawing as necessary. As a result, the drawing would possess Monet’s painting texture, style, and colors, in which smooth strokes and the light blue/yellow color give off the vibe of dreamy and nostalgic. Seeing Monet’s style applied on to their drawing, the users are given a moment to enjoy how their drawings have transformed and actually engage in remembering the object they’ve drawn, as Monet’s paintings are known to induce some sort of nostalgia or a romantic feeling.
Inspiration: I was initially attracted to the idea of making a simple drawing come “alive” by adding real-life patterns to it. I was fascinated by how a dad transformed his son’s rough drawings to real-life animals/objects that the son tried to draw by adding real texture and background. I thought this was a good concept to help people re-imagine their work and know that nothing is too simple for anything. Since this is a class that fuses machine learning with arts, I thought of using machine learning to help make art come alive.
Process & Code References:
Initially, my idea was to have an “artist palette” where the user could apply different styles of different paintings to their drawings according to what theme they wanted to associate their drawings to. For example, the palette would have a Monet option and a Picasso option that relates to the theme of nostalgia and perplexity, respectively. I wanted to use Runway on this since it already has pre-trained models, but Ellen told me it’s not possible to switch between checkpoints on the current version. Thus, I looked into the Style Transfer model from ml5.js library because I could train my own models and achieve the palette I want. I referred to Yining Shi’s tutorial on how to train a model using Spell, a tool we went over during class. However, I wasn’t lucky with using Style Transfer as I wasn’t very knowledgeable about Terminal, and I perceived that the training would take forever if I were to train multiple paintings of different artists.
Slightly simplifying my project at the last minute, I went back to Runway because it was more doable for me. This project was constructed strictly using p5.js and RunwayML. Both applications were dependent on each other to create the final result. I first explored Runway to see what kind of models would best support my vision in “re-imagining” art. The Adaptive Style Transfer and the Fast Style Transfer were the closest ones, but I went with adaptive style transfer since it has the nostalgia-linked art styles more than Fast Style Transfer does. I liked Monet’s painting effect because it was dreamy, and I have always been a fan of Monet’s art. After choosing the model and the style checkpoint, I went to p5 web editor to create a doodle canvas, a color palette, a function that generates random prompts related to nostalgia, and other necessary elements such as buttons. To create the drawing canvas, I referred to Yining’s code. Then, I filled in the HTTP request code, ran Runway remotely in the background, and started drawing on the canvas to test if it’s working. Luckily it worked, and from that point, I kept testing and refining what was already working.
Audience/Context: I believe this project is best suited for adults like myself who are becoming more distant from the past as we age. Whenever I am bombarded with homework and work, I relieve my stress by drawing or coloring pages. I wanted to provide the same experience to people like myself who are busy and stressed, and wanted them to focus on the theme of “nostalgia” for a double healing effect. I wanted them to relieve stress through drawing & remembering their past at the same time. Prompts such as “draw your childhood crush” make it a fun, interactive experience. It’s for fun, but at the same time, emotional.
User testing: Before the user testing session, I only had the drawable canvas and the submit button where the transfer would start. During the user testing session, I received the following feedback from my peers:
- Using a trackpad to draw is uncomfortable, use a mouse or an iPad instead
- Add more effects to drawing
- More simple and specific prompts that are related to small objects (a suggestion that given was “gift”)
- Add hue/saturation effect to personalize the input and output more.
I applied all feedback, except I didn’t implement the idea of using iPad to draw. I couldn’t figure how to do it, honestly, and I couldn’t find an iPad pen to borrow as well. If had more time, I would have figured
Next steps: If I had more time to work on this, I would definitely trying connecting the iPad & pen to replace the trackpad for drawing. Also, I would try to go back to ML5 Style Transfer model and take the time to drain different artists’ paintings.
