Enter the Art
We have invited artist and creative coder Andreas Refsgaard to develop a tool to make it fun and personal to explore the SMK collection. Here, Andreas tells the story of how he’s making it possible for you to jump into your favourite artwork.
A few months ago, SMK approached me with the task of helping them explore new ways for guests to experience and establish connections with the paintings from the SMK collection. They wanted to use new digital tools to extend and transform the interface or connection between the visitor and the artworks, and create fun and/or beautiful ways for people to picture themselves in the art. After exploring a few different ideas I focused on how to allow people to combine photos of themselves with a handful of iconic paintings. Going beyond just swapping faces, I imagined how people could take photos of themselves and place their bodies wherever they wanted to inside an original art piece. However, copying elements from photos and pasting them onto a painting without having the end result look like a collage of unrelated separate elements is a challenging task. A bit of research led me to a scientific paper by researchers from Cornell University and Adobe Research who presented a technique called Deep Painterly Harmonization. Their approach allows you to use segmented cutouts of faces and bodies and seamlessly merge these onto paintings as though they had appeared there in the original painting.
In order to allow people to see themselves painted onto the pictures of SMK, I imagined a website experience that gives visitors a lot of creative freedom. Because of COVID-19, it also made sense to make something people could experience from their own home, without necessarily visiting the physical museum. The proposed user flow was as follows:
1) A person starts out by capturing or uploading a picture of themselves from their computer or mobile phone.
2) A machine learning algorithm (like BodyPix directly in the browser or via Person Segmentation from RunwayML server side) performs segmentation and creates a mask containing only the part of the picture where the person appears.
3) The person positions the isolated image of themselves anywhere on top of a classic painting from the SMK collection.
4) Using the Deep Painterly Harmonization technique, the person would be merged with the original art piece, creating a new version of the painting as if the person would have been part of the original all along.
Result
The preliminary outputs show the result of this process: art pieces where the person is organically merged with the original painting, appearing as though they were part of the image in the first place.
Implementation and limitations
Not all combinations of painting styles and photos work equally well, and it would require some tweaking and further experimentation to ensure good results, but the outputs are pretty convincing overall. So far it is just early explorations and ideas, and while the Deep Painterly Harmonization model currently requires a lot of time consuming calculations on powerful computer processors, perhaps future museum guests will one day be able to see themselves in classic paintings at the click of a button?
About the author
Andreas Refsgaard is an artist and creative coder based in Copenhagen.
He uses algorithms and machine learning to make unconventional connections between inputs and outputs. Find out more about Andreas at his website or follow him on Twitter.
