Runway
Published in

Runway

Machine Learning En Plein Air: Building accessible tools for artists

A short story in two parts.

Part One: Painting outside

Until the mid-seventeenth century, at least in France, painting techniques and procedures were developed and taught in an almost cult-like manner, as methods were passed down in secret through generations of elite artists. Groups of masters essentially managed access to the craft, and taught their apprentices with esoteric writings, some of them actually called “Books of Secrets”¹, which were available only to the chosen few. One of these rare texts was “Les secrets de reverend Alexis Piemontois.” It contained guidance on various topics, including the crucial instructions for how a painter should prepare their pigments.

The secrets of Reverend Alexis Piemontois — (Paris, 1557)
John Goffe Rand collapsible paint tube patent drawings.
Claude Monet Painting by the Edge of a Wood (1885) by John Singer Sargent. Oil on canvas. 54.0 × 64.8 cm. Tate Gallery, London.

Modern Pigment Powders

Artists before the invention of the collapsible paint tube were experimenting with novel ways of creating paintings beyond the confines of studios. Fast forward 150 years, we now find new generations of artists trying to experiment with novel ways of using digital technology in their work. I like to think of these current attempts as the outdoor paintings of the XXI century. But just as their colleagues in the 1800s had trouble making and using pigments before collapsible paint tubes were invented without access to extensive and exclusive training, artists today are having trouble fully engaging with emerging technical fields and incorporating the latest tools and technologies in their work because few tools have been designed with them in mind.

The secrets of Reverend Machine Learning
What if more people could have access to research like DensePose?

Part Two: Building the right tools

I first approached machine learning because I wanted to build creative, weird and unexpected projects with this technology. That was one of the reasons I decided to come to NYU’s Interactive Telecommunications Program (ITP) almost two years ago. My first project using machine learning was for a class I took with Dan Shiffman. It was an app that narrated stories based on a combination of pictures using a pre-trained neural network that was able to generate captions. I then took that same model and, as part of another class with Sam Lavigne, made a tool to generate similar semantic scenes from a pair of videos. You input a video and get a scene that has a similar meaning in another video. I was fascinated with what just one machine learning model could achieve.

Scenescoop: A tool to describe and found scenes in videos [Cristobal Valenzuela, 2017] https://github.com/cvalenzuela/scenescoop
Uncanny Rd: Using GANs to synthesize a never-ending road [Anastasis Germanidis and Cristobal Valenzuela — 2017, http://uncannyroad.com/]

But first: Models.

For the most part, machine learning consists of a series of steps designed to generate models out of data points. A model is a machine-learned representation of the input data. Thus, a model created to recognize faces will first need to be trained on datasets of many different types of faces. Once trained, it should be able to recognize faces from pictures it has never seen before. This model is a highly abstract representation of the input data that is now able to recognize faces. In reality the process is a bit more complex. It involves collecting a large number of data points, creating training, validation and test data sets, choosing the right architecture, selecting the right hyper-parameters and then training your model using, if possible, a graphic processing unit (GPU) for faster results.

Progressive Growing of GANs for Improved Quality, Stability, and Variation: http://research.nvidia.com/publication/2017-10_Progressive-Growing-of
Deep Photo Style Transfer: The first column is the input, the second is the style and the third one in the generated output. — github.com/luanfujun/deep-photo-styletransfer
Basic Usage Instructions for Deep Photo Style Transfer — github.com/luanfujun/deep-photo-styletransfer

Runway

My ITP thesis project, Runway, is an invitation to explore these ideas. If models are the building blocks of machine learning, how can we create simpler tools to access them? What should be the requirements to run and use models like Deep Photo Style Transfer? If we could have modern zinc paint tubes for digital artists, what would they look like?

Runway has 3 main components: Inputs, Models and Outputs.
Runway’s integration with other software
Gene Kogan and Andreas Refsgaard using Runway to teach at the Copenhagen Institute of Interaction Design Summer School in Costa Rica

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Cristóbal Valenzuela

Building creative tools for machine learning and AI | Author of @runwayml | Research Resident @ITP_NYU - @ml5js | cvalenzuelab.com