From computer bits to human creativity (and back)
On April 5, 2016, at the Looiersgracht 60 Galerie of Amsterdam, dozens of people and television crews crowded an auditorium to assist at the unveiling of “The Next Rembrandt”, the culminating event of much anticipation and publicity over the previous weeks. The unveiling immediately became a worldwide trending topic on Twitter, with thousands of users exchanging tweets on the event.
On the outside, the unveiled object at the center of all this attention looked like a standard Rembrandt painting. In the precision of its execution, in the way light and color played over its visage, it was a work unmistakably amenable to the Dutch master. Even the most unexperienced person in the audience instinctively recognized that unique painting style.
Still, something was amiss. In fact, the painting was not a real Rembrandt at all, or even a skillful copy. It was a piece of art that was fully computer-generated, the result of an ambitious 18-month project carried by ING Bank, J. Walter Thompson Amsterdam, and Microsoft, with advisors coming from TU Delft, The Mauritshuis and the Museum Het Rembrandthuis. The engineers from this vast team embarked on a colossal project for producing the painting, first analyzing in amazing detail 346 known works from the artist, and then generating a 3D image of uncanny resolution evoking the master in all details, from the choice of the subject to the slightest artistic touch. The result was impressive: after centuries, the spirit of the old genius (or, at the very least, a very significant avatar) lived again in the auditorium.
With him, a question that has haunted artificial intelligence researchers for decades lingered over the audience: is there a role for computers in art? Even more controversial in the era of machine learning, can we teach a computer how to be an artist?
This feature article is part of a series we are writing to promote the 3rd edition of the Data Driven Innovation summit, the Italian conference on data and its applications. If you are interested, come join us for a two-days immersive experience in data-driven technologies — its’s free!
Human, machines, and everything in the middle
As with all things concerning the boundary between humans and machines, the project gathered an incredible feedback over the following weeks, sparking thousands of articles and videos worldwide and earning the companies millions in media revenue. In the meantime, it reignited public interest on computer-generated art and its artistic value.
For many people, machines will never exhibit creativity, by definition. “A computer can only do what we program it to do” is in fact a very old maxim, something that can be traced back to the XIX century mathematician Ada Lovelace, daughter of Lord Byron. For others, like the British-born computer scientist Simon Colton, it is instead only a matter of complexity. According to him, machines’ output could, in principle, become so complex and impossible to fathom for a single person, to be worthy of the adjective “creative”. Colton, who is a full professor at the University of London, is considered one of the pioneers of the field of computational creativity, a broad scientific endeavor with the aims of pushing the boundaries of artificial imagination as far as possible.
While this is a debate worthy of discussion, it is not what I want to focus on here. This article is instead about something slightly different. In fact, I do not even want to argue whether a machine can be considered an artist, or whether this is a privilege that will be forever a human prerogative. I want us to consider a trickier question: can machines and humans together become better artists?
For me, what the audience present at Amsterdam that day witnessed was not about artificial intelligence at all. It was about the incredible sleeping potential that data and computing power have to ignite and revolutionize the art world, with teams of creatives and engineers working together to unlock imagination in ways that were undreamt of before. It was, as we will argue in this article, machine learning finally transcending its computer science bounds and becoming a workable tool in the hands of makers. What we can rightfully call “data-driven creativity”.
Endless variations in a world of algorithms
Of course, computer-generated art is not a new concept. Machines have helped us create things since they came into existence (so much so that the Victoria & Albert Museum has an entire collection devoted to the topic). As a famous example, Harold Cohen, a British artist who passed away in the same year the “new” Rembrandt was shown to the public, spent decades of his life building AARON, a powerful software-programmed machine able to generate realistic paintings.
While the Rembrandt was 3D-painted from a digital design, most of AARON’s output was obtained by executing real brushes on a canvas through sophisticated (mechanical) mechanisms. Started as an experiment in what constituted a drawing, the abilities of AARON increased over the years with the increase in ability of its creator to program it better and better. Many of its creations have been exposed in galleries and exhibitions around the world, most notably at the TATE museum and in San Francisco.
Much has been written on whether AARON can, and to what extent, be considered an artist. Again, this is not the point we want to make here: Harold Cohen itself has always been careful in being non-committal to this notion. At the same time, he liked to remark that AARON was (and always remained) a much better painter that he himself could have been. Neither of them were very good on their own, but together, the human and the machine created something remarkable. Machines can only do what we program them to do, sure, but what we can program them to do is not as simple as we like to think.
To stress this point, consider another famous creation in the world of computer-generated art, The Painting Fool. This computer software is the brainchild of another decade of work from Simon Colton, the computer scientist whom we already mentioned before. While being similar to AARON, The Painting Fool is also able to extract clues from its environment, most notably news articles, to generate its works, such as creating impressive war collages based on the analysis of news from the war in Afghanistan.
Both AARON and The Painting Fool show the amazing capabilities of computers to generate endless variations while playing around from a template. Nevertheless, the latter also shows something more, the main idea we want to discuss here: the ability of some computerized art to learn from data, and not just to follow a preprogrammed course of action. In a world where data literally floods us, adapting from it becomes the key to unlocking the potential of artificial creativity.
Interlude: machine learning and deep neural networks
Powering any data-driven technology, of course, are machine learning algorithms, which, as most of the things we discuss here, have a long and fascinating history. Computer scientists have dreamed of computers having the power to automatically extract knowledge from data since the turn of the century, dating back to cybernetics and the beginning of artificial intelligence.
I already discussed the rise in machine learning interest over the last few years in an article for the previous edition of the conference. Most of this interest has been driven by the arrival of “deep learning”, a broad term describing a set of algorithms (most commonly in the form of so-called “artificial neural networks”), that have allowed to tackle machine learning problems considered impossible up to that point. Deep learning has revolutionized many fields, attracting billions of dollars of investments from companies around the world: whenever you hear “artificial intelligence” in the news, chances are you are just watching a different application of deep learning techniques. Results range from near-human image recognition capabilities (that first showed the power of these techniques) to defeating world champions at Go.
Deep learning can process complex inputs that, up to that point, were considered the domain of human sensory processing and required complicated engineering algorithms to make them accessible to computers: most notably images, videos, unstructured text, and audio. These are all domains where the information is encoded in millions of tiny pieces of content (think of the pixels in an image), and their understanding requires the simultaneous processing of all of them. Deep neural networks achieve this aim by learning multi-layered representations that extract the relevant information in multiple processing steps.
What we see with projects like The Next Rembrandt is that deep learning technologies are quickly moving beyond their engineering bounds. The combination of deep neural networks and computational creativity is explosive: the possibility for creatives of using algorithms that can automatically process audio, videos, and texts opens up a combinatorial growth and sophistication in what is achievable by computational means. Furthermore, much of this benefits from the fact that most tech companies are now pushing for open sourcing their codes and their implementations, making them accessible to a much vaster audience.
Creativity in the era of deep learning
Let me describe two examples of the power of this combination (algorithms, openness, and creativity), both originating in Google research projects.
The first one is Inceptionism, also known as DeepDream. DeepDream was a technique that a team of researchers from Google developed with the aim of extracting useful knowledge about the mechanisms of deep neural networks, to understand better what it is that these networks learn from data. Since different parts of the network can in principle specialize in recognizing different concepts, the idea was to create artificial images that maximally “activate” one or more of these parts, to visualize what the networks had learnt.
To the surprise of the researchers, the result were sometimes mesmerizing images, with absurdly vivid colors mixed to impossible superposition of objects, that were described as “the dreams of neural networks” or (for the more comically inclined) “neural networks under LSD”.
The idea could have been a simple research curiosity, if not for the fact that the dreamy quality of the generated images was so curious that it attracted the attention of thousands of people around the world, who wanted to exploit this effect for much more than just understanding the neural networks. Thanks to the open source code, DeepDream was soon incorporated in hundreds of demos, applications, and projects, from VR interfaces to song music videos, making it one of the most popular manias of 2015.
While the temptation is to discard DeepDream as one of the many quirks of the Internet, you should not be so quick in dismissing it as a joke. It is a perfect example of what happens when you mix a state-of-the-art data-driven technique, make it accessible to the public, and stir it up with human imagination: the results can go much beyond what its creators could have believed possible.
If you want a more serious example of deep learning applied to art, take a look at Google Magenta, another research project from the US tech giant. The aim of Magenta is to combine the latest innovations in deep learning with the insights of musicians and the computing power of Google’s infrastructure to generate music of incredible sophistication. Just like DeepDream, all of the Magenta’s results are available to play around with, from demos to code.
Computer-powered creativity: the next frontiers
Just as deep learning gave new life to machine learning, it is revitalizing the idea of using computers and data to help us create things. Today, the idea that computer art is a niche for a selected group of people proficient in art, computers and math is not true anymore. Machine learning is seen as such a transformative technology that companies like Microsoft and Google are giving their best to make it “democratic” and accessible to everyone. Similarly, projects like machine learning for artists or the artists and machine intelligence residency program at Google were created precisely for making data-driven technologies more accessible to an artistic audience.
Are we just scratching the surface of what is feasible with such technologies? This is the last question I would like to explore in this brief overview. To this end, allow me to make two final remarks on ways that could vastly expand the power and reach of data-driven creativity in the future.
First, up to now we discussed algorithms that can learn from data stored on a disk, like images or audio. However, we humans are able to adapt to a much wider range of information, including human feedback and the actions of other people. Researchers have tried for a long time to include this kind of feedback inside data-driven algorithms, with techniques having names like multi-armed bandits, reinforcement learning, and interactive evolutionary computation.
Today, we are starting to see a fascinating mix of these methods with deep learning at work to slowly revolutionize entire fields such as advertising. Companies such as Netflix can now offer personalized advertising (e.g., in the form of artworks and covers), whose visuals are adapted for each subscriber based on their feedback: a fascinating form of data-driven “micro-art” that has wide implications going far beyond advertising.
Painters, makers, advertisers: can data-driven creativity also change other artistic fields, like design and architecture? In order to answer this question, let me go back to our description of deep learning: a technology for automatically generating hierarchies of representations from raw data. A fascinating insight of the last years has been that these representations are not only useful for the task at hand (e.g. recognizing objects in an image), but they can be exploited to manipulate concepts at higher levels of abstraction. One of the most famous examples of this is the neural artistic style transfer, a technique that allows mixing the “style” of a picture with the content of another one. This is obtained by combining the internal “style” and “content” representations learnt by a neural network.
Can we apply similar ideas to computational art and design? Imagine a sophisticated photo editor that, instead of allowing you to play around with pixels and patches in an image, has buttons to change and manipulate some abstract qualities of your photo, like its style, its “melancholiness”, and so on. This is far from a future technology: similar ideas are being actively developed under the proposed name of artificial intelligence augmentation. Once again, the problem is not in the technology itself, but in designing user interfaces and high-level abstractions allowing non-technical people to leverage their power.
The sleeping power of data-driven imagination
Not much of what we discussed here is new. Machine learning, computer art, and user interfaces have existed, in one form or the other, since we invented the first computers. However, the potential of mixing them together has always been relegated to a very specialized audience that wanted to explore the philosophical limits of computer creativity, or who had the technical capacity of developing highly sophisticated machines like AARON.
What we are witnessing today is a complete change, both in the scale of what is feasible, and in the number of people that have access to these technologies. Pushing for openness and democratization is allowing thousands of developers and artists to have access to the latest deep learning tools for their projects, which are quickly moving from including object recognition in Android applications to having generative data-driven technologies in art installations around the world, manipulating abstract concepts, and including personalized feedback from the viewers. This is affecting all fields were creativity is valued, from advertising to design and art. How much further computers and deep learning can change these fields will depend on how good we will be at making them accessible to everyone.
If you want to learn more about data-driven creativity and its far-reaching implications, come join us and discuss them at the third edition of the data driven innovation summit in Rome!
About the author
Simone Scardapane is a post-doc researcher in Rome, specializing in machine learning. He is also co-founder of the Italian Association for Machine Learning, a no-profit organization devoted to promoting machine learning in Italy.
Thanks to Sara Di Bartolomeo for feedback on a previous draft of this article.
