Using Incubation to Unlock Your True Creative Potential
Why rest and leisure are crucial for innovative thinking
I had been struggling with an idea for months. I had spent lots of time reading and actively thinking about the problem. I knew exactly what the issues were and what I wanted to achieve. The only piece missing was how I could actually achieve it. Despite all the hard work that I put into it, a solution just wouldn’t present itself.
Then one weekend, I was hiking with my girlfriend in Hakone, a region in central Japan known for its hot springs and scenic views of Mt. Fuji (see the photo at the top). We spent most of the first day there soaking in a beautiful outdoor bath surrounded by mountains and nature, and on the second day we set out on a long hike.
My mind was (seemingly) as far removed from the problem as it could have been. I was fully present in our conversation and the hike, which was crucial in order not to trip over the many stones and tree roots on the steep and narrow paths. Not the slightest thought of work entered my (conscious) mind.
But at one point, our meandering conversation sparked a small offhand remark — and suddenly, it hit me. It was right there: the solution to my problem. The following Monday, I just had to verify it and work out the details. But the key insight came in that one moment through a chance comment during a hike.
I’m an AI researcher, and the problem I was struggling with had to do with how we represent data as abstract mathematical objects, so-called “embedding vectors”, in order to do things like similarity comparisons. In one form or another, this basic concept is behind many of the applications of AI that we encounter on a daily basis; for example, finding similar-looking images, the song or movie recommendations of streaming services, or Facebook’s photo tag suggestions.
In a somewhat simplified and handwaving way, we can think of representing each piece of data in a dataset. For example, we think of each image as a point on a map. The closer two points are, the more similar the data they represent.
The problem is that given our current algorithms, these maps essentially contain hidden “valleys” or “mountains.” However, the distances we use for comparisons are straight lines, as the crow flies. So while two points might look like they are close together, they might actually be separated by a huge invisible obstacle, and not be as similar as another point which seemingly is further away, but connected via a flat and easy-to-traverse surface. Due to this kind of issue, our current similarity comparisons (and other tasks) can return sub-optimal results.
I knew of a way to figure out where these valleys and mountains were. But the difficult part was how to incorporate this information into the distances. Essentially, I was looking for a way to distort the maps of data points in such a way that distances became more meaningful (or more precise and mathematical — I was trying to flatten the metric of the underlying space). This was the problem I had been struggling with for months and couldn’t make any progress on.
Until I went on a hike.
We had been out in the mountains for a few hours and were considering how to eventually get back to our hotel. So we consulted a bus map we had previously taken at the local tourist information.
The map was in a simple hand-drawn style. It was not a perfectly accurate to-scale representation of the real world, but was distorted in such a way as to highlight the most important sights and routes. At that point, my then-girlfriend said how she had recently been reading a book on the history of cartography, and something along the lines of, “isn’t it interesting that people used to draw ancient maps so that things that were considered more important were shown in a larger way?“
That’s exactly when it hit me. I immediately started thinking of modern maps that are transformed in such a way that the size of the area is scaled by some property of interest — like that region’s GDP, its population, or its agricultural output.
I realised that I could use exactly the same technique to solve my seemingly unrelated problem. If there was a “mountain ridge” in my data map, I could similarly blow up this area to make points separated by the ridge more distant. The more distance, the “higher” the mountain.
When I went back to work on Monday, I quickly figured out that these types of maps were called cartograms. I also found a scientific article that was published literally the week before on a new method for faster calculation of such cartograms.
From there it was quite straightforward. It took just a few days to test my intuition of applying the cartogram idea to these data representations, and it turned out that it worked fairly well.
After a few more experiments and some detailed analysis, two of my colleagues and I wrote a scientific paper on our new method. And all of it was based on that one sudden illuminating idea I had during the hike.
I had to get away from work in order to make progress with work.
On the Confusion of Busyness and Productivity
We currently live in a society where busyness, stress, and overwork are often worn as a badge of honour, showing how accomplished and important we are. When faced with a difficult problem, we think that we just have to grind it out. We assume that putting in more hours will lead to more and better results, equating busyness with productivity.
But the truth is very different from this! We actually end up sacrificing a lot of creativity and productivity (as well as life quality) as a result of our obsessions with the grind.
“Busyness is not a means to accomplishment, but an obstacle to it.”
— Alex Soojung-Kim Pang
Good work, particularly of the creative and innovative kind, needs rest and relaxation just as much as it needs time actively engaged in work. And effective rest is a skill that can — and should — be honed.
Creativity is about connecting dots. But if we try to force it, if we’re too focused on it, all we ever do is connect neighboring dots, resulting in a rigid grid of stale thoughts. To really see the interesting connections we need to get a new perspective and gain some distance. Then the truly interesting connections will reveal themselves.
The Importance of Idea Incubation
Graham Wallas was a founding member of the London School of Economics and served as its first Professor of Political Science. But maybe his most lasting contribution to humanity was his 1926 book “The Art of Thought”, in which he presented a theory of productivity and creativity as a four-stage process.
Despite being almost a century old, this text still has a strong influence on creativity researchers today and is widely cited in the academic literature.
I have previously written about these stages in another article.
In Praise of Deep Work, Full Disconnectivity and Deliberate Rest
Thoughts on improving productivity and creativity.
But here, I would like to revisit them and show how they directly apply to my own story, and how you can use them in your own creative process.
The four stages Wallas proposed were preparation, incubation, illumination, and verification.
When we start working on a new project or face a new problem, the first thing we have to do is actually sit down and do the hard work. We have to fully comprehend the problem, familiarize ourselves with its intricacies, research and understand relevant past problems, and plan a way to tackle it.
This is a highly focused stage. In the case of my problem, it was all the time spent actively thinking about data representations, reading related academic literature, and trying some ultimately unsuccessful approaches.
But in most cases, as in mine, it won’t get us to the desired solution. It is only Preparation.
Once we take a step back from the problem, either because it’s the end of the workday or the weekend or because we are so stuck that we can’t see a way to make progress and (temporarily) abandon the project, our subconscious mind kicks in and gets to work. This is the phase of Incubation.
While our conscious mind is at rest, the subconscious starts examining and analyzing the problem. But its approach is very different. Instead of being highly focused and directed, it makes loose associations between different concepts, as well as previous experiences. Often, we are not even aware that this is happening.
What feels like rest and leisure to us is actually a highly productive time, thanks to the power of our subconscious mind and incubation. This kind of productivity cannot replace the one we usually think of, the active work, but true creative innovation also can’t happen without it. The two forms are complementary.
Incubation doesn’t have to be a time of complete inactivity. It can also happen while we work on other problems. As Wallas remarks,
“Voluntary abstention from conscious thought on any problem may, itself, take two forms: the period of abstention may be spent either in conscious mental work on other problems, or in a relaxation from all conscious mental work. […] We can often get more results in the same way by beginning several problems in succession, and voluntarily leaving them unfinished while we turn to others, than by finishing our work on each problem at one sitting.”
This idea of using one type of work as time off from another has been taken up by several great minds past and present.
Existentialist philosopher Søren Kierkegaard recommended using the agricultural idea of “crop rotation” and applying it to mental tasks, essentially allowing our mental soil to recover from one problem by working on a different problem.
However, this is not permission for constant task switching or multitasking. At any time, we should be fully immersed in one thing only. Kierkegaard’s idea of mental crop rotation is also closely related to what economist Tim Harford called “slow-motion multitasking” in a recent TED talk:
“Slow-motion multitasking feels like a counterintuitive idea.[…] We’re in a hurry, we want to do everything at once. [But if] we were willing to slow multitasking down, we might find that it works quite brilliantly.”
— Tim Harford
Subconscious incubation happens whenever we are fully immersed in something other than the actual problem itself, whether that immersion is in high-quality leisure or deep work on another unrelated problem. But the key here is full immersion, not absent-minded distraction and incessant task switching.
If we make time for incubation and trust in the power of our subconscious mind, magic happens.
This magic is Illumination, the elusive moment of sudden inspiration or revelation. The stereotypical light bulb moment where our mind goes “Eureka!” and everything suddenly makes sense. It was my insight that I can use an existing method from cartography to solve my seemingly unrelated problem.
Illumination cannot and should not be hurried. Wallas warns about this in his book. If we feel like we are getting close to a solution but it’s not quite there yet, we should not force it, or the insight might disappear.
We need to give our subconscious mind the space to approach the problem from with a broad and unconstrained view.
Had I actually been consciously focused on the problem at the time of the comment, my mind might have been much too focused to make the distant connection between these maps and what I was trying to do.
Finally, it’s time to actively get back to work and engage in the last of the four stages: Verification. We have to confirm that our aha-moment actually did produce something useful, and if so, how exactly it can be used to solve our problem and progress our project.
“It never happens that unconscious work supplies ready-made the result of a lengthy calculation […] All that we can hope from these inspirations, which are the fruit of unconscious work, is to obtain points of departure for such calculations.”
— Graham Wallas
I still had to look into the details, find out that these maps I was thinking about were called cartograms, figure out how they work, and verify that the same method could actually be used in the way I imagined.
In most real situations, we encounter multiple cycles of different stages, and different stages may overlap. Wallas was fully aware of this.
“In the daily stream of thought, these four different stages constantly overlap each other as we explore different problems. […] Yet, […] the four stages of Preparation, Incubation, Illumination, and the Verification of the final result can generally be distinguished from each other.”
While fairly simplified, this four-stage model is a great framework for understanding the creative process. The model is still applied by creativity researchers today.
And it shows us that rest and leisure are just as important to work as the actual “work.”
“Often when one works at a hard question, nothing good is accomplished at the first attack. Then one takes a rest, longer or shorter, and sits down anew to the work. […] All of a sudden the decisive idea presents itself to the mind. It might be said that the conscious work has been more fruitful because it has been interrupted and the rest has given back to the mind its force and freshness.”
— Henri Poincare
So make your conscious work more fruitful and give your mind back its force and freshness, by embracing rest and allowing your ideas to incubate!
On running as a problem-solving tool
Before sharing with you my own approach, I would like to briefly mention a great example of a method that is complementary to incubation.
Terry Rudolph, a professor of Quantum Physics at Imperial College London (where he was my Ph.D. supervisor) and co-founder of quantum computing startup PsiQuantum, uses running as a very deliberate tool to work on some problems.
It’s not exactly the same as the Incubation phase mentioned above since he is actively thinking about work, but it is still a way for him to look at ideas and problems in a different way than if he was sitting at his desk.
Before going out on this kind of run, he will sometimes make a list of problems to think about before he sets out.
“In that case I run much slower. The advantage of this kind of run is that it gets you away from distractions like your phone and your computer, but also from notebooks. The majority of my work and calculations have to be done in a notebook. But mentally, if I write something down, then my brain feels like ‘okay, I don’t have to store that anymore, I can just reference back to it’. But there are some things for which it’s actually important to try and keep the whole thing in your head. […] On a run, you have no pen and paper, you can’t write it down, you can’t get lost in details. You just have to think what is the narrative, what is the story, what am I trying to accomplish? You have to focus on the bigger picture.”
[More great advice from Terry and others, and their approaches to work and rest, will appear in our forthcoming book: Time Off.]
Getting away from work is crucial to seeing things in a new light and to focus on the bigger picture.
Whether that’s the complete detachment and subconscious incubation you get from a hike or other forms of true time off, or the more “active” approach Terry uses on his runs to think about things without the mental crutches of notebooks and computers.
How to Find Your Own Style of Incubation
So, how do you actually go about creating periods of incubation in your own routine?
I think the answer to this question is actually quite personal, and there is no one-size-fits-all approach. This is true for most forms of time off. Some people find time off in solitude, others surrounded by friends. Some people like to schedule regular small breaks, others work hard for extended periods of time and then take a lengthy sabbatical. And so on.
So too with incubation — you have to find your own style.
The directed approach
Let’s first look at the active approach to Incubation, like Kierkegaard’s crop rotation. This tends to be a more scheduled approach to blocking out incubation periods.
At one end of the spectrum, this can start as a micro-habit, dedicating individual hours to individual tasks and then switching to another task. I would say though that one hour is the absolute minimum unit of time here. Remember, crop rotation is not permission for multitasking and permanent task switching. Each task, while we are actively engaged in it, should get our full and undivided attention.
On an intermediate time-scale, we might focus on individual projects for days or weeks at a time before switching to another project. The popular idea of Design Sprints somewhat falls into this category. The sprint acts as a focused form of preparation, which can then be followed by incubation once it is over (and we are maybe engaged in another sprint).
On the very macro scale, we can think of projects that span months or even years. Harford’s above mentioned slow-motion multitasking falls into this category. Many polymaths get fully absorbed in one project for extended periods of time, or they might somewhat loose steam (or simply get excited by a new interest) and throw themselves into a new project, but return to the original one after their subconscious has gained some new insights and motivation through Incubation.
There is also a limit to how many different projects we should be engaged in at any time. Again I think this is quite personal, my own ideal number is probably two or three, but it’s unlikely that juggling ten projects at a time is optimal for anyone.
The open approach
A similar spectrum of approaches exists for a more open approach to incubation. This depends on you noticing when you need incubation time, and then allowing for it.
On a micro level, maybe all we need when we feel mentally drained or stuck with a problem is to get away from it for an hour, for example by taking a walk in nature. Just detaching from the problem for a little bit can help us get unstuck once we return to it.
At an intermediate level, we have periods of time off that last anywhere from a single day to a week or two, like my hiking trip to Hakone. This allows us to get a bit more distance from the problem, and as a result, make more distant connections between ideas.
Finally, we can think of extended periods of rest — starting anywhere from several weeks upwards. Especially when combined with extended travel, which in itself is a great tool to get a new perspective and detach from our usual thought patterns, this can be a highly effective form of incubation and lead to very innovative ideas.
This is certainly not always true, but as a very loose rule of thumb, we can maybe think that the longer the time for Incubation, the grander the scale of insights it might deliver.
When I’m writing something, a short walk can suddenly bring a great sentence to mind, or a better way to structure an argument, but nothing ground-breaking. More profound insights might take some longer time away from the problem, like my cartogram revelation.
But there are certainly exceptions to this, and depending on our precise problem and where exactly we are in our preparation phase, even the shortest bit of Incubation can lead to a tremendous illumination.
There are also different approaches as to whether we deliberately schedule Incubation time into our calendars before even knowing that we are at the right stage of preparation, or we just do it on an as-needed basis. Again, I think the key here is to try different approaches for yourself, see what works, and ignore the rest.
The “crop rotation” approach
My own personal approach is a mix of all this.
If there are no meetings or other obligations that prevent this, I tend to split my day into two parts. Mornings are usually reserved for writing. My lunch break then serves as time off, before spending the afternoon on my actual job.
This scheduled cycle is my (more or less) fixed form of crop rotation, and I often get great sudden ideas for my writing while working on AI in the afternoon, or vice versa. And I think that as a result, I make more progress on each of the two than if I’d spend my full day on either of them.
In addition to that, I also just take breaks whenever I feel like a brute-force approach simply won’t get me anywhere. I might go for a walk, or take a nap, or go for a run or to the gym, or simply call it done for that day.
I think knowing when it’s best to step away from a problem and take time off — realising that more work would actually be counterproductive, and not feeling guilty as a result — is a key skill that needs to be gradually improved over time. Surprisingly, time off can often be a much more difficult choice than busyness. But it’s worth it.
Finally, on the larger scale, I might treat myself to a weekend trip, or short holiday, or an extended break. As just mentioned, knowing the importance of time off is only the first step. Really implementing it, fully trusting in its importance and ignoring the stigma that’s unfortunately associated with it, is yet a different matter and takes practice.
It’s both something we need to get better at on a personal level, but also something that needs to change from a societal point of view.
The Need to Rethink Our Rest Ethic
Rest and leisure are absolutely essential to creativity. We need to seriously rethink our glorification of busyness, and take time off seriously.
Our “rest ethic” is as important as our work ethic, if not more so.
We often give too much weight and credit to the active Preparation and Verification stages of the creative process and to downplay or ignore the importance of the passive Incubation and Illumination phases.
In 1948, philosopher Joseph Pieper spelled out this problem in his wonderful and still very timely book “Leisure — The Basis of Culture”:
“Exaggerated value […] is put upon the ‘difficult’ simply because it is difficult. […] Man seems to mistrust everything that is effortless; he can only enjoy, with a good conscience, what he has acquired with toil and trouble; he refuses to have anything as a gift.”
— Joseph Pieper
And this misperception has only gotten worse in recent years, especially in tech and startup culture.
Being constantly engaged in work, and trying to force solutions and attempting to compensate for lack of ideas with pure quantity of work, does not only make you miserable, it’s simply counterproductive.
Ironically, our zealous pursuit of productivity might in many cases be achieving the exact opposite effect we are aiming for.
Work — especially the creative kind most knowledge workers are paid to do — is not linear or additive. No matter how much time you put in, the breakthrough insight may not come.
So the next time you are stuck with a difficult problem, just walk away from it for a while (maybe literally). Let your ideas ferment and incubate. Trust in your subconscious mind’s power to quietly do a large chunk of the hard work for you while you enjoy some quality time off.
For incubation — and ultimately, illumination — to happen properly, we need rest, detachment, and a fresh perspective.