Quantifying and Visualizing “Deep Work”

One of the best books I read in 2016 is Cal Newport’s “Deep Work”. In his book Cal explains that technology and various social practices have eroded our capacity to work without distractions and that we need to find ways to spend more time doing what he calls “Deep Work”: long stretches of time of uninterrupted full-focus work.

It turns out that, having a classic ADHD-y kind of personality, I have been deeply aware of this problem for many years and I have, over time, implemented various ways to cope with it. One of these is the habit of keeping track of when, where, and for how long I manage to immerse myself in “deep work”.

For the end of 2016 I decided to do a more systematic analysis than usual and to share it with the world thinking that: (1) you may be interested in doing the same and (2) you may have ideas on how to improve the process.

I am going to to first explain how I collect data and what criteria I use to define work as “deep”, then I’ll show you, through a number of charts, how my deep work in 2016 looks like, and finally I’ll offer a few reflections on the process and how I plan to develop it further.

Keeping A “Deep Work” Log

I have been logging deep work since 2012 without interruptions. I have tried tens of different methods and tools. You know what I always end up using? Just a simple google docs spreadsheet (all apps I tried on my iPhone or Mac ended up either being too complex or inflexible, I also tend to forget to use them). Following the KISS (Keep It Simple Stupid!) philosophy I always try to reduce the technological burden and I bet only on technologies that are infinitely flexible and are going to be around in ten years.

Various friends, when I describe my methods want to know what information I log. Following KISS again I log only very simple information. Here are the fields:

• Date
• Start Time
• End Time
• Duration (automatically calculated from start and end)
• Project Name
• Activity Type (writing, reading, thinking, …)
• Location (where I am located)
• Notes

Here is how my spreadsheet looks like today (as you can see writing this post counts as a deep work slot!):

This is the spreadsheet I use to keep track of my “deep work” sessions.

I write the start-time when I am ready to start and annotate the end-time when I am done with a “deep work” session.

How do I distinguish between deep and non-deep work? Very easy. I have a simple rule: deep work is work I track. If I don’t track it, it’s not “deep work”. It may seem odd but it works for me. The thing that makes it work for me is that when I track “deep work” I have to consciously think: “this thing I am going to do now is going to be deep work”. In turn, this is a signal to my brain that I am going to shift gear and that it’s time to get rid of distractions. Intentionality (similar to what happens in meditation) plays a big role here: it feels like “tuning” yourself to a specific mode.

Visualizing “Deep Work”

Now that you know how I keep track of deep work, we are ready to look into how my 2016 looks like. Here is a sequence of visualizations I developed for this purpose.

“Deep Work” Peaks and Valleys: What Happened Over The Year?

Peaks and valleys of “Deep Work” in 2016. Many peaks are about writing proposals. Between March and April I had a big hiatus, why? I can’t recall having had any vacation!

There are lots of high peaks and low valleys in my 2016. As you can see the biggest deep work efforts are related to proposal writing. No, professors don’t do research, they write proposals to get funding so that their students can do the research (fun eh?).

Two things surprise me when I look at this picture. First: what did I do between March and April? I can’t recall having had any vacation there. I have actually been working really hard on papers we submitted to IEEE VIS’16. I suspect what happened there is that I was mostly supervising my students’ work and since I had to mentally jump from one project to another I never managed to work deeply. That is, I did a lot of work but not “deep work”.

Second: between May-July I worked my ass off writing an NSF CAREER grant proposal (this is NSF’s way to fund early career professors like me), which was later declined. While I do subjectively know I have done a crazy amount of work there, I was not aware of how it compared to the rest of the activities I have done in 2016.

Oh my gosh, it’s huge! This is one of the most surprising findings for me: realizing the work I have done on a given project is way way more than I had expected. It feels like I can now quantify time and effort in a much more tangible way because I can see it and compare it across projects. When I look at the graph above I can kind of guess this period of work counted for about 1/3 of what I have done in the whole year! Wow! Was it worth it? I don’t know … But this is for sure going to give me some sense of the “opportunity cost” of working on something like this in the future.

Deep Work “Rhythm”: When Do I Work “Deeply” During The Day?

My high productivity hours are between 9am-1pm and between 3pm-8pm.

After observing how my work distributed in volume over time throughout the year, it’s time to look at what time during the day I typically work. This is what you see in the chart above: number of “deep work” sessions by hour of the day. More precisely, this answers: “what time during the data do I start a deep work session?”.

Probably no big surprises here. This is my daily rhythm: I tend to work deeply most of the time between 9am-1pm, then there is a big cliff, followed by a wider but shallower window between 3pm-8pm.

It’s interesting and useful for me to observe that I can work more often in the morning but only for a limited small window of time (I indeed often feel very scattered in the morning). Whereas in the afternoon everything is much more “mellow” and I am able to work for longer stretches of time (but I feel more tired sooner).

“Deep Work” Depth: How Long Can I Work Deeply?

I typically manage to work deeply for 30min-60min before I take a break. Sometime I worked for more than 2 hrs in a row. Sometimes even 3 hours!

For how long can I work deeply before taking a break? The chart above shows the distribution for 2016, that is, how many sessions I had (y-axis) of a given duration (x-axis). It looks like I typically manage to work between 30min-60min, which is actually better than I expected. If you look at the far right end of the chart you can see that a few times I worked deeply for a very long time! It’s great to see a few deep “zen” work sessions happened this year. I hope I’ll manage to nail more of these in 2017.

Another interesting pattern in the chart is the long bar centered at 60min, preceded by a short one at 55min. I suspect this is due to having experimented with the Pomodoro Technique over summer. With this technique you commit to work for a predefined amount of time rather than just stopping when you feel like. I remember having set the time to 60min for a period of time and it seems to have worked quite well. This is something I want to keep in mind and experiment more in the new year!

“Deep Work” NYC Spots: Where Do I Work When I Work Deeply?

I don’t know if this is the same for you, but for me finding the perfect location to work deeply is extremely important. And since I live in NYC, there is definitely no lack of perfect spots to get some deep work done. So … Where did I work from in 2016?

In the chart above, I have plotted amount of work (circle size) over time (x-axis on a weekly basis) split by different locations (y-axis).

Three locations are always there. My defaults are: my office at NYU, home, and Bobst, the NYU library; which is a few steps away from where I live (note: the Null row is just for times where I did not record the location).

But if you look at the chart above more closely you can see some more interesting patterns.

In July I was working very hard on writing the proposal I mentioned earlier and I felt the really strong need to be outside as much as possible. Having to work so hard was painful enough, but I could just not bear the pain of spending most of my time without seeing the beautiful sky and sun New York had to offer. In fact the locations listed there, Freehold, Greecologies, Sasaki Garden, and Think Coffee are all places with either a garden with tables to get work done or easy access to a garden.

That row of grey bubbles shows times when I was traveling and had to get work done while being in a hotel. Do you know that terrible feeling when you would like to just go to the hotel’s swimming pool and relax but you still have to finish preparing the presentation you are supposed to give in a few hours?

At the bottom right of the chart you can see two new entries: TShop and Elk Run. These are two spots I discovered lately and I am deeply in love with. TShop is a proper Chinese tea shop serving amazing (and yes, expensive) oolong teas that never stop surprising me. And Elk Run is the place from where I am writing these words now. This is next to my kids’ school and it’s tiny, cozy, with a very nice view on the street, and amazing coffee and food. This is how it looks right now, while I am writing this post …

Elk Run. The coffee shop from where I am writing this and I am recently getting a lot of good “deep work” done.

Another interesting angle is to see whether my “deep work” habits change according to where I am located. Here is a chart similar to the one above but with a zoom-in on the top three locations I use:

As you can see when I am at NYU I get to work more between 1pm-3pm. But when I am in other places I tend to have a proper lunch break; which, for an Italian like me, is a really really good thing to do.

In the chart above you can also see that I never work at night from the library; honestly it’s just too depressing. And I never work from home between 7pm-9pm. This is when we are getting the kids ready to sleep, so there is absolutely now way for me to work at that time. Nice to see it appearing clearly in the graph!

Insights

What knowledge did I gain from this personal data analysis exercise? And how is this knowledge going to impact my decisions and behavior in the future?

• Efficiency may be more important than volume. When I look at big big peaks of work in my initial timeline I don’t feel like I have produced more or better work. I just worked more. This is something I still have to process mentally and decide what to do with it. In the future I want to better understand how I can produce a lot of output in a very small amount of work and possibly with limited stress. To this purpose, I am planning to track more objective performance metrics. Since most of my deep work is about writing I’ll experiment with tracking number of words written in each session. I am also thinking of tracking my mood more closely (iMoodJournal is a pretty good app I have been using for a while).
• Morning hours are sacred. I already knew that, but seeing it in these charts reinforces it even more: morning hours are the best to get work done. In the future I want to become even better at “protecting” these few hours I have in the morning and delay as much as possible all the low-impact tasks I have to deal with later in the day. Emails are an especially nasty offender, once I start I am just sucked in indefinitely.
• Preset 1-hour long time slots may improve “deep work”. Throughout the year I have been experimenting with two main different strategies. In the first one, I would just start working, keep track of the necessary data, and stop when I felt like stopping. In the second one, the one advocated in the Pomodoro technique I mentioned, I would set a timer with a predefined time slot and work until it rings. I suspect this second strategy, which I used much less frequently, may lead to way better results. The high peak at 60min in the chart above is a good sign this may actually work very well.

Thoughts on Data Analysis

What did I learn by collecting, analyzing and presenting the results of my “deep work” experiment? Since my work is mostly about data visualization and analysis I want to share with you a few reflections on what this exercise taught me.

1. “Low-quality” data can lead to useful insights. My “deep work” data is highly imperfect. It’s very subjective and incredibly noisy. Sometime I do work hard on something but I just forget to keep track of it. The definition of what counts and what does not count as “deep work” is highly arbitrary. The technology I use is incredibly rudimentary. And yet this did not prevent me to extract interesting conjectures from the data and generate useful questions for myself. I still believe that working with high quality data is important, but I am surprised how good low-quality data can be if used appropriately from a person who has enough knowledge of the process to fill the gaps.
2. Story building leads to better data analysis, which leads to better story building, etc. One unexpected lesson I learned developing this article is how much better my analysis of the data became when I decided to turn it into a story to share with others. Before this exercise I had this mental model that one does data analysis and then creates a data presentation out of it. But things are much more intertwined! When you think of what the message for your audience is and you process information in more depth, you come up with more and better questions. And these additional questions in turn lead to better presentations, and so on … This is an important lesson I want to keep in mind in the future.
3. Domain knowledge is essential to effective data analysis. I briefly mentioned this above but it’s worth repeating. Domain knowledge counts a lot in data analysis and we don’t talk enough about it. When we analyze data part of the generated knowledge comes from the data, but a good chunk of it comes from our internal knowledge structures and the way they are constantly morphed by our actions. This is the big advantage of working on personal data: you have a lot of knowledge on how the data are generated, what they mean, and how they relate to the reality they describe. A consequence of this observation is that when data describe phenomena we do not experience directly, we have to be extra careful with what we do with them. It’s always a good idea to: a) study the phenomena in much more depth and b) pair up with people who have the necessary background knowledge.

What’s next?

Working on this small project during this Winter break gave me a lot of joy and food for thoughts. From here there are tons of other projects that can be developed. Personal data analysis and visualization have endless possibilities.

Getting to better and richer data is the real bottleneck. There are many aspects of my mental activities I’d like to capture, but the biggest challenge resides in capturing them in a way that is sustainable and not a big burden on my daily life. I am now thinking how I can improve this part of the process, especially how to keep track of useful performance metrics to correlate with just amount of work.

I am also thinking of developing some personal apps to improve the way I capture this information. Maybe it’s time to go beyond the spreadsheet?

Finally, I want to work on more personal visualization projects. When you start thinking about it there are endless ways to mine personal data! Emails? Keystrokes? Browsing logs? Etc.

Update Jan 15, 2017: Word Counting is Awesome

Following up on my intents, I am back to working hard and started tracking the number of words I manage to type in a given deep work session. This already opened to me a world of possibilities. Such a simple metrics is giving me a much clearer picture of what it means to write and how it is related to efficiency and productivity. I can now calculate a simple ratio: amount of time over number of words. Yes, it’s very simple but also very powerful. I am very much looking forward to seeing how this is going to play out in the future. For now, it feels great!

Special Thanks

This post and the work described in it has been inspired by the work of many people.

Special thanks go to my friend Giorgia Lupi who created with Stefanie Posavec the insanely awesome self-tracking project Dear Data. Giorgia kindly agreed on reading a draft of this post and did not shy away from telling me what did not work.

The work of Nick Felton and his annual lifelogging reports have been a source of inspiration for many years. Lisa Rost with her Google Search History visualization project, instilled in me the idea that I could do something similar for real. And the work of Matt Daniels and his Polygraph, which just never stops to blow my mind, showed me how to write a story out of data analysis in a format that mixes text with charts.

Needless to say, I am very far from reaching the depth and quality of these amazing individuals.