The sun/moon calendar 2006–2017, by Lê Trọng Lục on Wikipedia

My Quantified Email Self Experiment:A failure

I have an archive of my own email going back 18 years, containing 450,000 messages. One day I decided to make it searchable. Not half-searchable but fully, dynamically, programmably searchable.

My big idea was: If I can quickly look through all of my old emails I will be able to observe how my thoughts have evolved. I’ll learn something fundamental about myself and how I’ve grown as a person —for example, the difference between being in my early 20s and being 40.

This seemed like an interesting thing to do, so I did it. But the experiment was a failure, and not very edifying.

Email v. Giant Corporations

First, I had to solve some technical problems. I use Gmail, and I pay for Gmail, but Google’s email search function is very specific. It will help you find a recent email from a recent person. It can find a needle in a haystack, but I wanted to jump in the haystack. I wanted something fast and fluid that can could pull up tens of thousands of emails in an instant. Here is a screenshot of Gmail’s search interface:

Given that I use a Macintosh computer, I could have used Apple Mail to download all my mail, and availed myself of Apple’s built-in “Spotlight” search. But — well — look.

In 1996 you could type ⌘-F then type the name of a file, and your Mac would find it in a few seconds; today, when you do that, 5,000 files that have nothing to do with you appear in a window, the result of full-text search, while your computer grinds to a halt. Repeat the search and…nothing appears. My relationship with MacOSX Spotlight is ten years old and is one of the more difficult relationships of my life. I don’t understand it. Clearly the problem is with me and I need to move on.

In conclusion, neither a $375 billion search company, Google, nor a $700 billion software and technology company, Apple, can triage my email. Which is fine. Target doesn’t have pants in my size, either, and I still go there for detergent.

Email v. Free software

Email is a problem that was created by free, open software, so, I reasoned, maybe that’s what will solve it, too. This turned out to be entirely true. I used a tool called offlineimap to download all of my Gmail. That took a few days. Then I needed to actually search this mail. There are a few options to do this, but the two that I like are called mairix, and mu. I’ve used mairix in the past, and it’s excellent, but mu has more options for listing and displaying email, so I chose that. The way mu works is you type:

mu find waffles

and it makes you a special mailbox with all the emails that have the word “waffles” in them. In my case, 99 emails.

Email v. Myself

So now I had a laboratory for exploring my own past. I began to interrogate my corpus. I learned that I’ve sent 82,865 emails in the last 18 years, an average of 4,600 per year. This is too many emails.

Then I started in searching for specific words, looking for emotional and intellectual pathways. For example, when I started this project, I’d just finished an essay on manners and politeness. So I went back looking for what I’d thought about manners and politeness. I typed:

mu find from:ford@ftrain.com polite

Which found 196 emails that I’d written, each with that word somewhere in the email (including in the quoted part). I read through them.

I have emailed roughly the same things about manners and politeness for 18 years. “I try hard to be polite,” I might have written, or “politeness is important to me,” or “I tried to be very polite and respectful when I met the radio people.” My opinions, core beliefs, assumptions, and manners haven’t changed.

Okay, that’s not interesting. But the web sure has changed in 20 years. And I’ve learned a lot about programming. I’ve learned how to build content management systems. My perspective on technology must have changed, yes? So I went looking for how my understanding of the web has grown. And I quickly found, by searching for “HTML,” that — entirely forgotten by me — I wrote a blogging tool for my friends in 1999. It didn’t have a name.

Let me know what you think of the web tool, and any fantasies you’ve had about using such tools so that I could perhaps implement. (I need to come up with a kick-ass default journal design to make the thing actually work, I know.)

I could have written that yesterday. I’ve learned a ton more about programming and databases; I’ve spent time getting the basics of computer science; and it’s all to just keep doing the same damn things over and over again, and then forgetting I did them, and repeating them. Like a version of Groundhog Day about making Groundhog Day. I kept paging through emails and I’ve had nearly twenty years of conversations about:

• Blogging;
• Content management;
• Writing;
• The future of magazines;
• The nature of technology.

And I’ve had nearly twenty years of fights about:

• Politics;
• Race;
• Identity;
• Gender;
• Sexuality.

(Ignore that this is an acronym for PRIGS.) Some years I’m the person telling other people how to behave; some years, they’re telling me. (When I do it, it’s telling; when they do it’s yelling.) The content of the fights hasn’t changed much.

I guess I need something new to get angry about, but what?

Before this experiment, I would have told you that I used to be very passive and conflict-resistant, and that it took a long time to get my back up — but now I’m much more willing to stand up for my ideas. But no, that’s entirely wrong, too. According to my archive I was constantly in some fight or another over email. I apparently have three inches of plate in my skull. And in fact, because I believed, and have believed for so long, that I once was passive but am no longer, I think I tend to be even more likely to be passive-aggressively aggrieved than the typical person.

My whole sense of progress is getting really messed up.

A few months ago a minister wrote me something like, I liked your essay on politeness but it had a lot about prostitutes so I couldn’t share it with teens. So I went back to the book of Luke in the Bible and re-read the story of Jesus visiting the house of Simon the Pharisee when Mary, a sinful woman, came in and washed Christ’s feet with her tears, and dried his feet with her hair, and was forgiven. This probably took place in Capernaum, or Nain. But the story was so familiar. After 2,000 years we still see each other this way: the sinful, shameful woman; the suspicious Pharisee.

I can tell you that the word “hey” appears a thousand times in the subject lines of my emails, and six thousand times in the bodies. I can, with a tiny command-line script, tell you the numbers of times “coffee” shows up in emails I’ve sent, with one “*” equal to 10 “coffee”s:

$ mu find from:ford@ftrain.com coffee|perl -ne '/(\d{4})/; print "$1\n";'|sort|uniq -c|perl -ne '/(\d+) (\d{4})/; print "$2 " . ("*" x ($1/10)) . "\n"'
2000 *
2001 ***
2002 *****
2003 *******
2004
2005
2006 *
2007
2008
2009 *
2010 ****************
2011 ********************
2012 ***********************************
2013 **************************************
2014 ***********************
2015 ********

There’s a kind of story here. Prior to 2005 I was dating and freelancing, and having lots of coffee. Then I took a job as an editor and met my now-wife — almost no coffee for five years. Then I quit my job in 2010 and have basically worked for myself since, so — endless coffee. Fun facts! But also, so what? It’s not really knowledge. It doesn’t reflect some inner change, just consumption patterns.

There’s also the fact that I spent a lot of those years yearning for change.

Of course I’m not the only person emailing. There are hundreds of people in this corpus. I still know many of them. It’s strange to see the conversations because we’re all still obsessed over the same things we were ten or fifteen years ago. We’ve gotten older, gotten married and divorced. Some of us are rich, some are poor, some like comic books, some are writing poems, some are writing novels, some are still wearing the same T-shirts. Children change us, and keep changing us. Divorce changes us, often for a while. We cling to life and resolve to do better and then just drift back to ourselves and the regular flow of life. Like a pile of rocks in a stream, time running around us. Occasionally it rains and a stone is knocked around. Change comes from without.

So that’s what I learned. That’s why the experiment was a failure. This is the era of the quantified self and radical transformation. And I’ve made charts and counted and poked around. I can tell you the top 20 words for each of my years, the number of times I wrote about weight loss, the first time I started thinking about being a father. My basic self is just this single, continuous, thread — quantifiable, in the form of actuarial tables, bank account statements, square footage owned, number of children. But counting things doesn’t change them.

In the process of writing this article I came across one email from 11 years ago. It turns out I pitched this exact piece— the one you are reading right now — as a commentary to NPR, under the title “Mining My Own Data”:

“I have email and file archives going back 8 years,” I wrote in 2003, “and I’ve long wanted to write a program that would go through all that email and…”

You know the rest.

They passed.

“This one,” I told them, “would take a little while to get right.”