Narcissus — A personal life logging data platform

I’m writing this to do one of the more ill-advised things I can do with a personal project; ie: talk about it before I’ve started.

The Problem

We live increasingly instrumented and surveilled lives. Companies, governments, all organisations are collecting and sharing copious data about each of us. It’s an inevitable feature of this point in techno-economic history.

This upsets a lot of folks, who feel their privacy being invaded. But personally, that doesn’t really bother me at all.

What does grind my gears is, why are they having all the fun? Why can’t I have all that data about me, and play with it? I bet there are extraordinary insights in there. I suspect there’s a whole world of knowing myself, of wisdom, hidden in the growing datasets and in the feedback loop they’d enable if I could just get my grubby mitts on them.

So, how could I have my own personal surveillance system, pointed at myself?

Does this already exist?

tl;dr: Sort of, but not completely, and anyway I don’t care.

What I’m envisioning is a system that can take data about me from wherever (mobile devices, wearables, cloud-based systems, static datasets in files), and put it in a datastore of some kind where I can poke at it.

Some areas that seem related: Personal Data Services, and Life Streaming / Life Logging.

I would like to describe this as a Personal Data Service. Unfortunately that name is already taken, to describe stores of personally identifying, identity related or sensitive data. Passwords, social security & credit card numbers, medical records. That’s not what I’m looking to gather.

What I am looking to gather is time series data from anywhere and everywhere about me, that’s not particularly secret. Where I go, what’s happening to me, what I do.

That’s a lot more like a life stream or life log. The original idea of life streams is about publishing everything about yourself. That’s not what I’m doing here (although I’m not particularly concerned about this information being public, for me). Life logging might be closer; for instance devices like the Narrative Clip have the philosophy I’m looking for.

Evernote would actually be a decent backend for this kind of project. They’re all about collecting endless time series data about yourself.

But I want control of the backend of whatever’s doing the collection. This is for nerdy reasons; largely because server side data wrangling systems is what I do. I’m happy with a high level system away from the bare metal, but if I can’t put my own code in it then what fun, let alone use, is that?

(Note to Evernote: don’t be upset, I think there’s a high likelihood that I’ll be using you, and torturing your API, in parts of this project.)

Project part 2: Talking AppEngine

There’s another project I’ve been toying with, and that’s writing about Google AppEngine and Google’s Cloud Platform.

At Training Evidence Systems we use these technologies, and by that I mean I live and breathe them. I’ve spent the last few years getting pretty good at AppEngine and related technologies, and I’ve been meaning to get to posting more about the techniques I use and ways of solving hard problems on the platform.

I can’t use our commercial codebase as a source of direct examples, so I need something to use as a sample app. It’d need to be something that can usefully use the same range of technologies, and be in the same kind of problem space.

This project fits that bill, as well it might, being inspired by the same kind of thinking that produced the Training Evidence System; that you should gather as much data as possible about your self (or organisation) into a system which is competent to extract insight from the firehose.

Is this a good idea for a project?

Personal projects fail. Failing projects are a bit sad and draining. So I’d like to start a project with a good chance of success.

Here’s an article I wrote on this problem some years ago. What I do is evaluate a personal project idea using these three rules:

  • Does it interest me?
  • Am I good at it?
  • Will I use it myself?

Does it interest me?

Yup.

My work life sees me collecting and wrangling data for commercial clients. It sounds dry, but it’s fascinating work, full of hard, interesting problems.

Also, I’ve been watching all sorts of apps collect streams of data by and about me: gmail, google+ & facebook, evernote, google drive, etc. Plus I get tantalising hints that due to Australian data privacy laws, I may be able to request data sets from places like my telco and my ISP.

I’m really keen to try this stuff I’ve been doing professionally on myself. I want to see those graphs!

Am I good at it?

Yup.

I’m aligning this project with the techniques and technologies I use in my professional life.

As I said above, I want a project to use as a basis for talking about AppEngine. So I’ll be using AppEngine on this project, and writing articles about it as it progresses. I want a way to hone my professional techniques and have a way to talk about them publicly in code-detail, including being able to get feedback and reintroduce that back into my work.

Secondly, these tools are excellent for the job at hand; they’re what I’d choose if I didn’t have the goal above.

Lastly, I’m already skilled at these technologies. That is, I’m good at it.

Will I use it myself?

Yup.

I’m aiming at a system with a userbase of 1, me.

That said, if I actually get somewhere and others are interested, I’ll be keeping the idea of other users in mind.

Where to from here?

This article is a start!

I’m hoping to write something once a week (or close to). That wont all be about this project, but hopefully a good amount will be.

I’m thinking the next article will be about the first shot at an architecture and design for the MVP.

Next Article: Narcissus — initial designey, planny thing