How I Stopped Fighting With My Girlfriend By Becoming A Cartoon, Using JavaScript and Computer Vision

First of all, to truly understand my story you need to know that I HATE photographs. Hate them. It’s not that I believe they steal your soul or something like that, I just don’t enjoy them — for me life is something that happens over time, and a single moment (as represented in the photo) is a dead moment.
Also, I don’t consider myself photogenic, and in this day and age it is far better to be photogenic than handsome.

But, I’m not living alone, and my girlfriend kept complaining that we don’t have any nice pictures on the walls of our living room. We had some posters and a painting, but she wanted to hang pictures of our friends and family and pictures of ourselves — which is pretty reasonable; I mean, it’s her home too, right?

So, I started thinking of a solution that would suit us both.
The answer seemed obvious — I’ll just hang a video instead of a photo; a live moment instead of a dead one!

So, I bought a reasonably cheap tablet and placed it in a lovely wooden frame (to give it that authentic picture-on-the-wall look). The frame had to be wide because it had to hide most of the electric cable. The guy at the frame shop loved the challenge and the result was perfect.

Now I can hang video art on the wall, question is “which video?”

My first idea was simple. she wanted pictures of friends and family, right?
I installed nodejs on the tablet and used OpenCV to work the tablet’s camera. I hung the tablet on the wall near our front door and wrote a simple JS app that activates the camera and apply the following logic:

1. If movement is detected in camera, display the camera feed (to create a mirror effect).
2. if a face is detected on camera, record video for 4 seconds.
3. if no movement was not detected in the last 2 minutes play a recorded video.

The result was amazing, my friends would activate the mirror mode without even realizing it, then they would stop for a second and look at themselves (as we all do when we see a mirror), at that point OpenCV (who did a reasonable job of detecting faces) would record them looking at themselves and pretty soon the tablet had 30–40 short videos of our friends and family, I added a simple web interface to delete bad videos and I was pretty happy with it, I called it “Magic Mirror” it was fun and random and I loved looking at it.

My girlfriend, however, was not pleased.
“It’s like living with ghosts” she said “I don’t like it, and I don’t like how I look in those videos, this isn’t life, this is dumb”.
Well, that happens sometimes, but the client is always right and since she’s the client I had to go back to the drawing board and design a different solution.

My vision for my next solution was life, I wanted to capture our mutual life, but the problem was that any video I’ll display will be only a short piece of life, right?
well, not necessarily.

First I removed the tablet from it’s place and found a different place for it, right in the middle of our living room. than I bought a reasonable cheap hd web cam and hung it so it will get a good shot of our living room, mainly our big couch.
Now all I needed was to change the code, the new code was even simpler than the last (logic wise), all it did was record from the webcam and play the recording with 1 hour delay, actually I didn’t even need code, FFMPEG was perfect for the job.
Now I had a tablet in my living room displaying the living room as it was 1 hour ago, I had my life as a living room art, I called it “Timeframe”.

And I LOVED IT, I really did. some days I would wake up, go to the living room and watch her wake up before me, pet the cats, drink her coffee, get ready and leave. I liked watching us talk, I liked watching her smile from outside the conversation, I liked watching her laugh at my jokes, I even liked having sex on the living room couch, but we had to make sure nobody is coming over the following hour, for obvious reasons.

She, on the other hand…

Well, as you can probably guess, she was not thrilled, not even a bit, mostly she found it creepy and weird, “I don’t like being so aware of myself” she said “I don’t like how I look in the video, this feels like peeping on myself”.

So.. back to the drawing board, right?
Wrong.
See, I really liked my Timeframe, I wasn’t ready to give up.
I just need to change it, just a bit.

The problem was, from my point of view, that the video felt too real, she couldn’t enjoy it as art because of it, so I tried making it feel less real.
the first few attempts mostly revolved around playing with the fps, I tried giving it a dreamy feel, or a movie feel, but that didn’t do the trick.

But that got me thinking, Instagram (and Snapt) use filters to give the images a playful, silly and unreal feeling, and that seem to lower the self-awareness of the user, maybe I can use some sort of filter on my video, so I started trying that.
The first problem here was that I couldn’t use really cool stuff, FFMPEG wasn’t the right tool for the job here, so I went back to JS, this time I needed 3 simple scripts:

1. a script that will take a snapshot from the camera for a given FPS
2. a script that will apply a filter to the saved images
3. a script that will play filtered images according to a given FPS with an hour delay.

The first and the third were easy enough, so I took care of them first, the filtering script was a bit of a riddle, see I couldn’t use a heavy filter, because the runtime needs to be very very short to emulate an actual video, so I had to use something that will be CPU-sensitive (this is a cheap tablet, no GPU), memory-sensitive (again, cheap) and most important of all, pretty.

Well, it took some time, I won’t lie, but after a couple of really bad attempts I managed to create a cartoon effect that was fast enough and kinda nice, It had a comic book kind of feel, so I added as small rectangle in the corner and wrote some text in it, the process looked like this:
img
.negative()
.despeckle()
.edge(1.5)
.negative()

and the result looked like this:

(not the best photo of me, and the living room is messy, but how cool is the effect, right?)

And now for the most surprising part of the story, that actually worked.

I know, I didn’t believe it either, but she LOVED it! the self-awareness was gone, out the window, now she couldn’t wait for people to come over just to see how they look as cartoons and my Timeframe still hangs in my living room, happily ever after.