The thrill of releasing a side project
At a conference I attended some years ago, someone stood up after a session and made a point about that we all have a lot of ideas, that most of them are bad, and that it’s a good thing. We needed those bad ideas to reach the good ones. As a student with decent skills, but a particular talent for bad ideas, that little speech gave me a lot of encouragement. I think it has helped shape the professional I am today.
Two years ago i started developing a tool. The idea was vague, but with a goal to make investigating large collections of documents easier. It was supposed to unpack, extract, organise and present all relevant information with ease and speed. If the tool helped analysts and investigators catch criminals, it was working according to plan.
The development has however not been a straight line, and the retrospective meetings on a one-man scrum-team is not very productive. I’ve started over several times, I’ve built a magnitude of user interfaces, and the project has had a bunch of different names. One benefit of this being a personal project, is that I on error don’t waste anything but my own time.
It’s still kind of difficult for me to explain simply only with words what this tool is and what it does. Therefore I decided to create an introduction video:
I spent a couple afternoons and an entire weekend creating the video, it took a lot more time than I had imagined:
- Creating the scenario and data
- Finding out what to say and what to do
- Recording the audio,
- Recording audio again,
- And again
This was three weeks ago, and I was still not really finished when Ben Halpern (@ThePracticalDev) announced that the weeks topic on #DevDicsuss on Twitter was “personal projects”. I realised I couldn’t let this opportunity pass, so I exported the video, drafted a tweet and set my alarm clock for 3AM (9PM EST).
The release
This Saturday I released the source code. My expectations for response was not very high, but I was very happy to finally have pushed the code. As expected, the immediate response was rather thin.
A good friend of mine recommended submitting the web page to the “Show” section of Hacker News. He also suggested I wait until Monday evening before submitting, as more people would be online to see it. I did not follow that advice, I decided to get the disappointment from not getting any attention over with, and submitted it right away at 9AM EST. About 30 minutes later the post had received enough votes to get on the front page, were it stayed for most of that day. By Sunday morning, the project was “trending” on Github. I also published the page on Reddit /r/programming, and I got a couple of votes, but nowhere near the amount of traffic I got from Hacker News.
The response overall has far surpassed what I hoped for. Besides the activity on Hacker news, Reddit and Github, I’ve received a bunch of emails (I will get back to you soon, I promise!), and even a couple of job offers. Still, the coolest thing is probably that people are coming up with other uses cases for the tool. That is awesome!
Now, however the wave of attention has passed, which Google Analytics makes crystal clear:
Why did the release go so well?
I don’t know. I posted another link on Hacker news on Monday, which got exactly 1 point (which is the minimum), so I guess you can say I’m back down on earth. I don’t even know if anyone is going to read what I’m writing now! Anyways, if you allow to speculate on what worked well:
- I created a video to easier explain what the tool is and what it does
- I created VirtualBox and Docker images so that it is easy to get started
- I had QA mechanisms like CI and code coverage in place from the start
- I have good friends and colleagues giving me support and advice
- I actually having a pretty solid product idea
- And most importantly: I had lot of good fortune
The road ahead
I guess that I’ve published 10–20% of all functionality that at some point have been part of the tool. I’ve had object detection in pictures using OpenCV, text translation with Moses, timestamp recognition in pure text, distributed processing of documents, monitoring of open sources, and more along the way. That is part of the idea: To easily adapt to new requirements, so that you easily can add things like video processing or password guessing if you need.
There are a couple of things I plan to include in the near future:
Improved graph interaction
Working with graphs can be messy, and I want to help make it more clean by introducing things like “clique-view” and background nodes. A clique is a set of nodes (email addresses, names, etc.) where all are connected to each other.
Standing queries
In investigations, new data may be added continuously, and not getting any hits on your search now, doesn’t mean you wont get any hits tomorrow.
It may therefore be valuable to have queries running in the background notifying the user when new hits are found. I also imagine that these
searches may be organised into hypotheses, making it easier to collaborate by sharing queries.
Improve and expand API
Getting Gransk to work seamlessly with other tools is important for it to be useful. I want to make it easy to add, remove and query documents using both a Python and a REST API.
Got any other suggestions? Get in touch!
