Lessons learned from my 10 year open source project

For the past ten years, I’ve been building an Open Source Intelligence (OSINT) tool called SpiderFoot, which is used to automate the collection and analysis of OSINT for attack surface management and threat intelligence. It’s open source, written in Python and is the top ranked project in many of the GitHub categories it’s in, with just short of 7,000 stars at the time of writing. A couple of years ago I also launched a SaaS version of SpiderFoot called SpiderFoot HX, but that’s another blog post (or series of posts!) for another day. This post covers what I’ve learned from that journey so far.

You’d rightly imagine that after ten years, SpiderFoot has become a pretty big part of my life. I spend most early mornings and late evenings “working” on it and I often talk about it with my wife, kids and friends. It’s brought me a lot of joy but also frustration at times.

So now feels like a good point in time to take a step back, reflect and try to draw out the lessons I’ve learned and internalized over these years. I’m writing this in part for my own benefit, but also in the hope it will help others who might be on a similar journey with their own open source project, or may be considering starting one but have been holding themselves back.

Let me first walk you through my journey to give you some context, but feel free to skip this next part and go straight to the following sections if you just want to get to the lessons learned.

The journey so far

The funny thing is, I actually wrote an initial implementation of SpiderFoot in 2005 as a way to learn C#. After I released the first version, it remained sitting on SourceForge (yes, remember that?) at “version 0.1b” for seven years until I started re-writing it in Python, late 2011. So if you want to include that time frame, SpiderFoot (at least as a name and concept) is getting closer to twenty years old!

Here’s what that C# version looked like. Yes, my design skills were amazing:

Here is SpiderFoot in the present day, a web-based and CLI tool written in Python:

So why the seven year abyss in between the first version and the Python re-write?

Well, my initial goal was to learn some C#, build a useful tool and get it out there, which I did. I didn’t have a long term picture in mind or ideas for expansion, it was just “done”, and that was OK for me at the time. I was also doing a bunch of interesting engineering projects at work back then, so didn’t feel much of a need to develop anything in my spare time either.

What I didn’t realise over those years was that for such a small one-shot tool, it was actually quite popular. SpiderFoot was covered in books and training courses, and downloads remained at around 500 per week — not huge, but not insignificant — despite the fact that parts of the tool no longer worked due to data sources disappearing.

After being in various management roles for a few years in my day job, I recall eventually feeling that something was missing. Looking back on it now, I missed learning through creating, which I had been doing for quite some time during my day job until all of my working hours became devoured by meetings, PowerPoint and email. It took me a long time to realise what was missing: a creative outlet unencumbered by company policy, technical standards, methodology requirements, the list goes on (I worked at a bank, need I say more?)

That got me thinking about what I could build in my spare time. I knew that I wanted to learn Python, do something involving the collection and analysis of data, and I wanted it to be somewhere within the context of Security — a field I’ve been working in for over 20 years.

SpiderFoot still ticked all these boxes! So in late 2011, I went back to the drawing board, learnt enough Python to get started and proceeded to push commit after commit to the shiny new GitHub repo until I released “version 2.0” in May, 2013.

That’s right, it took about a year of commits until I felt comfortable enough to cut a release and announce it to the world. That’s one lesson learned.

In the ten years since those initial commits, I’ve stumbled my way through learning and slowly gaining proficiency with Python, built up SpiderFoot from a few modules — SpiderFoot lingo for components that collect or analyse OSINT data — to over two hundred, incorporating many significant contributions from the community (especially _bcoles) and eventually getting to the point where I was able to also build a successful business around it.

To boil it down, in no particular order here are some of the most notable experiences I’ve had while working on SpiderFoot that have led to the lessons learned covered in this post:

• Receiving numerous random emails of thanks from users, from enterprises that have used it to find gaps in their attack surface or boutique security consultancies that used it to win new clients. Even from investigators have used it to help track down missing persons!
• Seeing it covered in books, magazines, training courses and conferences
• Seeing members of the community create tutorial videos and blog posts about it
• Landing in the top of HackerNews, and I wasn’t even the submitter
• Learning new tools and technologies
• Working with people in the community on SpiderFoot or helping them use SpiderFoot for their projects
• Building a business around SpiderFoot, alongside being full-time employed
• Dealing with intellectual property rights with three different employers
• Getting approached by potential investors and even potential acquirers
• Giving various talks about SpiderFoot
• Using the funds earned from SpiderFoot HX to hire others to contribute to the open source version and even sponsoring community efforts like OSINTCurious, abuse.ch and TraceLabs
• Writing associated blog posts like this and others
• Meeting other founders/CEOs, being invited to join various communities
• Having it (mis)used to claim election fraud in the 2020 US presidential election
• Seeing someone else create a business around SpiderFoot, eventually shutting it down
• Generally figuring out how to balance time with family, work, friends, etc. (definitely a work in progress!)

All of these experiences and more have shaped the lessons learned below.

OK, enough navel-gazing, let’s get onto the part you clicked on a link somewhere for…

Lesson 1: Writing open source software can be very rewarding in ways you can’t predict

Writing open source software is not just about putting code out there for others to use. Sure, that’s fundamentally the process behind it, and yes the people using that software are getting something (hopefully useful) for free, but after so much time working on SpiderFoot, it’s still hard for me to truly grasp all the benefits I alone have reaped from working on it. These come to mind:

• Hearing the gratitude from users where SpiderFoot has really helped them over the years has probably been the most personally rewarding and motivating. How many people can say they do something once, and then many thousands of people benefit and continue to benefit without additional effort?
• I am a far better programmer and overall engineer than I was when I started thanks to feedback from people like _bcoles along the way. This has helped me in my day job(s) as a manager when working with engineers and also bring the skills and knowledge gained from SpiderFoot into my daily work.
• I’m sure it played a role in helping me land multiple jobs and speaking engagements.
• It’s been a great platform to meet and engage with some really talented people, giving me instant credibility in a number of contexts since people didn’t know who I was, but they knew about SpiderFoot, so I was able to use it as a quick way to get my foot in the door.
• It eventually gave me insight into the business opportunity with SpiderFoot HX and served as the seed for that.
• Continually investing effort over the long term improved the product and compounded the positive effects of the above.

This also leaves out all the more personal and indirect benefits I’ve realised over the years: developing more discipline, understanding the trade-offs I’m willing to make technically and personally (and why), improving my time management, seeking out ways to increase my energy levels and health, the list goes on.

Lesson 2: Be in it for the long haul

How many times have you searched for an open source tool to meet a need, and dismissed what you’ve found because the last commit was years ago, or because there was one commit with the whole code base by one person and not touched since?

So much about a project’s success can often be that the project has stood the test of time and outlasted other projects. If you are consistently working on improving the project, it will get noticed and used over the flash-in-a-pan software that appears and eventually gets abandoned.

No one is saying you have to commit every day, week or month, but do your best to ensure that the software continues to work as dependencies change. Also keep an eye on issues being reported — if they point to a widespread issue, that should be given attention.

Lesson 3: Ship it and ship regularly

Let me be clear, there is much I regret within this ten year journey so far. For instance I should have put more effort into unit tests, code documentation and code quality from the beginning, but I also know that if I had spent the time learning and implementing those things to be great at them before putting SpiderFoot into the public domain, thousands of people wouldn’t have used SpiderFoot and gained the value they did. I may even have grown frustrated/bored during that time and not shipped it at all.

Keeping lesson 2 in mind, rest assured that you’ll have time to perfect things. Just put time aside each day or at least a couple of days a week, where you can work quietly on building something great and ensure the product works and works well. You will have to make trade-offs from time to time in order to keep shipping and not get stuck in trying to achieve perfection, and that’s OK — just about every software company in the world does this.

Lesson 4: Have broad, open-ended goals

My goal of becoming proficient in Python was really just one of my motivators for working on SpiderFoot. My broader and longer term goals for SpiderFoot have been:

• Craft a product I can be proud of and focus my creative energy where I see fit, the way I want to
• Generally develop and keep my technical skills active since my career was moving more and more into the managerial realm
• Go through the experience of building up a popular, successful open source project that is widely recognized as the go-to tool for OSINT within my industry
• Build a successful business around it, with Enterprise features

Note how these aren’t quite fixed destinations but are rather broad and open ended. I’ll never be done with any of these since each of them doesn’t have an end state; they can be stretched every year.

For instance, maybe I can re-write the front-end in React or implement a new test framework to develop my skills this year, and next year migrate to a different back-end database. And maybe I want to double MRR (Monthly Recurring Revenue) with SpiderFoot HX, or reach some other kind of business milestone this year.

These all serve as guiding principles rather than fixed goals, and therefore keep me looking ahead and motivated to work on SpiderFoot day in, day out.

Lesson 5: If you care enough, you’ll find the time

When people find out that I have a wife and three kids, a full time job, and run SpiderFoot HX as a side-business, the first question is always some variation of “where do you find the time?” Of course I’d like to think I have some kind of super focusing ability (I don’t), or am an incredible engineer (I’m really not), but I think I can boil it down to a few key things:

• The time I allocate is typically the time when I know I won’t be interrupted for a couple of hours. That means early mornings and/or late evenings depending on the sleeping pattern my one-year-old has decided to adopt.
• I watch very little TV (maybe a couple of hours a week) and only use social media to promote SpiderFoot (see lesson 7). Reading random things on HackerNews is probably my biggest procrastination tool, but I don’t comment on posts unless they are my own so that limits time spent.
• When I can’t code because I’m away from my laptop, I’m researching new data sources, looking up some new Python tips, seeing what other tools in the same space are doing, responding to emails, etc.
• After so much time, working on SpiderFoot has become a habit. It’s something I “just do” and can’t really avoid it for longer than a couple of days or I start to feel like something is off.

Lesson 6: No one cares about your unit test coverage

Or, put another way, people really care that your software does what it says on the tin and does it well.

The reality is that no one (OK, very very few) people will avoid your project because the code lacks unit tests or is messy. People care that the stuff works, isn’t a pain to get going, is documented and will keep working over time (think back to lessons 2 and 3). How you achieve that is really up to you.

So don’t fret about code quality, test coverage, etc. Many open source developers are not professional software engineers anyway, and it’s really a shame that so many people are intimidated by putting what they think might be “bad code” out there for fear of criticism. Code can be re-factored! Tests can be added! You can even re-write the thing in a completely different language if you want.

Just put something out there that’s useful and be receptive if people point out your mistakes, be humble and learn from it. Some SpiderFoot code is terrible but in ten years I think I’ve seen comments about this twice. Who cares?

Remember: it’s your project and you have time to make it better (lesson 2).

Lesson 7: There’s no shame in marketing

At some point, if your software does what it says it does and does it well over a meaningful period of time, others will start promoting it for you.

That being said, you still need to put in some initial effort to get the word out, and early feedback can be really helpful in understanding whether what you thought would be useful is actually useful to others.

So, know your user and where they hang out (Twitter? LinkedIn? TikTok?) and reach them there. Reach out to influencers (e.g. newsletter and blog authors) in your space who would find your software useful and ask them to try it out — you’ll be surprised how receptive people are to open source developers because they appreciate the time you’ve invested to help others.

Pro tip: Aside from getting the word out, have a nice README.md for your project that explains what it does, how to get it working and pointers to resources to learn more. Maybe even have a separate website that doesn’t look terrible.

Lesson 8: Clear it with your employer

The intellectual property laws protecting the work you do outside of working hours will vary country-to-country, employer-to-employer, but this is one area you don’t want to get bitten in. Most tech and tech-friendly companies these days should have a process around approving side projects, and hopefully most are flexible about allowing them. I Am Not A Lawyer, but ensure you cover both the work you’ve done before joining the company and the contributions you make after joining.

It’s much easier to get the intellectual property topics sorted out a) before you start working for the employer and b) before you start the project. I’ve even gone so far as to declare projects I want to eventually work on but don’t yet exist, just to ensure I’m covered for the foreseeable future.

If you’re joining a company that’s hostile to you working on open source projects in your own time, you should really ask yourself whether the role is worth that trade-off, and that’s a much easier decision to make before you’ve signed the contract.

Lesson 9: Foster community

This has probably been the one lesson I’ve struggled to learn the most in that I see it’s importance but it’s really a lot of time investment and something I should have worked on building from the beginning. I did create documentation from the beginning, but nothing else, assuming it would be enough. These days, it’s probably not.

By “foster” here I mean creating forums and resources for users of your project to go deeper, connect with others and get/give help if needed.

Here are some examples:

• Documentation, tutorial videos, blog posts, etc.
• Listening to feature requests and implementing them (as long as they fit with your goals and vision)
• Creating a Discord or Slack for people to ask questions and support others
• Presenting at conferences

Lesson 10: Keep it enjoyable

This should probably have been first lesson since it’s such a fundamental point, but I made it last because I want it to be the most recent point you take away from this post.

When you embark on any personal software development project, unless it’s incredibly trivial you need to immediately abandon the idea that it will one day be finished. Even if you consider it functionally complete at some point, dependencies will change, lessons will be learned, you’ll want to refactor things, users will bump into issues and have feature requests evolving with the times and so on. Even “competing” projects may emerge.

This can become overwhelming at times.

Once I accepted this ride will never end unless I willingly abandon it, a lot of my frustration and sense of urgency disappeared. I now spend fewer crazy late nights just trying to push out that new feature, or stressed over the growing number of open issues or pull requests, or feeling bad about test coverage not improving.

There’s no rule that says you have to merge every pull request, respond to every issue or really do anything at all you don’t want to, no matter what your reason is. Doing anything by force will only dampen your enthusiasm over time and may kill the project entirely, which is definitely a problem!

So ask yourself from time to time: are you doing the task at hand because you want to, or because you feel like you have to? If the answer is the latter, maybe take a break, work on some other aspect of the project or try to re-frame the task, otherwise leave it on the backlog for another week, guilt-free.

Conclusion

Beyond the above, of course I learned a lot of Python and other skills over the years — too much to write in a blog post. But when I eventually embark on my next project, (maybe something in Golang?), I’ll take these lessons with me. I hope that you do too.

If this post inspired you to finally put your project into motion or change your approach, I’d love to hear about it in either the comments below or otherwise feel free to contact me directly (just use the contact form on the SpiderFoot website — I receive all of them and respond to the non-spam personally).

I also want to take a moment to thank people who have been instrumental in this journey:

• My family for their ongoing support and patience
• _bcoles for all his contributions and guidance
• hakluke and his team of content folks for helping me with getting the word out
• The many contributors over the years that have helped grow SpiderFoot

So, what are you doing still reading? Go get coding..