How to test and validate your product with new users
Hi there, I’m Jeffrey.
A few weeks ago I set out to build a web app to help Todoist users find and share their to-do lists. The product is called TodoTemplates — a community gallery of Todoist projects. You can read about how I started from scratch and built a working prototype in about 24 hours using Meteor and an open-source Product Hunt clone called Telescope.
Today I’m going to tell you how I went from prototype to validation.
But first, a story about creativity and ideation. This has an important lesson, please read if you can!
We’re all creative
If you’re like me, you love to come up with crazy ideas, tinker with things and apart to see how they work. As a kid, I would gleefully take apart my remote-controlled cars, inspecting all the electronics, pulling out some bits here and there. I was pretty good at dismantling things (still am). One thing I’m not good at is putting it back together. I would toss all those electronic parts into a small box. It became a mini-graveyard of electronic devices, I simply didn’t know what else to do with it.
You might not know this: creativity is a skill and can be learned.
There’s a common belief that most people have a combination and preference for how they are creative. Here’s a quick breakdown.
We all have a preference for our creative strengths, it’s not a matter of good or bad. I’ve always been a strong ideator and implementer. In other words, I’ll come with an idea and jump on it. But the idea probably didn’t solve a problem, and I probably wouldn’t bother putting it through rigorous testing and refinement. If you have a low preference for all of these skills, you are said to be an “integrator” (think, facilitator or manager).
Back to the story…
Fast forward 10 years to 2010. In college I studied electrical and computer engineering at WPI. I went through great lengths to built fascinating things. My favorite creation was a CNC-machine. It was massive, taking up a 5 ft x 7 ft x 3 ft space in a bedroom of my apartment.
The actual machine was much smaller, but since it was part of a family house I had to make it as quiet as possible. It was a LOUD beast. I went to Home Depot to collect supplies so I could build a sound-dampening chamber (think particle board and spray-foam insulation). Also, because of all the dust it produced, I had to build and attach a vacuum hose and filtration system. The entire machine cost around $1,500 and over 3 months to build, but was totally worth it.
Success without purpose is short-lived
During an engineering competition at WPI, my team used it to create a printed circuit board (PCB) for a prototype product that won us 1st place. I was stoked! My plan was to create PCB’s and try to make some money to offset the cost of building the machine.
OK, fast forward a few months after I built the machine.
I hadn’t used the CNC machine in weeks after the engineering competition. The PCB’s I created all had small defects, and they definitely weren’t high enough quality to sell. In fact, I couldn’t make a working PCB — it was a miracle the prototype used at school even worked at all! Fixing the machine wouldn’t be worth the effort. It’s just taking up a huge part of the bedroom, and I didn’t know what to do with it.
It sat there until I eventually had to move, so I dismantled it and sold off its most expensive parts, throwing the rest in the dumpster. What a drag to see your creation and hard work go to waste like that!
This was an important lesson for me, and totally changed the direction of my life. I loved jumping in head first, bringing my idea to life in the shortest amount of time possible. But, I didn’t think it through. My idea came to life sparkling and shining, impressing my friends and family. People would “Oooh” and “Ahhhh” when they came over to visit. “What the hell is that thing!?” they would ask me. Explaining it was fun, but using the machine was a pain. It was imprecise. Setup and alignment would itself take a half hour. And it wasn’t a guarantee it would stay aligned. It led to imperfections in the boards which rendered them useless.
So then my idea quickly died, laying lifeless in the middle of my bedroom. It was motionless, sprawled out across my desk. It wasn’t long before it became an eye-sore, just something that got in the way.
Learning Lesson
Fast forward to today. I’m much less impulsive and rarely go full-throttle anymore. I’ve become very skeptical of my own ideas and now ask myself clarifying questions like these:
- What problem is this solving?
- What’s the end-game?
- What do I get out of this?
- How much effort would be involved?
- Do I realistically have enough time to devote to this idea or project?
Thinking like this has its pros and cons. Obviously there are less ideas I’ll pursue so I actually end up producing less into the world, and that’s by design. I save a fair amount of time that might have otherwise been wasted on poorly thought-out ideas.
The downside is that I haven’t really jumped into any of my ideas — it’s as if I’m too afraid they’ll fail. Anyone who is interested in startups knows that failure should be embraced. It’s only recently that I’ve found a solution, a balance, to this problem.
The moral of this story is this:
Find a balance between being productive and being useful.
As with most things in life, moderation is key. Use the creative thinking diagram to find the skills you need to develop. Better yet, find smarter people to fill those gaps, and bring them on your team.
“Everything in moderation, including moderation.”
If you’re too impulsive, you’ll burn out quickly and risk having anything left of value. Focus too much on only being useful, and you’ll lose touch with the creative path that can lead to break-through ideas.
We’ve already discussed how the TodoTemplates idea was born, and how it was implemented. Now let’s chat about validation — the clarify step of the creative thinking process. I knew the problem existed, but I needed more proof to be certain that TodoTemplates could solve it.
How I validated the TodoTemplates product with real users
After getting a few users to post comments and a couple projects, I started to reach out to more users. I looked up some who wrote comments on the The Todoist Team blog, gave a quick shout on the Twitter, and even went so far as to write directly to some support reps at Todoist to get their opinion.
OK, so that’s an indirect validation from someone who is very close to the Todoist user-base. Let’s see how people respond to a quick invite to view the prototype. Here is a screenshot of a conversation I had with a user on Disqus:
If you noticed, I proposed the idea months ago, and only recently took the initiative to produce a solution. The idea was stewing in my brain for a while, so when the timing felt right, I jumped on it. This is a great example of balancing my preference for ideation and implementation with my less-favored clarify and develop skills.
Be real and be real helpful
I also reached out to a few other people on the blog, with limited success. Some people would add me on twitter, but most had no response. That’s probably normal.
Here’s where things really start to get interesting.
Based on the limited feedback I was getting, I didn’t have much to adjust on the site, so I kept up the marketing efforts. Spent a few hours per week promoting where I could and trying to be helpful, not spammy, like a good marketer :)
Check out the traffic on Google Analytics. There are spikes in traffic on Tuesdays for some reason. I couldn’t tell where it was coming from. My gut was telling me they were bots, but after looking at the user behavior, it seemed natural.
Visitors were spending around a couple minutes each, and the bounce rate was pretty low. Acquisition was coming from 40% direct, 40% social. So probably some are bookmarking the site and coming back to check on things — that’s a great sign!
The numbers on Reddit are also looking good. Over the past few weeks I’ve been helping Todoist users in /r/Todoist with prompt and helpful responses, and asking for feedback. The last 2 months have seen the largest increase in unique users within the past year.
One user mentioned that he/she wouldn’t import any projects unless he/she could see a preview of it first. That makes sense. From now on, I’ll be including a preview screenshot in the body of the posts. Check out this response:
I’ve been in so many situations where I just wanted to be heard — especially when it comes to developers. It looks like I’m not alone.
Look closely at the numbers
The telescope app, the thing that powers TodoTemplates, comes with Meteor analytics integration built-in using Kadira. This gives you a deeper insight into user behavior on your app. I peeked at this once and haven’t touched it since. Everything seems to be running just fine (kudos to Sacha Greif for a well-made tool). If I see a drop-off in users, or the bounce-rate spikes, I’ll know to check Kadira to make sure there aren’t any problems with the app.
Oh, if you use Google Analytics, don’t forget to filter out traffic from your own IP so you don’t skew the results.
I created a dashboard on Dasheroo. If you haven’t heard of it, you definitely need to check it out. You can create a dashboard with your very own key performance indicators (KPI’s) to keep track of your progress. You can then use the insight from the dashboard to see where you are under performing or where you can take advantage of opportunities. They have connections to a lot of apps already, such as FB, Twitter, and Google Analytics. The cool part in my opinion is their integration with Google Sheets. You can practically track anything with that combination!
I’m keeping track of daily sessions, total subscribers, email list health (bounce/click rates), and starting to track FB/Twitter followers.
I think the spike in list subscribers is just the day I started tracking, in reality it has been a gradual increase.
Results
I’m seeing positive indicators across the board, and I have a special post coming up that will have a huge impact on traffic — I’m super excited about it!
Tune in next time to hear what exactly the secret sauce is, and why I’m confident it will be a turning point in growth for TodoTemplates.
Thanks for reading!
If you enjoyed this article, please click ❤️ so others can find and enjoy our story!
