Building the experimentation process at Josh Talks

For a product as interactive as Josh Skills where each day a user spends an average of 51 minutes, we want to make sure that what we are building is based on tested hypotheses and not assumptions or intuition. That is where experimentation comes in. So, how exactly did we build this process? Here’s how.

It all started on a trip outside the city — Shobhit and I were sitting together discussing how we could introduce a streamlined experimentation process. Experimentation, you see, is one of the most important weapons in the arsenal for scaling products successfully.

I remember, one of the key issues that we were looking to solve while building this process was effectively managing experiments and not getting lost in what we were building. Essentially, we were confronted with these two questions —

• How can we have our team run experiments?
• How do we effectively manage these experiments?

We wanted to make sure that every team member feels challenged but still comfortable with this process.

Getting started

We started by reading a book that Shobhit had found called ‘Hacking Growth’ by Sean Ellis & Morgan Brown. I believe it’s one of the best books that I have read till date. It’s a textbook which doesn’t just tell you what to do, it tells you how to actually do it.

The key highlight of the book was hacking an open forum where not just your product team but tech, testing and all the other teams could share as many ideas as possible to take the product forward. Self-censorship is discouraged and any and every idea is entertained.

Sharing ideas

All of us spend so much time with the product that we usually keep on ideating on how we can make it better. Every person has ideas to share and let’s agree on one thing — they SHOULD BE able to share them because you never know where the next BIG idea will come from. And, these ideas should be shared through a platform where they can be perfected.

So what next? We decided that all of us will have to share as many ideas as we can. But for this, we wanted a platform where they could be managed from. After a lot of research and testing, we finalised JIRA.

On JIRA, we started with the two projects — core product experiments and growth experiments.

Whatever ideas improved the delivery of the product’s value proposition went under CORE PRODUCT and others that were related to growth of the app went under GROWTH.

The metrics that mattered to us were added as ‘epics’ in the project (Example: under ‘growth experiments’, one of the epics was called — Download to Purchase Conversion Ratio of 8%). Epics refer to a body of work in a project that can be broken down into smaller work pieces. We created and added epics under the two projects, and had a discussion with the team where they were edited. Finally, we could now start sharing ideas.

All of us could submit our own individual ideas. Everyone had 3 days to put their ideas in this format –

Nominating ideas

Once the ideas were shared, the team had 3 days over the weekend to go through each idea and nominate 3 ideas each. Regardless of which area (core product/growth) you worked on, you could nominate any idea which according to you was relevant and important to move the epic (read: metric) forward.

Selecting ideas (The Tuesday Meeting)

Tuesday was decided as the day where the nominated ideas were discussed, closed and finally selected. We all sat in a meeting room, some of us in person and those who weren’t present physically, joined remotely.

As this was the first time we were doing this, we all presented ideas that we had nominated as part of the epic. Those in charge of managing the epic heard the nominated ideas and decided which ones they wanted to build on.

Building and experimenting

Even if we feel strongly towards one idea, we can’t really predict if it will bring the desired impact. We have to build it, test it and then can we know if our change has the desired impact.

One of the most important things that we were looking to solve by creating this experimentation process was to effectively manage our tech sprints. Our team had expanded from 6 to 24 in a span of just two months and we wanted to ensure that our productivity and focus was as clear as when we were 6.

With all of the ideas now mapped out for each epic, we started experimenting those ideas. We decided to follow a 4 day sprint — starting Tuesday and ending next Monday.

For the first tech sprint, we only picked those ideas that we could actually build in 4 days. We obviously had a few misses but by continuously experimenting and iterating on the sprint length, we reached a 14 day sprint where ideas were built, tested and pushed by the end of it.

Going fast and breaking things

In the first 3 months of building the experimentation process, we ended up writing a total of 43 experiments, giving way to a fast evidence-based process. Experimentation has become a part of our innovation process now.

For a team that is scaling rapidly, following a process can be tough and we did leave scope for some chaos so that we don’t lose touch with what we believe in — ‘going fast and breaking things’.

What building this process did was bring the team extremely close together. Something that was just the product team’s place became a forum where every team member could share their ideas and have a say in building the product.

If you too are someone who loves to put their ideas to test and are excited to solve problems at scale, we are hiring for roles across product and tech. Write to me directly at harsh@joshtalks.com!