Failed projects and a different approach: product discovery at Mindsay
Rethinking how we prepare, plan, and iterate.
As an engineering manager in a product company, a big part of my job is to help my team plan and execute new features that will make our product more valuable to users. The process for this at Mindsay used to be quite uncontroversial: decide what the feature should do, design, work out the technicals, deliver iteratively.
But recently, our team was shaken by a small revolution that caused us all to reconsider our approach, try something new, and emerge much more confident in delivering future projects. We did this by taking a much more iterative approach to all steps of a project (planning, design, specification), instead of just the implementation.
I will not dwell on theoretical considerations too long. Instead, I want to focus on the difficulties we faced delivering impactful projects, how we were inspired to change, and what we did to turn things around.
The examples of our last two projects are striking: both were sound, relevant features that seemed to answer a clear need from our users. Both were delivered in a timely, iterative fashion. Both were a source of pride and praise for the team. And still, one of them had vastly more impact on our product than the other.
This was not pure chance, but the result of a large overhaul in the way we think about and deliver product improvements. That’s what I would like to share with you today.
What made us change
I promised this part would be brief. What I can say is that everything started with this guy:
Maybe you know Marty Cagan, he is quite a star in Silicon Valley. I strongly encourage you to watch the video.
Here is the central message:
To deliver a feature that works for your users, iteration is key because. But you are not iterating in a relevant way if you only do so during delivery. Many companies prioritize and specify features in a non-iterative, monolithic way. When that happens, regardless of the adaptability of developers, many projects will fail: if the functional requirements are rigid, no amount of delivery iteration will make the feature more relevant. That is because getting specs right before you can get any feedback from users is nearly impossible.
In practice, you’ll notice it by a history of projects that get little to no adoption, that you have to work hard to push to users, and that feel like they are missing that “little something extra” that would make them invaluable. Failure can happen, even though the team produced quality software that matched the initial requirements.
To solve it, you should emphasize product discovery over specification: define features as you go, working hand in hand with your users and technical experts.
As an engineering manager whose job is to ensure that the team delivers timely, impactful projects to keep our product great, this resonated with a lot of my recent experiences.
What struck me when we started to dig deeper was both how easy it was to agree with Cagan (we recognized ourselves in much of what he said); and how difficult it was to decide on concrete changes to match this mindset. Cagan gave us value. We needed actions.
This is the experience I want to share here: for one project, we chose to take a leap of faith and commit to an extreme interpretation of these principles. On the whole, this was a great experience. It was also full of learnings.
Aaaand, action!
The project
At Mindsay, we enable our customers to build chatbots. This involves training machine learning models, which can take quite some time.
Which brings us to the project: we wanted to improve the experience of users training their bot. At the time, monitoring trainings in a bot looked like this:
Yuuup. You got it right. That’s a Slack channel. This came with tons of drawbacks (internal use only, very messy, hard to investigate errors…), and it was far from the only issue we had with trainings: long waiting times, unclear automatically triggered trainings, …
So we bravely set out to righting this wrong.
The “wrong” way
We actually started planning out this project before shifting to our product discovery approach. So we followed our usual process:
- Product managers prioritized the feature and inserted it in the roadmap
- Some time before the start of the project, we identified the pains with the current feature. We used it to propose a list of things we wanted to achieve in the project.
- Then we sat down and worked out a release plan. This was quite hard and time consuming because we spent a lot of time arguing about the best order, about interactions between features if the release order was wrong…
But we did come up with a plan that made us feel smart, well-prepared, and confident. Here it is:
- And finally, Axel, our Head of Design, created a nice version of a revamped training UX. This also took him some time, and he only designed the final version.
Now, this does not look so terrible, does it? But in practice, batching all of the product and design effort as a preparatory phase is both a big risk for the project and a waste of many people’s time.
- While doing this, we were constantly guessing. When we had questions like “will this be useful?”, “which of those alternatives is the best?”, “is that really necessary?”, all we could do was debate and try to project ourselves.
The result is that we took decisions that sounded reasonable, but whose real impact was unproven at best. - We overplanned: we spent a lot of effort specifying things that we knew had little chance to ever exist.
- We overdesigned: I can tell you now that 90% of what Axel designed at that point was lost. It was good, pretty, clever work, but it was more than what we had time to deliver, and it did not take into account all of the unexpected insights that inevitably come up during a project.
Once you come in like this, you can have all the agile delivery cycles you want, and your feature is still going to be off the mark. Even having SCRUM-like review will not save you, because at that point they are mostly progress reports: you are not tailoring your feature exactly to your user’s needs. You are working towards an output, but with no proven outcomes.
What we did instead
When we decided to rethink our approach, we chose to do everything we could to get as much feedback from users as early as possible in the process.
First day, first prototype
We agreed that to really commit to this mindset, we wanted to have a demonstrable prototype by the next day.
We took an hour to brainstorm a super basic status page for trainings, the APIs we needed, and the rough visual outline. Then we all set out to coding, with two of us on the backend and one on the front. We specified as we went, working side by side and communicating constantly. Within a day, we had something ugly, slightly bugged, with an improvised UI, but representative of what we had in mind.
This allowed us to actually go to users and ask them what they thought about it. We were very surprised by their reaction: they loved it! It was a gross simplification of what we first had in mind with our big release plan, it was raw and unpolished, but it gave us all the data we needed to move forward.
In particular, it confirmed one thing we were very unsure about: while we had initially planned to integrate the training information in various places of our bot-building experience, a simple, easy-to-make status page was more than enough for our users’ needs.
Cruising speed: 2 demos a week
To keep gathering feedback, we imposed ourselves a rhythm of two demos every week with users. We made a point of letting users use whatever we had built rather than showing it to them, even if it was unfinished.
Interacting with the feature helped users find out what they really need. Rather than having to specify the feature in a vacuum, the next steps emerged in a natural way.
Starting from a quick and ugly version also allowed our designer to observe users in action on the feature and design a UI that really clicked with users.
Throw-away code
When you start cheap and dirty, you can afford getting things wrong. The idea with this approach is that you first write code as though you plan to throw it away in a week.
This allowed us to validate hypotheses: at some point, we had a big debate about the best layout for the page. We simply implemented both! Once users used it, our decision was a no-brainer.
Finishing up
When you have a plan, you want to do everything. Sometimes, you’re going to rush your core functionality because “I need to make this thing we planned.”
Here, we were able to focus on usability rather than extensiveness. We were comfortable working and reworking the same part of the feature until we were sure that users could understand, use, and draw value from it.
When we were confident that a functionality was here to stay, we took the time to clean it up and bring it up to our code quality standards. In the end, we finished the project with a robust feature that was immediately adopted by many users.
The project was a great success. It was useful, and it steered clear of overengineering. We have kept improving the feature ever since and now people can’t live without it!
Learnings
Overall, this dive into product discovery was a great experience. It broke taboos about planification, writing trashable code, and how often we can interact with our users directly.
We also made tons of mistakes in the process, and it allowed us to draw learnings for our next projects:
Accept improvisation… But come prepared
At first when we chose to switch to a product discovery approach, we dismissed the release plan we had made as a mere waste of time, which would be made unnecessary by our iterative approach.
In a sense we were right: the complete release plan was not a realistic view of the project. On the other hand, thorough preparation of the project meant that we had tons of insights into the pains of our users, and ideas on how to address them. As a result, whenever a demo was unconvincing, we were able to come up with new, inventive approaches.
This is important because usually the most useful feedback you get from users is negative: they can easily tell you what doesn’t work for them, but are rarely able to know exactly what they need. For one, their lack of technical knowledge means they do not even know what is possible! In those cases, you’ll be happy that you did your homework!
In practice, this means that ahead of the project you should do a lot of data gathering, and reflect on the objectives and overall philosophy of the project. This can come in many forms: sift through existing user feedback, interview your customers, explore your analytics… The main point is to have a firm understanding of the problem you are trying to tackle.
First, do no harm
Quick and dirty iterations are by no means an excuse for breaking your product.
The prototyping phase must be well-isolated from production code, and remain invisible for your customers (for instance using feature toggles). At the same time, you need to ensure that any code that remains at the end of the project complies with your production standard.
In this project, we found it difficult to find a middle ground between final code quality and iterative, quick-and-dirty development. A lot of the time, we were not 100% clear on when we were writing complete throw-away code and when we were building to last.
That means that we were usually kind of in-between: when you’ve been used to following strict good practices, even fought for them, you feel terrible writing ugly code. So just in case, you make it a lot cleaner than you have to. But it’s still just okay code, not production-grade stuff, so you still end up rewriting most of it. This was pretty frustrating.
Something we plan to try is to clearly define a binary status for the code:
- Discovery code: it can be as dirty as needed, what counts is functionality. You can even ‘cheat’ (for instance do stuff on the frontend that you should be doing in the backend), it’s all going to be dropped anyway. Do not build anything on top of this code, otherwise you risk getting stuck with it. Either drop it, or if you decide to keep it fully refactor it.
Do not feel bad about it being terrible, any effort you put on code quality will go to waste. - Production code: once part of a feature is definitely here to stay, build it to last, following all of the usual guidelines.
If you stick to this and have a consensus in the team about which is which, you’ll save a lot of frustration.
Conclusion
By rethinking our approach to project planning, and accepting the uncertainty that comes with any project, we’ve been able to use everyone’s time more efficiently, and greatly improve the final quality of features. It is also a more fulfilling way of working: rather than creating friction between disconnected plans, pressured tech teams, and frustrated end-users, it brings everyone together.
Beyond this specific approach, I think the lesson here is: don’t be content with your current processes. You can do better. Keep getting informed and trying out new things, and any time you lose figuring out what to do will be worth it ten times over.
About us
Our mission at Mindsay is to help companies provide simple and efficient interactions to all of their customers at any time.
Check us out!
We’re also hiring
Additional resources
Another video which had a big impact on our iterative strategy:
In this video, Tom Chi explains the approach his team had to building Google Glass: instead of running what he calls “guessathons” (meetings where everybody tries to anticipate how useful a feature can be or how users will react to it), they just tested their ideas in practice. They built dozens of prototypes and made a point to get people to test them as fast as possible.
An anecdote I love: at some point in the project, he had the idea to include a laser projector that could overlay a keyboard on any surface. Instead of thinking about it forever, he just tried it out in a super basic way by attaching an off-the-shelf projector. He first found it amazing (I can type anywhere!), but as soon as he came across anyone he realized he was blinding them! This made the whole thing unusable.
