How to apply Experimentation, Lean & Agile thinking when you are not producing software
Being Lean, agile, experimenting, and iterating are all the hype these days. Everyone wants to have validated learnings before fully launching a product or committing millions in production.
I am totally aligned with this line of thinking. But, what happens when you are not producing software? When you manufacture real, tangible goods like clothing, beauty products, or packaged food? How can a lean, experimentation mindset work in these cases?
In this article we will define what is an experiment and how it can help reduce risk and uncertainty, when to test and how to set up an organization primed for experimentation. Last but not least we will mention some practical ways to experiment and tackle uncertainty.
What is a Market Test / Experiment for physical goods?
For the sake of this article let’s say that a Market Test is the launch of an actual product (in an MVP form or not) in a Market, usually in small quantities and in limited distribution in order to test the Product-Market Fit and get early insights directly from Consumers & Retail Points of Sale. A Market Test has a commercial element, meaning that we actually need to sell the product to our end-consumers.
Testing for 4 types of uncertainty
In general, one can view testing as an activity that aims at resolving uncertainty in innovation and new product development.
However, not all uncertainty is alike, and here are some of them:
a. Technical uncertainty that comes up after the exploration phase of solutions (e.g. materials) that haven’t been used in this specific way before. That is often linked to product functionality and can be managed through careful prototype testing.
b. Production uncertainty reveals itself when there’s ambiguity towards the functionality of a technical solution that works well in prototypes but goes south when mass production is on the table. What may work in small quantities may not be feasible when production ramps up.
c. Need uncertainty originates because of the rapidly changing consumer demands. Consumers are rarely able to specify their exact needs or cannot describe their needs on products that do not exist yet.
d. Last but not least, in the presence of innovations, market uncertainty can be so notable that organizations are not taking the risk to allocate sufficient resources to the development of products for those markets, since they cannot assess them.
When you should do a Market Test / Experiment
As a rule of thumb, there are 2 parameters to consider when deciding to do a Market Test.
- High uncertainty in one or more of the aforementioned uncertainty types and lots of assumptions to validate
- High investment required in order to produce full scale (i.e in capability or manufacturing & supply-chain CapEx)
With the benefits of early prototyping and market testing, there remains the question of how frequently or how many experiments should someone carry out. It is true that experiments can be costly. However, the opportunity cost of finding problems later or not experimenting on promising ideas should also be factored into the cost-accounting equation.
Here is a helpful exercise:
Start by using historical data to find out your average cost to develop and launch a new product. Calculate all the launches you made in the past 5 years and see how many products have failed (or are underperforming). Then estimate an average cost to design & launch a Market Test.
If you had invested the money to test each product before fully launching, would that approach save you money & time on avoiding failures? If the outcome of this exercise makes sense, then you have a good product candidate for Market Testing or Experimentation.
Before you begin: Having the fundamentals in place is crucial
1. Set up your organization for experimentation
Experimenting and failing often (and fast) requires a totally different mindset (and probably skillset) than what big organizations are used to. Before emerging into your experimentation journey you should set up your organization appropriately, otherwise, you are setting yourself up for failure.
I like the model of Pioneers -> Settlers -> Town Planners, by Simon Wardley. This is the notion of having cell based structures (e.g. think Amazon Two Pizza teams) which are populated not only with aptitude (the skill to do something) but the right attitude (type of people). When experimenting with new products you’ll definitely need some “Pioneers”
Don’t get me wrong. As Wardley says “Pioneers, Settlers, and Town Planners are all brilliant people” . You need all of them in your organization. But in terms of Product Innovation & Experimentation, you will definitely need most traits of the “Pioneers” type.
In the beginning, you will probably need a dedicated team (a separate structure from the rest of the organization) to run these product tests. A team that moves fast and works in agile ways.
But you need to be really careful here. One common pitfall is that most organizations create “innovation islands”. Spaces where “innovation happens”. If you do that, the rest of the organization will probably treat innovation as something “that is not their business” and that “happens somewhere else”. As Wardley notes: “Each group innovates but innovation is not the same for each group, i.e. the innovation of an entirely new activity is different to the feature differentiation of a product which is different from converting a product to a utility service. Unfortunately, despite being different forms of innovation that won’t stop people pretending there’s only one and it’s all the same. Try not to do this.”
To solve this challenge, it is very important to have many people onboard that would drive the change. Also, it is very powerful to enable those who want the change to happen to be the ones to make it happen.
2. Use experimentation for building a learning muscle and acting as a change agent
Market Tests & Experimentation should not be treated just as a verification method (i.e we launched a small product batch in an area and measured if consumers did buy the product or not). You need to instill this mindset of change, learning, and agility into the whole organization.
Development teams that undertake the design of new, innovative products rarely know in advance whether a particular concept will work exactly as intended. This means they have to find ways of rapidly discarding dysfunctional concepts while making sure they retain others that are promising.
In order to achieve that, structured experiments are required. The directed effort there would be to manipulate variables of interest. In an ideal experiment, the independent (the ‘cause’) and the dependent (the ‘effect’) variable should be separated. The step that follows is to manipulate the former to observe changes in the latter. The manipulation, followed by careful observation and analysis, then gives rise to learning about relationships between cause and effect, which, ideally, can be applied to or tested in other settings.
The key element here is to share back any feedback and learnings not just with the product teams but with the whole organization as well. And do it in a fast and consistent manner.
3. Invest in Infrastructure for Experimentation
You don’t just wake up one day, set up a team, and experiment. You need to make some investments that will allow you to experiment & do market tests in scale.
If you are Zara, for example, you would invest in strategic, technology-driven inventory management, that will lead to greater visibility and control over supply chains.
Zara has a process of bringing new apparel to life (in limited quantities for testing & experimentation) in just 25 days from design to store.
If you are producing beverages you might want to invest in creating a small, flexible boutique plant so that you can quickly produce small batches of products that could be shipped fast-track to points of sale, or partner with a number of Co-Packers to do small scale bottling.
Practical ways to tackle Need or Market uncertainty
Although there are different ways to approach testing & experimentation here are some practical examples that have worked well in the past
Print it instead of manufacturing it
With 3D printing on the rise, testing a new package should be something that can be done relatively easy and at a low cost. Even more challenging products (like a beer bottle) can now be 3D printed and tested.
James Boag’s Premium Lager underwent a bottle makeover and wanted to test how a new bottle design would perform with consumers.
To do that, they 3D printed the prototype. To be convincing, the 3D printed models needed to have the same clarity and hue as glass as well as the same in-hand heft.
It took only a week for the models to be completed, at which point they were outfitted with labels and caps and filled with liquid. They were then put in a shop for monitoring, and feedback showed that the new design was a hit, giving James Boag’s team the confidence to go ahead with the full redesign.
Selling a “non-existent” product
Getting validation on Need and/or Market uncertainty is something very tricky to do. Many organizations run surveys to understand if consumers would buy a product but usually what somebody says in a survey (intent) vs what they actually do when it’s time to buy (behavior) can be very different.
In the software industry, startups can sometimes have pre-registration periods even when the final product is not ready yet, just to create some word of mouth and gauge on consumer interest.
On product design, an organization could fund a product by promoting it to Kickstarter to find out if there is any interest before mass producing it.
But what happens with more “mainstream” products? One way we have seen working well is “selling” a product that does not actually exist (yet).
The concept is that you create the full product proposition (from branding and packshots to pricing and promotional materials) and put it up for “sale” either on your own digital channels or using 3rd party e-retailers (after an agreement, of course). This way you can measure how consumers react to this new product and if they are actually buying it.
Of course, after someone places an order you reveal that this product is actually a test. You should also offer the consumer some reward for participating in your experiment. Although it may sound easy, there are many details to pay attention to like displaying the product only to a representative sample, dealing with all the legal stuff, getting feedback from the consumer after their “purchase” and so on.
Getting feedback from actual consumers using your products
When you can produce a small batch of products it’s really helpful to have your product tested in real-life conditions by real consumers. I’m not talking about getting feedback from focus groups.
There are agencies out there that can help you recruit the right consumer sample, ship consumers the actual product, and get their feedback.
Some brands can go even further by creating their own consumer panels in order to test new products and ideas before massively launching them in the market. A good example is Boots Volunteers.
To wrap this up, we recommend you follow a Design Thinking approach for rapid testing
Sometimes you need to understand if a product you currently have, would work on a different market or through different channels (D2C). This is what we wanted to explore with Coca-Cola in Austria.
To do that we had to bridge the gap between demand and limited distribution for certain products. Such cases include limited editions like custom Coke Cans with your name on it, SKUs extremely difficult to find on the shelf like Coca-Cola Signature Mixers, or apparel from the latest partnership of Coke with Diesel.
Instead of spending months building a business case, sourcing products, managing distribution, and then scoping a huge eCommerce platform, we chose to run a rapid experiment that would give us both an early validation if such a solution would have any market fit, but also get real market data to build a solid business plan.
Using our own version of the Design Sprints Methodology, we run our experiment in just 5 days getting some very valuable insights. Watch the video for all the details:
