Harvard professor Thomke on Why Business Experimentation Matters
Stefan Thomke, an authority on the management of innovation, is the William Barclay Harding Professor of Business Administration at Harvard Business School. He has worked with US, European and Asian firms on product, process, and technology development, organizational design and change, and strategy.
Since joining the Harvard faculty in 1995, Professor Thomke has taught and chaired numerous MBA and executive courses on innovation management, R&D strategy, product & service development, and operations, both at Harvard Business School and in individual company programs around the world. He is chair of the Executive Education Programs Leading Product Innovation and Driving Growth Through Innovation, which helps business leaders in revamping their innovation systems for greater competitive advantage, and is faculty chair of HBS executive education in India. Professor Thomke is currently on the core faculty of the General Management Program (GMP) and has previously taught in the Advanced Management Program (AMP). Previously, he was faculty chair of the MBA Required Curriculum and faculty co-chair of the doctoral program in Science, Technology and Management. He is the recipient of numerous awards, including the Apgar Award for Innovation in Teaching at HBS and a HBR McKinsey Award Finalist.
Professor Thomke’s research and writings have focused primarily on the process, economics, and management of business experimentation in innovation. He is a widely published author with more than one hundred articles, cases and notes published in books and leading journals such as California Management Review, Harvard Business Review, Journal of Product Innovation Management, Management Science, Organization Science, Research Policy, Sloan Management Review, Strategic Management Journal and Scientific American. He is also author of the books Experimentation Matters: Unlocking the Potential of New Technologies for Innovation (Harvard Business School Press, 2003) and Managing Product and Service Development (McGraw-Hill/Irwin, 2006).
Professor Thomke was born and grew up in Calw, Germany. He holds Bachelor and Masters degrees in Electrical Engineering, Masters degrees in Operations Research and Management (MBA equivalent), and a Ph.D. degree in Electrical Engineering and Management from the Massachusetts Institute of Technology (MIT) where he was awarded a Lemelson-MIT doctoral fellowship for invention and innovation research. Professor Thomke was also awarded honorary degrees in Economics (Doctorate from the HHL Leipzig Graduate School of Management) and Arts (Masters from Harvard University). Prior to joining the Harvard University faculty, he worked in electronics and semiconductor manufacturing and later was with McKinsey & Company in Germany where he served clients in the automotive and energy industries.
Contact information: Harvard Business School, Soldiers Field, Morgan Hall 489, Boston, MA 02163 (U.S.A.); Telephone: +1 (617) 495–6569; Fax: +1 (617) 496–4059, E-mail: t@hbs.edu.
Arjan: Could you tell the readers how you got interested in experimentation?
Stefan: I was a doctoral student in electrical engineering at MIT and worked on multivariate process control of chip manufacturing. Our research group was exploring the efficacy of adaptive statistical models rather than traditional control methods. It was around 1990 and there was a lot of interest in topics such as quality control in manufacturing, robust engineering, etc., which meant that you had to learn and apply experimental design methods. So I became an “experimental methods” nerd, without thinking about the bigger picture. A few years later, I became interested in the broader application of experimental methods to product design and development. After I joined the HBS faculty in 1995, I made the process, economics, and management of business experimentation in innovation my main research thrust. It wasn’t clear at the time how the academic business community would respond to the work since I was deeply influenced by my engineering perspective. Fortunately I could bring considerable depth to the topic and, over the years, was able to publish a large body of work in top academic journals. HBS decided that I had graduated from an engineering nerd to an influential business school academic and decided to keep me. In some ways, my academic career was one large experiment that worked out. Initially, I wasn’t interested in an academic career at all.
Arjan: Your book is in my top 5 favourite business books of all times. After reading it I was determined to make a substantial contribution to the community of business experimenters.
Stefan: You are very kind. I wrote Experimentation Matters (HBS Press 2003) to show how innovation managers can take advantage of the rapid changes in experimental methods and also to provide theories and frameworks for academics. So I violated the golden rule in publishing: pick a target audience. My editor was concerned that it would be too applied for academics and too academic for managers. But the book did well and the publisher has sold more than 10,000 copies with no advertising.
Arjan: Your HBR piece with Jim Manzi had a huge impact on the industry. It received the “HBR McKinsey Award Finalist” for the best article in 2014. Could you share some of the positive reactions to the HBR piece?
Stefan: Jim and I were very pleased and surprised by the award. These awards usually go to big picture articles (like leadership and strategy) and our piece was fairly hands-on and technical. But it also broadened the topic of business experimentation beyond R&D and made it clear that this is something that every senior executive needs to care about. It also showed that experimentation is not just “trial and error” but that there is a discipline to doing it well. And if you don’t experiment and test, you can get yourself into a lot of trouble.
Do you like what you have read? Get a quarterly update of what I am busy with.
Arjan: Because experimentation sometimes makes clear that assumptions and intuitions of executives aren’t always correct, I can imagine there was also some pushback? Was there any, and if so could you elaborate?
Stefan: I didn’t say that it’s all experimentation and no intuition. I said that when it comes to innovation, just relying on experience and intuition alone is a recipe for disaster or lost opportunity. It’s always a mix of experience, intuition and experimentation but unfortunately well-executed, disciplined experiments get short shrift because executives are usually in a hurry or don’t know how to do it. That’s what happened to Ron Johnson at JC Penney. Once in a while, I get some pushback. A while back, I gave a lecture on the topic and a CEO in the audience really pushed back and gave me an earful about the importance of intuition (a la Steve Jobs). In principle, I agreed with him but most managers aren’t Steve Jobs. But what I couldn’t say in front of everyone is that his company was already a regular and successful user of the disciplined experimentation methods that I described. For some reason, he wasn’t aware of it and I didn’t want to embarrass him.
Arjan: Let’s be clear on what we mean by an experiment. Everybody nowadays is running experiments, trying different things, but what does qualify in your opinion as an experiment?
Stefan: I have a clear definition. In an ideal experiment, the tester separates an independent variable (the presumed cause) from a dependent variable (the observed effect) while holding all other potential causes constant, and then manipulates the former to study changes in the later. The manipulation, followed by careful observation and analysis, yields insights into the relationship between cause and effect, which ideally can be applied to and tested in other settings. There are, of course, many practical considerations. For example, companies often confuse experiments and commitments. So I tell them that an experiment needs to be reversible; otherwise it’s a commitment.
Arjan: Experiments always consist of similar ingredients, right? You have a hypothesis, you might have a control group, and you will have to think about the sample and its size. What more standard ingredients do you have and which of them is the hardest to get right, and why so?
Stefan: Yes, Jim and I identify five critical questions that have a set of design issues that need to be addressed. Again, this is an ideal that companies should aim for, even though it’s not easy the first time around. First, does the experiment have a clear purpose (question, hypothesis)? Second, is there a commitment to abide by the results (if not, why bother to experiment)? Third, is the experiment doable (testable prediction, sample size, etc.)? Fourth, can we ensure reliable results (randomization, controls, etc.)? And fifth, can we get more value out of the experiment (ROI, context, etc.)?
Arjan: What do you think of the worldwide growth of the A/B test movement?
Stefan: I think that the interest in A/B testing is great. It tells us that there is huge interest in the topic. However, I do see two issues. First, I am concerned that it’s sometimes not well-executed and/or used to address questions that aren’t suitable for A/B testing. Second, and the bigger issue, A/B testing requires big sample sizes and are mostly suitable for direct channels like the internet (like testing webpages). However, most consumer business is conducted through complex channels that suffer from a variety of analytical complexities that practical sample sizes are unable to address. Such environments require more sophisticated approaches, both analytically and managerially.
Arjan: Finally Jim Manzi also authored Uncontrolled in which he makes the case for running experiment to improve and innovate society. You have a clear focus on experiments for commercial organizations. Can you tell us why you are so committed to businesses (editorial note: Stefan has been working on this topic since the 1990s, trying to get businesses to run experiments to fuel innovation for almost 3 decades now)?
Stefan: You need to ask Jim about his passion for politics ☺ I’ve always been interested in businesses and believe that if I can help managers to become better at innovation, societies will ultimately benefit from it. It’s not always a straight line but if you look at all the progress that we’ve made in the last hundred years, innovation is the fuel that keeps societies moving forward.
