Six Principles Of Applied Research
Thanks to the rise of Agile methodology, and implementations like Scrum, effective processes for building products are well understood. But this understanding tends to break down when products rely heavily on research. And conducting research in a commercial capacity — conducting Applied Research, in which the outcome of the research is applied to the company’s product(s) — is increasingly important, as companies seek to distinguish themselves by developing new technology that gives them an edge over their competitors.
Frameworks like Scrum don’t work well for research projects. Scrum is designed for delivering clearly-defined features in short development cycles where the amount of work required can be estimated; research projects tend to have less clearly-defined outcomes, involve longer timeframes and require amounts of work that are tougher to predict. So, for research teams to work efficiently, different frameworks are required.
Getting the framework right is important, as the costs of not doing so are severe. If a research team is too detached from the rest of the product drive, they’re likely to build great things that don’t shift the needle from a product perspective (and research not applied to product is a waste of time and money for everyone involved). But if they are treated too much like a product team, using processes akin to Scrum, they likely won’t get any effective research done.
In the course of leading research teams in two product-focused organisations, first at Jukedeck and now at Bytedance, I’ve developed a set of principles aimed at ensuring that our teams work on research that has the maximum possible impact on product, spanning every stage of the process from planning to implementation to evaluation. These are set out below.
1. Ensure research has a clear, agreed-on product aim.
Put simply, you shouldn’t be doing a research project if you haven’t identified a clear product benefit that would arise from the work going as planned.
Of course, the work may not go as planned: it may not be successful, or it may lead to an outcome that is valuable for totally different reasons to those you predicted (from the microwave oven to penicillin, there have been many famous instances of huge innovations being made entirely by accident, by-products of research that were never intended). But, if you don’t set out a clear product aim from the start, most research projects will be wasted time. The research graveyard is littered with projects in which the question ‘What impact will this make?’ was asked too late.
2. Involve product in setting research objectives.
As outlined in (1), research objectives are product objectives — and researchers shouldn’t be setting product objectives. They have significant expertise in their area of research, but they don’t have the understanding of users’ needs, overarching view of the company’s strategy and general product expertise that a good product manager will have.
The product team should be involved in setting research objectives right from the start, as well as revisiting priorities as research progresses and results start to be delivered or work is delayed — and, importantly, the decisions regarding what a research team’s objectives should be should ultimately be taken by the product team. If the product team doesn’t make these decisions, any product aims worked towards by the research team as part of (1) are likely to be the wrong aims to focus on — aims that are unimportant to the business.
Involving the product team in setting research objectives means that both teams can play to their strengths: the product team can ensure that the business’ most pressing priorities are being addressed, and the research team can bring ideas to the table and inform the product team what is and is not realistic.
3. Estimate how long the research will take.
It is almost impossible to estimate accurately how long research projects will take — but it’s important to come up with an estimate nonetheless.
There are two principal reasons for this: first, if you don’t, different people will make different assumptions about when results are likely to be delivered, and this will lead to problems when these differences of opinion emerge later in the project; and second, estimating in this way means you can better prioritise different research initiatives. (If Initiative A is estimated at one year’s work and Initiative B, having only half the expected impact of Initiative A, is estimated at one month’s work, the product team may rightly want to choose Initiative B.)
The estimate the research team comes up with should be conveyed to the product team during the planning process. It won’t be accurate, but it will help make sure everyone is on the same page about the proposed scope of the project, and will let the product team weigh up alternatives effectively.
4. Let researchers choose their own methods.
An additional benefit of having the product team ultimately determine research objectives is that this can operate as a clear line at which their involvement in the planning process ends. Where researchers don’t have the information required to make decisions about product direction, product managers don’t have the information required to make decisions about the research avenues most worthy of exploring in pursuit of the agreed-on objectives.
So, once the research goals have been set as part of (1) and (2), the researchers should be empowered to make their own decisions about how to achieve those objectives. What is important to the company is the effect achieved, not the method by which it is done. At Jukedeck, for example, the product objective was to build a system that could generate music. The method our research team used to achieve this was training recurrent neural networks — but this was their choice, and it was ultimately no concern of the product team, as long as it performed the functions the company needed it to perform.
Letting researchers choose their own methods not only ensures their expertise is fully utilised, without being hindered by impositions from the product team — it also gives researchers a real sense of ownership of research projects, which means they’re more invested in the outcome and generally helps make sure everyone is pulling in the same direction.
5. Have regular reviews of progress, and consider whether to continue the work at each review.
It is important to have regular, scheduled reviews in each research project, at which the researcher presents their progress and outlines any key developments (including, importantly, any negative results or delays). The product team should be present if the update affects the product or timeline of the project, and the outcome of the review should be a decision as to whether or not to continue the project.
There are three main reasons for these regular reviews: first, it gives researchers short-term goals to work towards, which would otherwise be missing from what can be longer-term research projects; second, it can ensure successful research projects are drawn to a close and integrated into the product as early as possible, as it provides an opportunity for the product team to spot that a product objective has been met, where a researcher may be inclined to continue the project in order to make further improvements; and third, it means priorities can be reevaluated as more is known about the likely success and timeframe of the research project and new priorities emerge from elsewhere in the organisation.
Of course, the hope should be that a research project is completed, and ideally this is the standard. But regular reviews will ensure that you don’t continue with a research project when, for whatever reason, it’s no longer a priority.
It’s worth noting in particular that there is nothing inherently wrong with the estimated timeline of a research project being revised as part of these reviews. A change of timeline is a good thing: it shows you’ve increased your knowledge of the domain. Any changes to the estimated timeline should simply be explained by the researcher(s) working on the project. It shouldn’t automatically lead to a project being scrapped: but it should automatically lead to a project being reassessed. This way, regular reviews provide an opportunity to consider whether any changes to the estimated timeline should affect the decision of whether or not to continue with the project.
As to how often these reviews should happen, every two weeks seems a good rule of thumb, as it provides enough time for significant progress to be made (a week is too short) but not so much time that a project can continue long after it should have been scrapped (a month is too long); but the exact schedule will depend on the project. And you should always be prepared to hold ad hoc reviews if there are big, unexpected updates that merit immediate discussion.
6. Reward good choices and speed to conclusions, not successful projects.
As discussed earlier, letting researchers choose their own research methods gives them ownership of the projects they work on — and ownership begets accountability, a useful tool in the arsenal of a company seeking to empower its employees to use their initiative to deliver maximum-impact changes. But what should researchers be accountable for, given that research can so often not work out as planned?
Researchers should be judged on the choices they make given the information available to them at the time, and how rapidly they reach conclusions, rather than on whether or not the output of their research works as intended. This ensures researchers are motivated to consider what the most effective solution to a product problem is, and discourages (i) focusing on pet projects that may not be the most effective solution and (ii) reframing negative results as positive, which researchers may be tempted to do if they are judged simply on the success of the project, and which would lead to bad decisions around product and next steps being made.
In short, it’s OK for a research project not to work out, as long as the decisions involved in the project were made for the right reasons, the reasons for the project’s failure are explained, and the failure is highlighted as early as possible so that next steps can be considered.
Getting research right is about making sure everyone is working towards a common goal, and using their own specific skills to do so. It is about aligning incentives: making sure the incentives of the research team are consistent with those of the organisation as a whole. And it is about understanding the ways in which product and research should be linked, and the ways in which they should be separated. Research is a very different beast from product, but it benefits hugely both from being driven by product needs and from using frameworks that borrow from established product methodologies but know where research and product diverge. Get the team and the framework right, and the research will follow.
