Research QUEST Framework

Research projects, particularly those involving machine learning and AI, differ from traditional projects in that they have an extreme level of uncertainty and a large number of failed experimental results. It’s hard to make accurate planning and estimation. It’s hard to promise results. It’s hard to make decisions about continuations when deadlines are constantly being pushed back.

Research QUEST is an agile management framework and a set of best practices that gives the upper hand to project managers who are responsible for the money and success of a research project.

There are five main stages of the Research QUEST: Questions, Uncertainties, Exploration, Solution, Testing, and finally the encounter with the Boss.

Questions

Research projects are usually expensive. These projects always have sponsors who pay for your research and, based on your results, they make certain important decisions. Your team is expected to create a report to prepare a one-time decision, a dashboard for recurring human decisions, or an AI algorithm which will make decisions without human participation.

Here is the Root Question of your Quest: what decision do they want to make (Decision-to-be-made) and what missing information needs to be collected, prepared, transformed, and presented to make this decision self-evident.

Don’t hesitate to ask questions to understand the details of your Mission. Don’t worry if some questions remain open so far. This brings you to the next phase.

Topics: What does your Quest Giver want from you? — MISSION Canvas — Decision-to-be-Made — Ideal Dashboard for human decisions — AI is a decision-making machine — Dimensionality: How does AI Recognize? — Missing Information: How does AI Predict? — Overabundance: How does AI Choose? — Mistakes: How does AI Learn? — All you need is … Data.

Uncertainties

It’s ok to have uncertainties in a research project. Each uncertainty is your separate research item. Organize open questions and emerging ambiguities in a list, your R&D backlog.

At this point, the sponsor will want you to have a plan and budget for the project. All tasks can be either research (high-uncertainty discovery) or development (low-uncertainty delivery). It is easier to estimate development tasks because you are doing something you’ve already done before. Research problems will have a much wider range of estimates. Some problems may appear to be really strong ones, where the real volume of work can grow tenfold. So the estimate is always subject to change.

Your research will be like moving through a labyrinth, and the main path along which you start moving might turn out to be a dead end. So create a map of uncertainties that’s as detailed as possible, explore, and be ready to change your path.

Topics: Research is a Labyrinth — You need a MAP — Uncertainty Matrix — A Billion-Dollar Fortress Siege — ESCUDO: Currency for estimates — 3D Backlog

Exploration

When there is a clear scope of research work, it’s time to organize a systematic exploration. At this stage, we are still working with unknown unknowns and there is no need to rush with trying to prove your hypotheses. Better to focus on less formal assumptions and insights, expansion of the search space coverage, and the systematic retrieval of unexpected findings.

Findings can be of different types: data sets, insights, alternatives, or pieces of working code. We need them as trial components in the next stages to design and test different variants of a solution. It makes sense to structure even seemingly insignificant findings, or rather side findings. Maybe you’ll need them on a bypass route.

Topics: Exploration RAIDs — Massive RAIDs — Museum — Customer Discovery — Exploratory Data Analysis — Long Tails in AI — Feature Engineering

Solution

Based on the collected, labeled, and analyzed datasets and developed pieces of code, you can now design different variants of your solutions. Basically, you can change two things: the architecture (the set of interconnected components), and the quality of the individual components. You may now design experiments and try to find a solution with acceptable, affordable, and achievable accuracy.

Beware, your research team may spend too much time trying to make one specific component work. Therefore, it’s your job to be with the team, actively manage priorities, simplify tasks where necessary in order to keep on time, and work with a sponsor to form the right expectations.

Topics: CRAFT: technology of making hypotheses — Product Genomics — Customer Discovery and Problem Hypotheses — Problem-Solution Hypotheses — Zero to Hero Prioritization — Product-Market Fit Hypotheses — Machine Learning Hypotheses — How to Design Experiments? — How to Reduce the Number of Experiments?

Testing

Testing different solutions can take much longer than you planned, since you don’t know which one will work, and the number of combinations can be overwhelming. It’s always a big question whether to keep trying with the current architecture or to stop and build an alternative construct from scratch. There is no definite answer, but there are some important signals that can help you make such decisions.

When you run low on time, and the solution is still half-baked, you need to be with your research team and understand deeply what they are doing. Some problems during the testing phase arose at the very beginning of the project when you negotiated and formed expectations and obligations from the sponsor. A common problem is that you were not provided with data on time and in the required amount. An equally common situation is organizational restrictions that you were promised to eliminate, but in fact, they remain and interfere with the project. Finally, the problem may be in the data quality itself.

To pass through wrecking you need to have a system of early warnings. You cannot prevent single failures, research itself is a try-and-fail game, but an experienced hero can lose the battle to win the war.

Topics: AI Development Process — AHEAD Kanban board — Don’t get eaten by DRAGONS — Crash-Boom-Bang! Demolition Time! — TATTOO — RESPAWN-meeting

Encounter with the Boss

Research projects cost and matter a lot for sponsors, so try to arrange the presentation of the results as a significant event. The main question that worries the sponsor: “Did what was conceived work out, and if not, how much more money is needed to get it right?”

Even if the results are not fully achieved, the sponsor wants to see that you are approaching the project competently. You need to show that you care about ROI, you have had several alternatives, you’ve made a reasonable choice, systematically worked out the options, and now you can present the results elegantly and concisely.

If you’ve done it right, you’ll get a new QUEST :)

Topics: Hard-won TREASURE — APPLAUSE