Innovation Pilot? How to Increase the Chances of Proof-of-Concept Success

I spoke with a good friend this week on how things were going with product testing of a new care management platform. Like many early stage efforts, the technology has been prototyped and is being actively tested in live-settings for further evaluation and refinement. She works in a large technology prime that typically refers to such projects as being in the ‘dog food’ stage, but this story would be the same for a startup or early stage company.

Health IT innovations inevitably go through multiple rounds of focused testing in order to clearly demonstrate the value to a potential customer. Altogether, my friend’s ‘proof-of-concept’ was well organized and was on-track to be completed on-time. There was, however, more than a bit of anxiety when she mentioned the unveiling of the forthcoming closeout results. The final client presentation at the end of the study would generate many useful metrics, but there was ambiguity around exactly how they would be interpreted or received by the client.

For this case, my friend could have greatly benefited from a proof-of-concept strategy like the one that I weave into all my early stage engagements. In each deployment, a set of SUCCESS METRICS needs to be jointly developed and agreed upon towards the start of the project.

Innovation success isn’t guaranteed, but everyone should know upfront what it might look like when they see it.

I’m fairly familiar with these types of proof-of-concept engagements, as they are necessary to show value to all parties. In an ideal case, the pilot leads to a commercial engagement. Pilots slow down commercialization, but I’ve yet to find one that didn’t yield useful insights given the time spent, if success is clearly defined in the early stages.

On identifying success, here’s what I recommend for all proof-of-concept studies and pilots:

• Communicate early on that the proof-of-concept is a collaborative exercise.
  Too often, one or more of the parties see the proof-of-concept as a one-sided responsibility. It should never be viewed as just a demonstration, as it sets the wrong tone for subsequent engagement. Ensure that there is mutual understanding of the value of success metrics and that their creation will be a gating event to many steps within the proof-of-concept.
• Build success metrics processes into your agreements.
  Almost every enterprise engagement will include a statement of work that outlines the broad activities of the project. In many of these agreements, developer and vendor responsibilities are outlined. Be sure to identify the generation of success metrics as an event that takes place early on as a required activity. Be sure to weave subsequent activities, based on their creation, into your statement of work timeline.
• After project start, circulate a draft set of categories that can be measured throughout the project.
  These are the areas that may be measured and will serve as a basis of discussion. Certainly, this will include traditional categories on performance, value, and ROI. Be sure to also include perception surveys for all parties, including satisfaction, beliefs, opinions. It’s a shame to realize in hindsight that the qualitative thoughts of all the stakeholders should have been systematically tracked from the outset.
• Engage early on to further refine and agree upon the success metrics.
  Set up time to openly discuss what is important to all parties in a proof-of-concept. Challenge everyone to answer: “This would be viewed as a success if X occurred for these metrics.” Alternately, “In my opinion, the risks associated with this project significantly decrease when I see X”
• All success metrics should have a target range and referential context.
  Since success metrics are agreed upon upfront, it is hard to anticipate the twists and turns of a deployment. Openly discuss what is the range of values that would represent success, and what is the likely context of that scenario.
• Structure your project and data collection effort to cross-walk to your success metrics.
  Oftentimes the raw data collected needs to be carefully analyzed and converted to the success metrics. Identify these relationships and build these functions early.
• Re-visit your success metrics throughout the proof of concept.
  Tracking success metrics is tantamount to an executive dashboard for the engagement. While the complete set of metrics is undoubtedly being continuously monitored, the success metrics collapse all your data into a single set that conveys overall project trajectory.
• Project closeout and next step transition will be guided by the success metrics.
  Adhering to this methodology reduces the variability of response at the conclusion of the proof-of-concept. It is also a useful segue to next steps in engagement and potential commercialization.

Proof-of-concept studies are expensive but sometimes necessary engagements — especially for early stage health technology products. They are crucial for innovations in demonstrating value and informing refinement.

The notion of buying promising early-stage health IT may be a source of discomfort for health plans and healthcare providers, but there are ways to mitigate risk and maximize benefit from cutting-edge innovation.

My advice to any organization embarking on one of these engagements is to build a process for jointly developing, agreeing upon, and evaluating success metrics. This should be done in an open and transparent fashion with the client, and should begin early in the process.

***

What do you think? Let me know your thoughts on successful proof-of-concept projects. Contact me at: wyu@healthcareinnovation.us

***

Wil Yu heads innovation strategy, business and partnership development for Lumiata. His work focuses on predictive data analytics, care management transformation, and related emerging health technologies. Previously at the U.S. Department of Health and Human Services, he led nationwide healthcare innovation efforts. Twitter: @hlthinnovation