For years organizations have used MBOs, KPIs, and OKRs to set and achieve goals and measure progress in a data-driven way.
While all of those are similar, there are subtle but important differences between OKRs and standard KPIs. You can read about theoretical differences here. But the best way to learn about OKRs is to dive right in. Here I wanted to share some examples of practical OKRs that can be set for AI programs.
Before we go ahead, a warning:
Yes, there is a chance that the unpredictable and iterative nature of AI programs will not lend itself to traditional methods of defining OKRs.
Now with that warning, let’s dive in:
Invariably when we start talking about AI programs, the first focus goes to the core data platform. So let’s just look at OKRs that are suitable for that area:
Data OKRs
Objective 1: Significantly increase Data IQ across the company.
KR1: Define a Data IQ a training and certification process that allows employees to learn about our data assets and about the tools and processes they can use to leverage those.
KR2: Target 5% of employees in non-data function, to have passed Data IQ certification.
Objective 2: Enable shared data governance, quality & consistency
KR1: Establish a Data Catalog and define a catalog owner in each org, with full access to the catalog tools.
KR2: Build a central repository of runnable Data quality checks and enable all orgs to contribute to those tests via an open-source type of model.
KR3: Publish data Catalog and Data quality dashboard at company level and each org level.
KR4: Target 5 orgs outside of non-data function to have an active process for contribution and review of Data quality and governance.
Here you can see additional such examples and discussion of OKRs for such function.
Now let’s look at some AI program OKRs that leverage the advances in core data function.
OKRs for individual AI projects
Objective: Improve AI objective (say loss rate, 7-day engagement, etc.) by 1%
Key result: Brainstorm and prioritize 3 hypotheses for testing towards at least a 1% increase.
Key result: Conduct tests and publish results and findings.
Key result: Based on the results either deploy the winning test into production or define a revised and focused testing plan for next quarter.
As you can see the AI OKRs are iterative. There is an assumed risk that the first 3 sets of hypotheses may not result in ‘success’, with the traditional definition of success. This may not work in some cultures. For such scenarios, it is better to break the objectives into AI Model development objectives and business objectives.
Here is an example:
Objective: Either increase the accuracy of the existing model or improve execution speed, by continuous improvements in modeling technology.
KR1: Explore new techniques / new features to increase Model accuracy in ‘test’.
KR2: Explore an alternative model that uses less ‘taxing’ data in production.
KR3: Conduct an A/B test to validate if model accuracy increase is sustainable in production.
KR4: Conduct an A/B test to validate if model execution speed can be improved?
KR5: Leverage the tests to increase Model accuracy by 10%
KR6: Alternatively: Leverage the tests to reduce execution speed by 10%
OKRs for AI Programs
For company-wide AI programs, the OKRs tend to be more ‘platform’ centric. Here is an example:
Objective: Scale Large Time Series classification capability to be leveraged across multiple functions
KR1: Expose large time-series visualization as a service such that any employee could upload / point dataset to visualize.
KR2: Target at least 10 users across 3 different functions to be active users in a given quarter.
KR3: Identify 2 new projects across the 10 use cases to test ‘predicting future value on time series.
KR4: Leverage auto-encoders, LSTM type of recently piloted techniques to build prediction models for the 2 techniques.
KR5: Target at least 1 prediction model to be leveraged for ongoing prediction of time series data.
As you will notice there is a level of ambiguity across OKRs for both AI projects and prgrams. Hence it is essential to define OKRs in such a way that are culturally appropriate and they set up the team for success, provided the team puts in the needed efforts!
Please do share if you have additional / better examples.
