Why ModelOps Is an Enterprise-Level Capability Under the CIO’s Accountability
Article in partnership with ModelOp
We all have heard of DevOps and the transformation it has brought to application delivery in enterprises. However, there is another equally powerful emerging capability, ModelOps, which can give the organization of the so-called Enterprise AI a competitive advantage too.
Introduction
Today, artificial intelligence applications are widespread and perceptible even in everyday life. Consider, for example, the ability of a car to brake automatically or not to exit the lane during a curve or, also, to the health sector, where the AI is able to monitor and report anomalous trends as well as helping to streamline processes to free up doctors’ schedules and therefore reduce costs. Let’s think for example of a computer able to identify the famous person we have in mind, asking ourselves no more than 10 questions, or how AI is constantly being adopted by many organizations in the finance and business sectors to restructure companies, improve earnings and experiences, reduce risks, and increase opportunities for the financial engines of our modern economy.
These are already existing technologies of more or less advanced artificial intelligence, which are changing the way we live, work and study. When the machine learns independently, thanks to these and through experience, managing to improve its performance over time by providing more answers or more functions, you will have machine learning.
However, AI and machine learning models introduce new risks into model operationalization (post-development). Many model operations processes are manual or managed using home-grown solutions that constantly need to be updated as new technologies, tools and governance requirements are introduced.
For this reason, AI must move beyond ad-hoc, small-scale experimentation conducted by data scientists, to automated pipelines that can deliver inference results directly into enterprise environments.
You might also like to read Model Operations for Secure and Reliable AI
What is ModelOps?
Meanwhile, organizations, to remain competitive, are trying to develop AI and machine learning models at an increasing pace to gain new insights, continue their digital transformation and reimagine their business, as we have stated the risk of failure and wasting resources, still remains very high.
It is also well known that organizations have been using models of all kinds for decades to support business. However, AI and machine learning models introduce new risks related to the operationalization of the model itself. Such models, which are probabilistic and non-deterministic by default, need to be monitored and governed (sometimes retuned, retrained, or even replaced) to effectively support the development of AI initiatives at scale.
ModelOps, as a key strategic capability to deploy, monitor and govern model life cycle in production, across the entire enterprise AI, is a novel framework and platform for the end-to-end lifecycle management of artificial intelligence applications artefacts. To ensure scalability and governance, the most innovative enterprises are using ModelOps as an enabling technology, which is a crucial step in the convergence of various AI artefacts, platforms and solutions.
How does ModelOps fit with existing CIO organizations?
The best way to understand the role played by ModelOps in enterprise AI is to start by defining the components of ModelOps capability and determine how to organize and manage the effort. In order for enterprise AI to have an impact across the entire organization, the business must have a unified strategy that allows each part of the business, the flexibility to utilize the AI tools that are best suited to their needs while ensuring that the results of those tools flow to the business quickly, reliably, in full compliance, and with full accountability.
You might also like to read Don’t Let Tooling and Management Approaches Stifle Your AI Innovation
Most enterprises have organizational setups that can negatively impact conflict among and between lines of business, IT, DevOps, DataOps, and risk/compliance teams.
By using ModelOps, conflicts between departments and operational frictions are minimized, while ensuring that the resulting outputs of all implemented AI tools flow into the business quickly, reliably, and with full compliance and accountability.
ModelOps is not a role, it is a set of capabilities that can reside in many different parts of an organization. There are many BUs in the enterprise where ModelOps could be housed:
- In an operations group.
- In a modelling group, such as a predictive analytics team.
- In a product management function, such as for a marketing team.
- In an IT group that has expertise in infrastructure.
Nevertheless, for many enterprises, a key question remains regarding what part of the organization has responsibility, and how much funding, for ModelOps.
When an organization is at the beginning of its transformation, they place ModelOps on an enterprise data science team under the Chief Analytics Officer. This would make sense when the goal was to understand whether data science could generate any value for an organization or not. However, as organizations mature in their transformation, they will typically start to see value from their efforts in developing capability around data science models. Thus, since data science models contribute to their future competitiveness, if not even their existence, it is time to review organizational responsibility for ModelOps.
Since ModelOps is a holistic approach that provides an organizing framework for the processes of model development and its operations, it should be handled by the CIO organization. In fact, if the data science teams released the models to the operational teams — DataOps, DevOps, ITOps, compliance, and so on -, they are lacking the right skills to meet the requirements for operationalization of the models, so they would try to put the models in production as they are used to doing it with conventional software. This approach obviously does not work, as the teams are not able to organize themselves in cross-functional workgroups, with defined processes and associated tools to manage the operationalization processes.
You might also like to read Unlocking the Value of AI in Business Applications with ModelOps
This disorganization of roles and responsibilities negatively impacts artificial intelligence initiatives, inexorably leading to long delays and low success rates. In addition, It is also the cause of the rise of AI-driven shadow IT, where individual BUs take over the operation of the model.
ModelOps, therefore, must be under the responsibility of the CIOs. In fact, the models in production must be monitored and governed 24/7, whatever the sector or field of application. The following infographic illustrates how ModelOps fits into an existing CIO organization. Additionally, it introduces two new roles essential to the success of this new capability: AI Architect and ModelOps Engineer.
According to ModelOps for Dummies’ author Stu Baley, Enterprise AI Team in the new organization will include the following:
Descriptions from the ModelOps Essential Guide
Data Scientists
Design, build, and test models that enable substantial business value for a particular use case Design, build and test experiments intended to improve models and to support models in the business.
Enterprise AI role:
- Deliver analytic models that create value when deployed
in business applications. - Develop tests for determining the efficacy of deployed
models and driving improvements.
ModelOps requirements:
- Freedom to use the most effective model development tool for each application and use case.
- Minimal overhead and constraints on model development are imposed by model deployment considerations.
- Automated packaging and delivery of models from development workbench to production environment.
- Real-time visibility to the performance of deployed models.
ITOps
Maintains the core infrastructure and associated services needed to support models running in business applications.
Enterprise AI role:
- Provide highly available, secure infrastructure to operate performant models in enterprise applications at scale, 24x7.
- Deliver performance metrics for deployed models and applications.
- Plan and execute infrastructure evolution to support AI technologies.
ModelOps requirements:
- Centralized catalogue of all models, model runtime requirements, lineage and operational history.
- Standardized models that can be deployed, monitored and controlled in a consistent manner, in any infrastructure (on-prem, cloud, hybrid) regardless of the tools used to create them.
- Automated alerts regarding model performance and behaviour.
- Automated processes and approvals for deployment, testing, refresh, and monitoring.
- Real-time visibility to the performance of deployed models.
DevOps
Develops business applications and ensures that they remain operational and available.
Enterprise AI role:
- Create and implement agile processes and resources for incorporating models into business applications and deploying applications into the infrastructure.
- Report on business metrics for applications.
ModelOps benefits:
- Abstractions that enable models of different types and from different tools to be incorporated into applications and connected with any type of data pipeline (triggered, batch, real-time) with standardized monitoring and control functions.
- Consistent visibility for every model regardless of source.
- Automated model management and refresh without the need for ad-hoc, custom scripts.
DataOps
Develops and tests data pipelines for input to a model and receives the output of model inferences.
Enterprise AI role:
- Provide relevant, curated data from multiple sources to all models.
ModelOps requirements:
- Complete, centralized model inventory with all relevant metadata regarding data sources accessed.
- Ability to quickly detect and correct changes in model behaviour.
- Visibility and controls to ensure and verify that data protection guidelines for security and privacy are enforced.
CIO
Provides appropriate technologies to support the business strategy. Monitors, plans, and budgets for infrastructure capacity. Ensures the efficiency of teams responsible for data security, network utilization, application development.
Enterprise AI role:
- Enterprise AI strategy, operational infrastructure, processes and oversight.
ModelOps requirements:
- Centralized, consistent tooling that allows automation of the processes that implement the MLC with the flexibility to accommodate each model and application use case.
- Real-time and trended visibility to the performance of all constituencies (Data Science, DevOps, DataOps, ITOps, Compliance) against their respective KPIs.
- Elimination of ad-hoc, fragmented approaches to ModelOps that compromise the business value and expose the organization to risk.
- Visibility and controls to prevent the spread of “shadow AI” as AI capabilities become available to “citizen data scientists”.
- Data to drive realistic planning for infrastructure and staff needs and to protect existing infrastructure.
LOB Analyst
Defines the requirements for leveraging AI/ML models that can provide uplift for their business processes. Monitors KPIs and reports to LOB executive.
Enterprise AI role:
- Work with the data science team to translate business requirements into model KPIs.
- Continuously monitor and report on model performance in terms of business KPIs.
- Ensure that variances from KPIs are discovered and resolved quickly.
ModelOps requirements:
- Clear articulation and automation of processes that implement the MLC.
- Real-time and trended visibility to model performance, especially at the business-KPI level.
- Ability to drive and track automated response processes when alerts show performance diverging from KPIs.
- Elimination of ad-hoc, fragmented approaches to ModelOps that compromise business value and expose the organization to risk.
- Visibility and controls to prevent the spread of “shadow AI” as AI capabilities become available to “citizen data scientists”.
- Data to drive realistic planning for infrastructure and staff needs and to protect existing infrastructure.
LOB Executive
Defines and executes strategies to leverage advanced analytics to increase the profitability of their business line.
Enterprise AI role:
- Take maximum advantage of the business value available from use of AI technologies.
- Ensure that business KPIs drive the development, deployment and operation of models.
- Mitigate technical and organizational bottlenecks that limit the ability to derive value from AI investments.
ModelOps benefits:
- Conformance with corporate AI standards with minimum restrictions on data scientists and developers to create value.
- MLC automation to minimize the use of staff time and delays and ensure compliance with internal and external approvals and regulations.
- Automated model updates (retraining) to limit decay and maintain maximum business value output.
- Continuous visibility to model performance in terms of business-level KPIs.
- Automated processes and approvals for deployment, testing, refresh, and monitoring.
- Real-time visibility to the performance of deployed models.
Compliance Manager
Ensures that models, their data and consuming applications meet internal and regulatory standards for efficacy, privacy, fairness, and other parameters.
Enterprise AI role:
- Minimize and mitigate risks and exposures created by the use of AI.
- Monitor and audit models and data for conformance with regulations and corporate standards.
- Communicate regulatory compliance to LOB, Data Science, DataOps and other teams to ensure models and data pipelines are developed to be conformant.
ModelOps requirements:
- Centralized model catalogue with comprehensive metadata.
- Automated reporting on the complete history and lineage of any model to satisfy audit requirements.
- Alerts that trigger when models depart from compliance specifications or process approvals.
AI Architect (new role)
Designs the cross-functional processes that implement a responsive and effective MLC for all models, business units and functional organizations. Architects the tooling used to automate the MLC and integrate it with the enterprise IT stack. Serves as the primary, hands-on interface between all groups that implement and benefit from Enterprise AI.
Enterprise AI role:
- Design the technical standards, process templates and KPIs for the end-to-end MLC for all models used in all applications.
- Design and oversee implementation and operation of the enterprise ModelOps platform.
- Report to all constituencies and executive management on the status and progress of the organization’s enterprise AI journey.
ModelOps requirements:
- Tools that enable design and implementation of processes that define the MLC for each type of model in each application.
- Standardization of model deployment, monitoring and control capabilities.
- Freedom to empower Data Science teams to utilize the model creation tools best suited to their unique requirements.
- Process automation that facilitates coordination of all constituencies with minimum friction.
- Ability to mitigate the impacts of Shadow AI as AI capabilities become embedded in common enterprise applications.
- Reporting that demonstrates the value of AI to the business
ModelOps Engineer (new role)
Serves as the primary, hands-on interface between all groups that implement and benefit from Enterprise AI.
Enterprise AI role:
- Assists in the integration of ModelOps capability into enterprise IT systems and business applications.
- Monitors model performance and ensures rapid response to any issues that arise.
ModelOps requirements:
- Ability to reflect technical and business KPIs in an automated ModelOps framework.
- Ability to enforce technical standards that conform models from any source to common standards for deployment, monitoring and governance, without imposing undue restrictions or requirements on Data Science, DataOps, DevOps, ITOps or compliance teams.
- Ability to implement processes that fit the unique requirements for consistent adherence to the SLA’s of the business through mutually agreed-upon rules, triggers, and/or alerts.
- Ability to quickly spot, resolve and report on issues with any part of the MLC process (technical, business, approval, reporting, etc.).
The benefits of ModelOps
With the new organization in place under the CIO, with one process owner in place for the entire model lifecycle, the company is now able to review and adapt its ModelOps processes. The new organization, therefore, leverages the transition from a functional structure in silos and associated processes to a more streamlined and cross-functional structure and set of processes. This will be more effective and efficient for the goal of operationalizing models and fostering the development of AI initiatives at scale.
To wrap things up
Innovation in business requires enterprises to use ModelOps at the core of their AI strategy, because it helps converge different AI artefacts, platforms, and solutions, ensuring scalability and governance.
Enterprise AI affects many areas of business and requires a unified strategy that allows each area of the business the ability to make use of the tools best suited to their needs, while ensuring that the outputs flow into the business efficiently, reliably, and in compliance with all regulations.
Having decided that data science models are not only valuable but essential to their future competitive standing, or even existence, to develop a common view of any aspect of any AI operationalization processes across the different departments and BUs including the processes and associated tools, it is essential that accountability for ModelOps must be under the CIO’s organization.
To learn more about ModelOps, I advise you to read ModelOps for Dummies.
The book explains an enterprise-level operational discipline called ModelOps, which has emerged as a key to unlocking the power of AI. ModelOps is a new enterprise capability that integrates and automates all the business, technical, and compliance stakeholders and activities across the organization to ensure that AI models — and all types of models — are governed, operated efficiently, and monitored continuously, producing value while remaining compliant.
You might like to read also How ModelOps Helps You Execute Your AI Strategy
Follow me for daily updates on Technology and Innovation
