AI Plumbing: complex, risky and expensive?

Deploying machine intelligence enabled products and services in the real world is quite a new discipline for most of us. It has different challenges to those of conventional software deployment, and best practice is still emerging.

We call the wide range of technologies, tooling and processes involved ‘AI Plumbing’ and like actual plumbing, AI Plumbing can be complex, risky and expensive.

Here we describe the challenges, and propose some areas where Digital Catapult could help.

Complex

Systems engineering and DevOps expertise may not be readily available in machine intelligence projects, especially in startups. Moreover, systems engineering and DevOps in data-driven projects (sometimes called ‘MLOps’) present several additional hurdles above and beyond those presented by conventional software engineering. For instance:

• Large assets (such as models and datasets) can make continuous integration and deployment slow and cumbersome if not handled properly.
• Quantifying acceptable performance of a machine learning model can lead to grey areas typically not found when testing code.
• Relating a particular version of the code and parameters to model output is essential if one wants to make systematic and informed changes.
• Data leaks, feedback loops and misconfigurations can produce dangerously optimistic evaluations of performance which can go unnoticed.
• Quality Assurance (QA) is fundamentally hard when deploying machine learning solutions. Traditionally, when software is written exclusively by software developers, one can rationalise about the behaviour of functions. Machine learning forces us to move away from unit testing, the testing of discrete logical parts, and towards black box testing where internal structure is unknown to the tester. QA is as integral to deployment as it is to development.

Risky

The landscape is fast changing. Whilst there are some mature offerings, chances are that there are several new initiatives at any one time each seeking to solve the same part of the AI Plumbing stack. Choosing one can be risky: they’re new (so they’re not proven and they might be hard to discover); they might be poorly documented and supported; and their failure (or acquisition) might make them suddenly unavailable. Startups tell us that they are ‘reinventing the wheel’, developing tools for common tasks, which diverts resources from their main mission. For vendors, the environment is challenging because the range of products that one needs to interoperate with is evolving, and because developers switch allegiance with rapidity when a new, more performant competitor solution appears.

Expensive

Tweet from Andrew Chen of A16Z 3rd July 2018

AI Plumbing choices can have long-term consequences for cost, but while it is important to budget for the plumbing in machine intelligence projects, it’s pretty difficult to do. This is true for every technology layer, starting with the difficulty of evaluating appropriateness and ROI of hardware choices (see Digital Catapult’s Machines for Machine Intelligence research report) and building from there. As the costs associated with machine intelligence innovation spiral, the ability to make informed technology choices and to reduce ‘technical debt’ [1] could be the difference between life and death for startups, and between a project with a clear ROI and one that gets canned in larger organisations. Discovering and evaluating tools costs time; becoming proficient in using them, and then perhaps having to switch to a new one, even more so.

So, where can Digital Catapult help?

• First, we have started to map the AI Plumbing landscape in order to assist discovery, research and planning tasks. We describe the landscape here, and invite feedback.
• We’ve encountered many of the challenges described above in our own machine intelligence projects and we’re lucky to have DevOps and systems specialists in-house that work with our machine intelligence team to solve them. We’ll be sharing some of these in upcoming blogs, along with useful tips and tools for overcoming them.
• We will host meetups and workshops to illuminate some of the AI Plumbing technologies and strategies.
• We believe that companies that have considered the ethical implications of the products and services they develop, and who monitor, manage, and communicate effectively about them, will have a competitive advantage against those who do not. Our ethics framework will help companies to implement responsible AI in practice.

Digital Catapult is keen to collaborate with organisations that help solve AI Plumbing challenges and assist startups with access to tools and expertise (an evolution of its Machine Intelligence Garage programme that focuses on computation and expertise in AI). Please get in touch at hello@migarage.ai

[1] E. Breck, S. Cai, E. Nielsen, M. Salib, and D. Sculley, “The ML test score: A rubric for ML production readiness and technical debt reduction,” 2017 IEEE Int. Conf. Big Data (Big Data), pp. 1123–1132, 2017

This content was originally published on migarage.ai.