Roles and responsibilities in AI projects

Artificial Intelligence is everywhere… It’s so all-pervasive that we get fooled redefining data science and machine learning to accomodate this trendy precursor. It’s been a long walk until we’ve found how to make data-{analysts | engineers | scientists} work together. This short text tries to reflect my standpoint on the issue.

I’ve been asked to figure out what roles are needed to succesfully run AI projects in an organization. My best guess is that this is a generic request and the result must fulfill the ‘one-size-fits-all’ assumption. It is difficult to specify the different roles in any type of AI project, since every AI project is different (online scoring models, chatbots, APIs, batch classifiers, you-name-it, etc…). There’re a so much expectation on it that many think that by simply come up with the “AI” prefix, magic will happen!

My take on this is, the more mature your organization is on ML and data culture, the less effort in trying to build AI products. But if your access to data and the impact of ML in real production systems is still very low, talking about AI is premature and misleading. So, let’s continue under the assumption that we’re somewhere in the middle of this maturity scale.

Let’s start by dividing the problem of defining roles and responsibilities in two dimensions:

  • AI Product Lifecycle: Or who is responsible for what on every phase of the ideation-experimentation-creation-deployment-operation of the product.
  • AI Product Components: Or who is responsible for making every component to work in harmony with the rest, in a coherent way.

Product lifecycle

I’ve seen in this post, that the traditional phases of an AI lifecycle should be: ideation, data preparation, prototyping & testing, productisation and overall system architecture.

Let’s consider, for a moment, that instead of an ideation phase, products are born as a combination of inception (or pure internal ideation) and direct request from business users. I love calling this initial phase: “t-1”, but let’s use a more formal nomenclature here, and call it “feasibility”.

The remaining stages of the lifecycle are OK to me, except for the fact that they do not perfectly reflect the iterative and cyclical nature of this type of projects. So, for the sake of simplicity, let’s assume that each stage is iterative, and that the entire lifecycle sequence can jump at any time to any of the previous phases. This idea can (somehow) be depicted as follows:

My mental diagram on how stages can connect to each other in real life.

Let’s digg a bit more on what is expected to happen at each of these stages:

Feasibility

The main actor in this stage are the sponsors, who meet with the team to discuss about a potential product. Sponsors can be either: who’s asking for the product or who is pushing to include it in the portfolio. Their responsibility is to run this phase to know more about whether the ideation is doable, how much it costs and what will be the product capable of.

The AI part here (and now) is very important. A normal dialog in the “traditional” data science world stops when all the parties involved discover that a model will be able to produce a regression/classification that solves a given issue with the existing data. However, under the AI assumption, requirements should also explicitly set goals on re-training, feedback, monitoring, alerting and a lot of “what-if” scenarios to produce an “AI” that robustly solves a problem.

The outcome of this stage is a “BUY/MAKE” decision, with (at least) costs and KPIs to assess product behavior.

Data Preparation

The first step at every data project is always to extract and prepare data to be consumed by data scientists. But it’s a mistake to consider this stage a manual ad-hoc process. Data processes must be prepared for any situation on the data feed (anomalies detection, imputation, reconstruction, etc.), and result in an idempotent, predictable and scalable operation.

Main actors therefore are data analysts and data engineers. They will ensure that data is consistently fed into the right infrastructure to enable the product.

Prototyping and Testing

Most likely in parallel with the above stage, while a ready-for-production feed is built, the prototyping and testing phase can start.

Prototyping refers to the engineering of the envisioned product, where all the components are outlined and assembled together to test load assumptions, scalability and interfacing with data in/out. An architectural analysis might be necessary, depending on the complexity of what we’re building.

Testing, on the other hand, is much more important and refers to the well-known iterative process of addressing the modeling part, where DL and ML are used.

Main actors at this stage, seem to be Data Scientists and Data Engineers. They should work together exchanging information about how both parts will finally assemble together (data in, model and data out). But, where is the AI? We should mentally think of AI as responsible of ensuring that everything works as expected. My mental picture of this is this:

Where to place the AI in a so-far typical ML product?

So, it is important to track down this process along time to know how is the model behaving (i.e.: is it being consistent, or does it need to be re-trained?). It is also important to deal with undesirable data situations with input or output. From this, AI might be responsible to fix or reject data in, or to decide whether an output is acceptable under certain circumstances.

Key role here is then played by Data Scientists and AI Engineers.

Productization

Now, that we know how to build our product, productization deals with the correct development of all the pieces to correctly handle data dependencies, connectivity, parametrization, signaling, exception handling, interfacing, telemetry, logging, etc…

ML and AI Models should be handed over the Data Engineering team to properly connect them to the rest of the system, the same way the people who develops the engine of a Ferrari hand it over to the next stage in the assembly line to build the “car”.

Key role is played by Data engineers and DevOps.

Architecting (Deploying)

Final stage is about putting everything in production. That means: security, environment, orchestration, scheduling, operation, monitoring and support. This is the work of DevOps, and I will not enter too much into details here, as it follows the same rules of any other deployable product.

Who does what?

We have outlined who drives each stage in a AI lifecycle… but what components are typically present in an AI product? I must say that this question is driving me crazy and my answer at this moment is that I cannot generalize a typical AI product.

However, I can say what is the alleged responsibility of an AI Engineer, as described by DataRobot here. Their definition of AI engineer describes unicorns, though I tend to agree (from my experience) with the threefold role: Know about infrastructure, work inside the machine learning process and decide on deployment readiness and monitoring.

From this definition, AI engineers participate along most of the stages of the lifecycle. Namely: feasibility → prototyping & testing → productization. It’s fair to say that data preparation and deployment are the only stages where they are not so relevant.

Now that we know what is expected from AI Engineers it’s probably easier to define the role of everybody else:

  • Data Analysts. Includes data architects and modelers. They know about data structure and best ways to store and retrieve information. Work with security in ensuring the application of their policies. Define compression, encoding, labeling schemes for fields and headers, and also version the data model. They must also work in building ELT/ETLs that are predictable, scalable and monitored. Participate in feasibility and data preparation phases, but also play a role in productization defining data output streams, if applicable.
  • Data Engineers. Includes system developers, backend and frontend developers, and primarily big data developers. Able to design pipelines handling large data volumes and scale out (distributed) architectures. Focused on designing robust distributed data pipelines with profound knowledge of machine learning technologies to correctly manage their specificities.
  • Data Scientists. Experts in statistical learning, machine learning, deep learning or any other field aiming at building computer models able to perform a complex task without the need to explicitly program it. Among the tasks suitable are regression, classification, clustering or representation, among others.
  • AI Engineers/Scientist. Able to solve complex machine learning problems, they also deal with natural language and knowledge representation and inference techniques. Build autonomous AI systems and take decisions on platform and deployment scenarios. They can also be referred to as machine learning engineers.