Introducing Seldon Deploy
When we released Seldon’s open-source machine learning platform in 2015, machine learning was a niche topic in the business world. But leveraging what is broadly described as AI is now at the top of the strategic agenda for organisations of all sizes across industries.
Seldon’s mission is to help people predict and shape the future with machine learning. We’re firm believers that machine learning is the enabling technology driving this AI renaissance, as it helps people solve problems of all shapes and sizes and enriches all of our lives.
I’m excited to start lifting the lid on Seldon Deploy — a new enterprise product that will be entering closed beta this summer. Seldon Deploy helps data science teams put machine learning models into production. It’s designed to streamline the data science workflow, with audit trails, advanced experiments, continuous integration and deployment, rolling updates, scaling, model explanations, and more — all served through a delicious UI. Seldon Deploy will perfectly complement our open-source machine learning platform.
Before diving deeper, let me give you some context on why we’re building this product.
Listening to the market
Last year Seldon was part the world’s #1 fintech accelerator, Barclays Techstars. This experience helped us gain vast amounts of product feedback from a broad range of stakeholders, and sign customers in the banking and financial services space. We also gained a deeper understanding of the machine learning use cases that large enterprises are prioritising and challenges they face when embarking on an AI project.
To shape and validate Seldon’s roadmap, we’ve been listening to our community of data scientists and customers about their priorities and the challenges that organisations face. In a survey from August 2016 we learned that stakeholders were prioritising on average 4.5 use cases, the most popular include customer segmentation (48%), content recommendation (36%) and churn prediction (34%). We also saw a long tail of niche use cases such as fraud detection, loan optimisation and medical imaging.
One of our biggest decisions last year was whether to focus on some of these vertical use cases or continue to build a foundational technology that operates on a generalised and horizontal basis. Most machine learning startups focus on serving a specific use case, and it’s a validated playbook that investors understand and buy into. According to UK AI market research published by David Kelnar from MMC Ventures, 84% of startups focus on AI for a function or sector. Companies can build a defensible moat by combining deep domain expertise and the data network effect that improves the service itself as more data is fed into it — in terms of accuracy and performance of the model.
52% of stakeholders want to build their own machine learning models in house. And a critical challenge that organisations face today is moving from an R&D environment into production — only a couple of percent of models made their way into production last year. Organisations care about performance, scalability and cost effectiveness, and there’s growing demand for the control that in-house deployments provide.
Deployment is an enormous problem that remains unsolved. This is why Seldon will remain in the 16% of AI companies building foundational technologies that can be used across industries and use cases. We have narrowed our product focus on deployment, operating across industries and enjoying success in banking and financial services.
The rise of open-source ML frameworks
Another significant trend is the increase in the quality and availability of machine learning and deep learning model building tools and frameworks, such as TensorFlow, Keras and new entrants like PyTorch and Caffe 2. At Seldon, we use TensorFlow in our client projects and are co-organisers and hosts of the TensorFlow London meet-up — a community of over 1200 data scientists and ML enthusiasts, endorsed and supported by Google.
Model-building tools and frameworks are undergoing a period of commoditization and rapid evolution thanks to support from tech giants and community traction. Data science is like agile software development: effective organisations don’t enforce specific programming languages, teams have the freedom and autonomy to use the best tools for the job and connect with the outputs from other teams via APIs and microservices. In data science projects it doesn’t make sense to enforce a specific model-building tool or service. This is why Seldon has always been framework agnostic. You can use it to deploy models built in your preferred model-building tool — it even supports models built with commercial tools and services where the model is exportable.
Seldon Deploy
It quickly became clear to the Seldon team that we had to build a new enterprise product, not a major new release of our current platform. I would go further and say solving deployment means creating an entirely new category in machine learning that sits on the intersection of data science and ops.
Let’s dive deeper into some of the key features:
Projects
Each project within an organisation can include multiple models, experiments, deployments to multiple clusters. Each Project must be assigned a leader and collaborators. Through a single interface you can manage multiple projects.
Deployment Workflows
One of the bottlenecks that organisation face is that members of data science teams map their specific roles onto a toolchain. Models often get built by data scientists operating as individuals or in disconnected groups with documentation spread across a variety of services. This causes confusion and delays because the people responsible for releasing new models have to pull together documentation about the model-building process and what has changed.
Seldon Deploy will provide a fully integrated deployment workflow via a delicious new UI that supports the following team members:
- Data Scientists — loads runtime models into containers as microservice components and runs sandbox tests from historical or live data feeds.
- Data Engineers — manages and deploys models to one or more clusters (e.g. production, staging, regions and departments).
- Data Manager — business stakeholders responsible for defining the input/output contract for each project and business KPIs, with access to real-time analysis.
Out of the above roles, Seldon Deploy focuses mostly on productising the function of a data engineer, the role for which there is the largest skills shortage in industry.
We’ve taken a view on how the above data science team should map to the features, but we appreciate all teams are different and have made the roles-based permission system customisable on a granular level.
Audit Trails
One of the useful outcomes of having an integrated workflow is that every action and approval in the system forms part of an audit trail. Resolving every output prediction to a unique snapshot of a model and knowing what journey that model went on to production will become more than a nice-to-have next year.
Model Explanations
In May 2018 the new General Data Protection Regulation (GDPR) will give consumers a legal “right to explanation” from organisations that use algorithmic decision making.
And as more important decisions are being made and automated on the basis of machine learning models, organisations are seeking to understand why models give a certain output. This is a tough challenge considering there are many types of models with varying degrees of interpretability. E.g. It’s easy to traverse the decision tree generated by a random forest algorithm, but the connections between the nodes and layers of a neural network model are beyond human comprehension.
Seldon has been implementing state-of-the-art methods such as LIME based on recent research in this field. The techniques that we are integrating treat the model as a black box and stress test the features to see what impact they have on the output. A higher dimensionality of features makes the explanations more computationally expensive. So, one of the engineering challenges we’re tackling is to enable model explanations as efficiently and accurately as possible considering your application and business requirements.
Advanced Experiments
About a year ago our open-source project adopted Kubernetes, which revolutionised how our machine learning infrastructure and models are packaged, maintained and deployed. This streamlined experiments that test the performance and combine the results into ensembles at runtime.
Seldon Deploy allows data engineers to combine multiple models with an experiment such as a simple A/B test or a dynamic multi-armed bandit to create what we call Predictive Units.
Seldon Core
We’re becoming a multi-product company, so we have renamed our open-source machine learning platform: it’s now called Seldon Core.
I’ve spoken at length about our experience and rationale for open-sourcing Seldon and what we built in our first year, and we are committed to ensuring that Seldon Core continues to be released under a permissive open-source license — currently Apache 2.0.
Seldon Core provides a fully integrated data science stack that runs within a Kubernetes Cluster. It’s used by a global community of data scientists across industry and academia to deploy machine learning and deep learning models into production on-premise or in the cloud (e.g. GCP, AWS, Azure). Seldon Core wraps models in microservices inside containers, which makes it framework agnostic — it supports models built with TensorFlow, Keras, PyTorch, Vowpal Wabbit, XGBoost, Gensim and any other model-building tool.
The current Seldon Core deployment principles remain the same, using containers and microservices to deploy models, so feel free to get started today.
Join our closed beta
Do you want to gain early access to Seldon Deploy? We’re looking for active data science teams to provide feedback to help shape the product before launch.
Apply here to gain early access to the Seldon Deploy Closed Beta.
