How we get accepted on the NeurIPS2022 DMML workshop (Challenges in deploying and monitoring ML systems), including our paper!

To answer the question shortly, it was just one inspirational moment before I went to bed.

I think we all have that moment when it is just out of no reason that something flies into our mindset, and then right after that, we start to make it concrete step by step. Ultimately, we get something beautiful and surprisingly good, even out of our expectations.

Back in September 2022, I heard so many different voices from the teams and counterparts that we spent so much time trying to get familiar with one deployment technique, and later things changed, and we had to fit into another due to this or that reason. Especially as the primary test engineers and CV engineers, who later need to give instructions to the whole team or even document step-by-step how to deploy our AI to the global systems, I feel so under stress facing unstable techniques.

Well, you probably would say that this ML systems Deployment world is adopting so fast that we have to fit ourselves into this speed.
However, I do see this as a common challenge after watching and trying different CI/CD techniques from scratch and diving into the research topics myself on ICML2021 and ICML 2020; in the academic world, people are also interested in those topics, and at least what I am facing and our team facing is looks like a common problem for most of the company on Machine learning world, at least on OEM industry.

We are facing a situation: ML and CV models and Data are viral topics; However when it comes to Deployment and making it into production. How and which CI/CD pipeline to pick, and how can we make my ML system, and my platform more stable and cost-efficient? Those “last-mile” problems are less well-known. Or maybe not that fancy enough?!

Partly reasons I can come up with are:

1. Technical reason. System design or Pipeline design is only interviewed on a higher level, as it is already higher bared, unlike coding or data structures that are reasonably accessible to the public.
2. Marketing reason. Many Service providers, such as PaaS-based like Opensift, only provide the advantages and what they can do. Still, for the limits or downsides of this CI/CD technique, only after we use or integrate it into our system it turns out not good enough or causes systematical damage, and we need to replace it. That’s why thousands of blogs of different official tutor documents are available.
3. Regulation reason. When it comes to a machine learning system, this does not mean the only (.pt or .pb) saved model file but integrated with a whole front end and back end and fully functional system, in which machine learning plays a significant role in it. The inference results will ultimately affect human decisions. So from this point, it is something other than a group or lab can easily afford, but more like industry scale level, such as those FANNG tech companies used ML for adv prediction systems or fintech anomaly inspections, OEM’s quality monitoring systems. Moreover, from the company’s interest and regulation point of view, which CI/CD techniques they used, why they used them and based on what evaluation metrics they chose, are impossible for us to access.

All this leads to less material for us to comprehensively evaluate what techniques fit for us on Deployment on machine learning system fields. Different people used partly included metrics, or maybe marketing over advertised advantages, or stakeholders only focused on a single factor such as cost to decide on deployment pipeline techniques, thus failing to build a stabilized production ML system later.

NeurIPS’22 DMML Workshop

What are the NeurIPS DMML workshop’s highlights?

This workshop brings together all the ML Fans interested in ML systems, especially the Deployment and monitoring field, and has organized over three interactions since its escalation; it has been held at ICML 2020 Workshop and ICML 2021 Workshop.

This year the objective of this workshop was to bring together people from different communities with a common interest in the Deployment of Machine Learning Systems.

With the dramatic rise of companies dedicated to providing Machine Learning software-as-a-service tools, Machine Learning has become a tool for solving real-world problems that are increasingly more accessible in many industrial and social sectors. The growth in the number of deployments also grows the number of known challenges and hurdles that practitioners face along the deployment process to ensure the continual delivery of good performance from deployed Machine Learning systems. Such challenges can lie in adopting ML algorithms to concrete use cases, discovery and quality of data, maintenance of production ML systems, and ethics.

What is the NeurIPS DMML workshop’s focus this year?

Starting in march 2022, it began accepting papers from these two main domains with a limited length of 4 ( but no limits on appendix). So it is challenging to articulate such a complex problem or idea into four pages. However, we made it!

The main topics.

1. Open problems in ML systems
2. Security and privacy in ML systems.

In particular, with a clear focus on :

• MLOps for deployed ML systems;
• Providing privacy and security on ML Systems;
• Ethics, fairness, trust, and transparency around deploying ML systems;
• Valuable tools and programming languages for deploying ML systems;
• Deploying reinforcement learning in ML systems;
• Continual learning, CI/CD pipelines for ML systems;
• Data challenges for deployed ML systems;
• Open problems in ML systems; challenges and blockers to the successful Deployment of ML systems.

In the end, only 17 papers were accepted across the industry. Each paper was at least peer-reviewed by 2 to 3 senior reviewers and also given corresponding comments or feedback; the author’s afflictions are pretty diverse; it has someone from famous FANNG tech and academic institutions as well.

For the detailed papers link, you can read from this link: https://sites.google.com/view/dmmlsys-neurips2022/accepted-papers

What is the central concept of our paper?

In our paper, Continual learning on deployment pipelines for Machine Learning Systems

Here is the takeaway:
We first analyzed and compared what state-of-the-art Deployment is. Starting from the static methods to semi-automata and, in the end, fully automated CI/CD methods. In the appendix, you will get to know the architectures of each technique in detailed UML views, which will help you to first to get some idea of how to adopt them in your project and what kind of skills is needed and includes some commonly used debug tricks and command book.

What’s more interesting, in each of these CI / CD deployments, we also documented the real cons when applying them; this is one of the benefits of our paper because usually all of that info can hardly be accessible from its official techniques tutorial page because of commercial reasons.

At the end of our paper, we innovatively proposed comprehensive evaluation criteria by completing a series of sample questions. It is possible to have a very clear and more comprehensive understanding of the Deployment of the model from the beginning of the system design, which can fundamentally prevent selecting the wrong deployment technology due to different criteria and coordinates chosen by different people, and subsequently repeatedly revise and make mistakes. Inspired by the stock market, this five-star vector chart allows people to visually see a deployment technology’s strengths and weaknesses, thereby allowing them to select the right deployment technology for the project. The scoring criteria can be automatically assigned with weighting factors, and we encourage people from different backgrounds to participate in the discussion based on this chart.

How difficult is it to join the conference community as an independent researcher or from an underprivileged or underrepresented group?

As an underrepresented member and independent Researcher, who just conducted this paper purely out of interest and hobbies, and also listened to several talks in NeurIPS’22, such as a workshop for Women and LGBT in NeurIPS’22, Workshop for DEI group in ICLR’21 and ICLR’22. I felt the same struggle and went through all the difficulties you might think about. Therefore I am here to give some tips for what I heard and learned.

1. Top-tier conferences DO have a very high standard bar, but blinded Reviews also have a high bias. Getting rejected doesn’t mean your idea or your paper is not good; I heard some authors just used the same content and resubmitted to NeurIPS this year and just changed the title, and it got accepted! So last year Neurips I submitted a paper on another topic and got scores from 3 to 9! ( 3.4.5.6.7.8). One guy even increased from 8 to 9 in the end, but it still didn’t change my paper’s fate, and it got rejected unfortunately, but I do think that is a great work, and I do feel proud of it. And this year, our NeurIPS’22 workshop paper — Continual learning on deployment pipelines for Machine Learning Systems — also received three reviewers' feedback. So, all in all, getting accepted just means good work + good luck. Getting rejected doesn’t mean lousy work; try another time or submit it to the workshop track! The main things are advertising, networking, making people use your work and get inspiration from it, or even creating real value from it. ( to your good news, this work is interesting by one of our clients in Zürich, and we will use this framework to create tangible benefits for the project!)

Last year NeurIPS’21 submission on paper: Heterogeneous Graph Pooling Neural Network for Sleep Stage Classification

1. @ML_collective and @CohereForAI, those NGO communities greatly supported independent Researchers or underprivileged or underrepresented groups. So if you already have any ideas but need more collaborators, you can easily find someone from their slack channel. And you will receive free GPU or even some advice on which Conference you should go to, or even waive your fee for Conference participants as it is sometimes not a small amount and could be a big blocker for Independent Researchers.
2. The workshop is the seed for the fantastic full paper. This statement is not from me; it is what I heard from NeurIPS22 invited talk by Isabelle Guyon on The Data-Centric Era: How ML is Becoming an Experimental Science, she also pointed out that by joining the workshop or presenting your workshop, you will pick up some good idea but maybe not fully well established or achieved work, and conduct it improve it, and in the end make it become a great work. She made one example by Edison. She said Edison is not the purely original inventor of light, Edison likes to try somebody’s failed experiments, he always looks at what’s failed in others' work, and he corrects it and in the end, makes it work! So if you have any good ideas, don’t be afraid to write them down or be shy to submit them to the workshop track.

So feel free to look at the details of our paper at NeurIPS’22, and you could add any thoughts! Always welcome new ideas!

Continual learning on deployment pipelines for Machine Learning Systems