ML-Mania: Progress in Review As a New Decade Dawns

With 2019 forming the end as well as the beginning of a new decade, I have been reviewing the machine learning investments we at La Famiglia have made over the past years. As the turn of the year provides an opportunity for reflection, I have summarised general trends, new frontiers and limitations, as well as three methodologies for framing the commercial potential of machine learning in the wild.

I) General Trends

A year ago, KDNuggets summarised ML in 2018 as “the year of refinement” with “low hanging fruits having been plucked” and media pushing “overblown fears” of the omnipotence of ML. At the same time, a “reckoning of applied machine learning” took place “as there are just some limits to what we can do with pattern matching alone”. Well, 2019 was something of an elegant albeit non-climatic continuation of these findings: clear progress without a drastic step change. While we have seen a few key advancements in research, such as BERT in NLP (end of 2018, more to follow later), this year was marked by the absence of manifold game-changing breakthroughs and instead presented a phase of maturity and refinement. It seems that ML is now growing up.

Macro trends: At a macro level, same as every year: the cost for sensors, compute and networking continues to decrease, building the foundation for ML’s steady commercial progress.

AI Chips: A range of companies have achieved key advancements in developing novel chips specialised in AI operations, from architectures optimised for data centres to edge devices, making 2019 a hot year for semiconductors (see Groq announcing the world’s first architecture capable of 1,000,000,000,000,000 operations per second on a single chip/ 250 teraFLOPS (compared to Nvidia x Amazon at 100 teraFLOPS), Intel potentially snapping up Habana Labs for >$1Bn , Graphcore & Microsoft announcing an AI cloud partnership, and Cerebras claiming to have the world’s fastest supercomputer due to a 400,000 core processor).

Self-supervised learning: In continuation of advancements in ML research at the end of 2018, 2019 saw the “quiet revolution of semi-supervised learning” , with new progress in semi-supervised learning (alternatively called self-supervised learning) promising to remove the cumbersome bottleneck of labelled data. Through the development of proxy tasks, self-supervised learning enables systems to learn without explicit supervision by learning relevant contextual information instead, thus not requiring pre-existing labeled data.

Why is that a big deal? Many companies still stuffer from barriers to automation due to a lack of appropriate data. There are three common barriers to ML automation, starting with a lack of digital data (1st barrier to automation), structured data (2nd barrier to automation), and labelled data (3rd barrier to automation and often the most cumbersome one since scaling labelling efforts remains difficult). Hence, the advent of systems that can be fed on structured digital data alone allures.

Due to a range of methodological improvements, self-supervised learning is now hitting its stride. With research teams publishing new methodologies (such as Google’s BERT and Allen Institute’s Elmo) self-supervised learning has had the biggest effect on Natural Language Processing so far, achieving the same performance as supervised learning with a 10x lower amount of training data. It was rolled out on Google in mid-2019, one of the largest changes to search ever, helping to provide more nuance and context in search results.

In May 2019, Yann LeCun stated accordingly: “The next AI revolution will not be supervised or purely reinforced. The future is self-supervised learning with massive amounts of data and very large networks.”

At the same time, since self-supervised learning also enables scaling with lower amounts of labelled data, together with ever decreasing total costs to run ML in the wild, the next decade promises ML at the edge, ML at scale and ML with less (labelled) data, getting us closer to the paradigm of “ML everywhere” than we might currently anticipate.

Exploring sensory frontiers: Ian Goodfellow’s introduction of GANs (Generative adversarial networks — the idea that neural networks contest and thereby improve each other) in 2014 provided a breakthrough in photorealistic image synthesis. In 2018, Deepmind provided more advancement through their work on BigGANs, which essentially scale up GAN models to enhance performance (for a history of GANs, check out this post). This year was marked by further progress of GANs and Variational Auto Encoders (VAE) for audio, image and video synthesis, focusing on first use cases such as fashion model creation for e-commerce sites (e.g. DataGrid), though commercial scale remains low.

Overall, through recent advancements, such as machines learning to smell (Google research) and image face generation now being indistinguishable from real photos (see Ian Goodfellow’s comparison), the next decade will further raise questions about the meaning of privacy and authenticity in a sensory world that can be manipulated at the click of a button, while also highlighting unprecedented potential for cost-saving and automation across creative industries.

Federated learning: Heralded as a potential breakthrough for machine learning in the previous year, federated learning has disappointed this drastic expectation, yet is also showing promising early application. Federated learning enables ML without direct access to training data — data remains in its location and only computed abstractions are shared with a central model. While providing a method to soothe privacy concerns and more easily process large amounts of data (see e.g. recent news of Nvidia’s use of the technology within hospitals), the commercial reality of federated ML is still marked by key limitations, such as edge device performance and coordination, which is aggravated by a lack of standardised infrastructure and resources on client/edge site, as well as back-tracing of local data. Nevertheless, federated learning is already in production in certain environments that allow access and control over the entire infrastructure, such as Google Keyboard, enabling accurate text predictions and autocorrect without data being shared back to Google (how’s that for win-win?).

AutoML: When it comes to AutoML, no big breakthroughs either. AutoML enables computers to automatically generate and test neural networks, thus creating better results than humans would. If leveraged at scale, this could solve a key human bottleneck, and at much lower cost (think about the beauty of ML model generation 24/7 with no breaks needed). Despite commercial releases by Google, the performance of AutoML still lags behind expectations of tech automation hopefuls. While research results are promising, showing that certain tasks can be automated, the key to strong ML performance still lies in the human element (i.e. hyperparameter tuning, architectural and loss function design). “Humans have lots of knowledge that I don’t think AutoML will be able to figure out,” said Quoc Le, lead researcher and creator of AutoML earlier this year.

Autonomous driving: There was a hype — and this year it evaporated definitively. Or as a founder in this space put it to me on a more positive note: “2019 has been a wonderful phase of increased realism — so far the kids were playing, now we’ve grown up”. As investment continues to be high, consolidation pressures among autonomous vehicle companies are picking up and new partnerships continue to be announced regularly (e.g. most recently BMW & Daimler). Across the industry, the reality of autonomous driving in the wild lags behind initial expectations: missed launch dates have become the norm and many players struggle to ramp up driven miles. While Waymo remains far ahead of the curve and Tesla works hard to show the world new form factors as well as its superior data collection and algorithmic capabilities, putting out a bold vision of enabling L5 without Lidar sensors, in 2019 the world agreed that L5 autonomous cars in mega cities remain far in the future. Still, we anticipate that other use cases in more structured and controlled environments (trucking lanes, warehouses, etc.) will provide promising business cases in the next few years.

Tooling: Great news for productivity junkies: On the tooling front, steady progress has been made, with a range of tools now making the work of engineers and data scientists more productive (see here for an overview).

Talent: As sought after as ever, ML talent remains a rare good that is worth fighting for, though Europe still is nowhere close to winning the battle. In 2019, a report from the Centre for Data Innovation concluded that the United States is leading in AI, followed by China and then the EU. While the US and China each lead in several categories (talent, development, hardware, research vs. adoption and data, respectively), the EU does not achieve any category leadership (oy gevalt!). Already frequently lamented by politicians and business professionals, I believe the next decade will escalate tensions in the race to AI supremacy, with Europe’s wish to be at the forefront remaining as vulnerable as ever. However, with the topic of ethics and governance becoming more important, as shown by several research initiatives across European universities (e.g. Facebook & TU Munich, Stephen Schwarzman’s historical $188M donation to the University of Oxford; both for the purpose of funding research about ethics in AI) the scales might soon tip more into Europe’s favour again.

II) Method I of III: Categorising the Potential of Automation

To help provide structure in discussing the potential of data and ML for automation, delineating areas in which humans vs. machines excel is key. While a lot of the public discourse focuses on AI totalitarianism, predicting drastic takeovers from machines and robots in all walks of life, the reality of how automation is embedded into our work and private lives is more nuanced. Depending on how well processes are defined, the availability of structured and labelled data, as well as ability to provide production-ready products, form important prerequisites for machine automation potential (note that automation can be driven by ML, but also the orchestration of digital workflows alone, which often provides much lower hanging fruits).

As reported by MMC Ventures in 2019, 1600 AI startups call Europe home. Most of them focus on the verticals of health, fintech, and media, as well as the business functions of marketing, BI & analytics. When assessing the ability of machines to take an ever bigger role in our daily lives, it is worth considering the strength of human capabilities versus the readiness of automation.

As the graph below depicts, the potential for automation can be categorised alongside a theoretical productive balance between humans and machines, with further productivity pathways delineating human as well as machine prevalence. Based on three such theoretical productivity lines, four fields emerge:

• The Field of Tooling: In this area, human capabilities remain far superior to machines. Humans use machines as a mere tools that have little to no influence on the quality of the output. In this field, machine impact is low, and human impact high. Examples would be software tools for writing (i.e. a best-seller will not be decided by using Word or Pages), or creating a musical masterpiece.
• The Field of Assistance: In this area, the human is assisted by a machine. Machine impact is medium because the human in the loop still crucial to the final output (e.g. managing a sales relationship). Here, machine impact is low to medium and human input is important. Examples include voice and text assistants, chatbots, L3 autonomous driving. We count portfolio companies like Affinity (ML-enabled CRM) and Impira (ML-enabled asset management system) into this category.
• The Field of Augmentation: In this field, the capabilities of machines exceed those of humans, enabling humans to do things much more effectively. Humans are still needed as a final control layer, but not crucial to overall quality of the output, which is mainly driven by machine performance. Here, machine impact is medium to high, whereas human impact is low but still necessary for seamless operations or near-perfect results. Examples include product recommendations, state of the art OCR functionality, predictive maintenance, L4 autonomy, and production optimisation platforms. We count companies like Fox Robotics (automated forklifts), Omnius (streamlined workflows for insurances), Alcemy (cement optimisation platform), and BigFinite (manufacturing optimisation for the process industry) into this category.
• The Field of Automation: In this field, machines have achieved prevalence: Human input is near obsolete — human work can be fully automated, with machines often achieving results far superior to human capabilities. In this field, machine impact is high and human impact low to nonexistent. Examples include fraud and cancer detection, image/video-based quality control, as well as specialised manufacturing processes. In our portfolio, we count companies like Osaro (automated picking and assembly) and CloudNC (autonomous machining) into this category.

At La Famiglia, we believe big businesses can be built across all four categories, albeit scalability concerns exist for the tooling space, as well as potential price pressure in the field of automation for low-complexity tasks, as has already happened across API-based business models.

In 2019, the fields of assistance and enablement got us most excited, since this is where great products make a difference to human workflows (the famous 10x), providing clever yet often simple ways for both to work together seamlessly. Moreover, the two areas cannot be fully separated, with many platforms showing characteristics of both areas (e.g. an ML-enabled CRM like Affinity providing augmentation through streamlined relationship information, yet still requiring the human in the loop for key relationship management). Looking at the next year, we will continue our hunt for companies tackling complex automation through best-in-class tech teams, as well as augmentation and assistance through companies that understand how to create 10x impact through streamlined workflows and superior analytical insights.

III) Method II of III: Dress to Impress But Better Low-ball for Success: Anticipating Human Expectation

To evaluate why some products have underwhelmed rather than overwhelmed on the automation expectation scale, examining human expectation is key. Primed by darwinistic survival instinct, negative events impact humans more strongly than positive events of the same magnitude (the negative feeling of losing $1 weighs more strongly than the happy feeling of gaining $1).

As the scale of measuring automation success is not one-dimensional, but often impacted by at least two aspects (overall quality & overall cost) this Negativity Bias has resulted in certain products remaining below expectation levels of the past decade.

The status quo being the point of reference for assessments of new automation products, machine translation, despite strong progress not being perfect yet, has found wide adoption since the expectation level for many languages was no or only very expensive translation — making even patchy translation a strong improvement.

On the other hand, interactions with voice assistants and chatbots continue to underwhelm customers. Rarely able to move beyond simple interactions, when comparing human-to-machine with human-to-human conversation, a clear gap to the status quo remains, driven by a lack of meaningful interactivity.

To examine ML’s past and future potential, we hence always ask ourselves whether ML-driven automation features of a product will delight and raise the bar, rather than underwhelm and annoy in comparison to the status quo, despite potentially promising other optimisation criteria, e.g. potential to decrease overall cost.

IV) Method III of III: Understanding What Moves the Needle of Your Business — Where Is the Money?

While I felt that AI-bullshit bingo has decreased in 2019, we still saw a range of companies this year that utilised AI and ML as a buzzword to jazz up their deck, rather than a technology that can drive meaningful value for their customers, i.e. help solve their problems. When determining if and how ML can add meaningful value to businesses, we find it helpful to separate three categories.

• Scale: Will the business benefit from driving higher scale through automation (100x more items processed, no need to rest)? E.g. generation of additional business where operational capacity is a key bottleneck (e.g. in labour-constrained markets); alleviating cost pressure in stagnating or declining businesses
• Quality: Can ML achieve better results than humans could? E.g. driving better quality in high-value and high-risk environments, such as cybersecurity or health. Important note: often the combination of ML and humans achieve best quality (i.e. humans providing context to pre-analysed ML results)
• Time: Often, ML can achieve results much faster than humans by quickly sorting through thousands of possible options. Does moving closer to real-time functionality actually add value to clients though? E.g. ML enabling servicing clients faster, thus generating higher customer happiness and driving retention

Good founders will have thought about the underlying business need of their clients, knowing exactly why automation features make sense and drive value.

V) Outlook

What will the 2020s hold? The 2010s have been the decade of unmet expectations on both ends of the innovation bullishness scale, disappointing harsh critics and strong enthusiasts alike. Coming after decade that moved from cries of “all hail the almighty algorithm” and computers beating humans in a range of complex tasks to an awakenings across the board, including chatbots still being shitty and the far away reality of fully autonomous vehicles, I believe the 2020s will move us into a world in which human-machine interaction becomes the de-facto status quo everywhere. Pioneers will achieve significant levels of advanced automation in select areas (e.g. warehousing), and even laggards will steadily exploit assistance and augmentation opportunities.

All in all, while the robots aren’t coming for us (yet), I look back at the past decade with a sense of deep serenity over the fact that novels won’t be written by computers any time soon. This serenity is met by excitement in the wake of a new age of automation and the benefits this would entail if executed correctly, yet also staggering unease in light of many societal challenges that will arise.

Products built by empathetic and purpose-driven founders will likely provide some answers — and even more new questions. So onto more golden ML years!

(Recommended Reads for ML research / investment overviews in 2019:

• https://speakerdeck.com/mmcventures/the-state-of-ai-2019-divergence
• https://www.stateof.ai/
• https://mattturck.com/data2019 )

Thanks to great ML startups https://luminovo.ai/ & https://recogni.com for your valuable feedback on writing this review!