A Year in Review

What Has Happened at SAP ML Research in 2017

The last year has seen a great deal of new trends and developments in the machine learning (ML) field. This rate of progression will definitely increase throughout the new year with more and more players joining the AI game of as the topic is gaining sweeping prominence. Moreover, it can be seen as the logical next step following the emergence and integration of ML services into the business sphere and into our daily lives.

A more in-depth look back at the year 2017 reveals a number of breakthroughs and findings. Among the most notable ones was the Google DeepMind AI program AlphaGo Zero, which outperformed its predecessor AlphaGo by 100 to 0 games. Prior to this event, AlphaGo, a machine trained with moves of historical games as well as self-play, had been praised for the defeat of human world champion Lee Sedol in the ancient Chinese game Go. The triumph of AlphaGo Zero in turn was achieved by simply learning the game of Go from scratch without any further instructions. Within only 40 days, AlphaGo Zero achieved to create winning strategies that go beyond human capabilities, just by playing Go against itself. Its victory uncovers that a self-learning system can acquire actually knowledge without the need of prior information.

One more game changing discovery facilitated by machine learning was NASA’s finding of another eight-planet solar system, specifically by leveraging the power of deep learning. Besides these major advances and trends, in recent time the ML community has also started to dwell not only on the technological progress, but also on its side effects such as the impact on society, biases, ethics as well as privacy.

As machine learning continues gaining traction in the business world, the SAP Leonardo Machine Learning portfolio will continue growing alongside the rising need for intelligent applications. However, although the technology can be widely applied in the business context, its development requires a strong partner and co-innovation network with continuous research effort.

Our SAP ML Research team spent the last year working together with diverse research collaborators to solve different ML problems with practical applications across multiple domains. Our research has focused on a wide scope of ML research areas in the last year and we also sponsored and participated in top ML conferences like ICLR, CVPR, ICCV or recently NIPS. Some of our research partners put forward their sponsored works at these conferences and different SAP teams presented their papers at the most recent ACL and NIPS.

Sponsored Research Contributions

• At NIPS in Long Beach, our research partners from University of Amsterdam/ University of Edinburgh presented their work Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols. In this paper, Serhii Gavrylov and Ivan Titov propose an approach in which machines develop their own language within a referential game setting. Within this context, machines are stimulated to learn a new artificial language (e.g. sequences of discrete symbols) that is efficient. They showcased the work in the context of learning a language used to communicate the contents of images between a sender and a receiver. Particularly interesting is the notion that the developed language implements some kind of hierarchical structure meaning the word order matters in the code.

Another area of increased interest is modeling of non-Euclidean structured problems. These are for instance problems where the neighborhood is not defined on a fixed grid or where data lies on complex manifolds with implications such that shortest distance between two neighboring points is not a necessarily a straight line. However, in the business context these approaches still largely constitute uncharted territory albeit many potential applications, with more breakthroughs to be expected in the next years.

• Our research collaboration with Max Welling and Thomas Kipf from the University of Amsterdam, follows up on this interesting topic. Their seminal work on Semi-Supervised Classification with Graph Convolutional Networks presented at last year’s ICLR uses a variant of a convolutional neural network classifier that allows operating directly on graphs. The presented model scales linearly in the number of graph edges and learns hidden layer representations that encode both local graph structure and features of nodes.Their recent work together with Rianne van den Berg on Graph Convolutional Matrix Completion is an application of graph convolutional networks to link prediction and entity classification, as well as matrix completion for recommender systems. Their approach allows many real-world problems to be cast into graph problems, which naturally would not fit into generic deep learning frameworks. In general, this research area promises to have a wide array of applications ranging from drug discovery to recommender systems e.g. in the field of e-commerce.

• Our research partners from Ludwig-Maximilian-University (LMU) Munich explore the potential of artificial intelligence in the healthcare sector. In the last year, we worked together with Prof. Dr. Christian Wachinger and his team at the Laboratory for Artificial Intelligence in Medical Imaging on a new project aiming at the early detection of Alzheimer’s disease. The team’s approach is based on the prediction of the brain age using MRI scans. A comparison between the predicted and the patient’s chronological age then allows for drawing conclusions related to anomalies in the brain. This research collaboration shows the strong potential of machine learning to identify dementia or other neurodegenerative diseases in their early stage, eventually long before symptoms might become visible. In the last year, the work has also brought about two papers: DeepNAT: Deep Convolutional Neural Network for Segmenting Neuroanatomy and A Multi-Armed Bandit to Smartly Select a Training Set from Big Medical Data (MICCAI 2017). The research project only marks the beginning of what will be possible for the application of machine learning in the future of healthcare and leaves room for further research projects in this area.

SAP ML Research Contributions

• Recently at NIPS, we presented our paper on Differentially Private Federated Learning: A Client Level Perspective in the workshop for Machine Learning on the Phone and other Consumer Devices. In this work, we follow up on the recent advance of federated learning in the domain of privacy protection. It deals with the typical scenario where ML practitioners and scientists have to work on sensitive data. In this regard, our team explores the possibility of training ML models under privacy constraints, rendering the sharing of data unnecessary as well as facilitating learning even in settings where the number of participant clients is not large. The findings arising from our research work have high application potential for hospitals, companies or institutions that want to use generalized prediction models but need to follow strong privacy guidelines. Following the approach, these entities could use a generalized model learned by many peer contributors without having the requirement of a centralized data repository and therefore safeguarding that no private information is exposed.

• Over the course of the year, the Machine Learning Team in Singapore worked on the problem of building effective neural models for aspect extraction under both supervised and unsupervised settings in the context of sentiment analysis. This work was done under the SAP Industry Ph.D. program in collaboration with the National University of Singapore and Nanyang Technological University and was also topic of a recent blog post. The team has already published three papers covering this research area and in the last year followed up on their EMNLP’16 paper on Recursive neural conditional random fields for aspect-based sentiment analysis. In their work presented at AAAI’17 the team investigated on Coupled multi-layer attentions for co-extraction of aspect and opinion words, advancing on the previous method by replacing pre-processed dependency relations with an automatic attention mechanism. Their recent ACL paper An unsupervised neural attention model for aspect extraction approaches the problem in an unsupervised setting using topic modeling. This research has a wide range of applications in different business and social domains by helping both companies and individuals to better understand opinionated information for decision making.

A selection of the presented works will be summarized in individual blog posts to provide you with more specific results and insights that we’ve gained during these research projects.

In 2018, we foresee important discoveries in establishing theoretical foundations for the variety of deep learning methods proposed over the past years along with further advancements in areas like reinforcement learning. Moreover, we expect an ongoing diversification of machine learning branches.

As we gear up for the upcoming months, we know that making progress in the ML field requires continued joint and consistent efforts. Besides building intelligent solutions for the business sphere, this creates the need for an up-to-date research portfolio. In the future, our team will therefore further drive ML research initiatives that create and provide impactful results.