Insights from NIPS 2017

Tassilo Klein, Marius Lehne, Brian Clarke and Steven Jaeger

Published in

SAP AI Research

8 min readDec 21, 2017

The Conference on Neural Information Processing Systems (NIPS) took place between December 4th and 9th in Long Beach, CA, USA.

As one of the top machine learning and computational neuroscience conferences, this year’s NIPS was a complete success experiencing an even larger rush of attendees compared to previous years. Apart from the rising number of participants, the conference has also seen a strong increase in submitted papers. Of the total 3240 papers submitted, 679 papers were accepted resulting in a 21% acceptance rate compared to last year’s 24%. This is an indicator of the conference remaining highly competitive despite its overall growing popularity.

As an official sponsor of NIPS, SAP’s ML researchers and data scientists were on site to present our SAP Leonardo Machine Learning solutions and research projects. Apart from connecting with the broader academic research community during the plethora of talks, workshops and tutorials, we presented our recent work on Federated Learning with Differential Privacy in the workshop for Machine Learning on the Phone and other Consumer Devices. Moreover, our research partners from University of Amsterdam showed their work Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols.

For more details, check out our full NIPS report.

Topics and Trends

As deep learning has developed to become a de-facto standard commodity in data science and research with numerous tools now readily available for non-ML experts, many new sub-domains of deep learning have begun to spring up. While the pre-deep learning area was dominated by feature engineering, the ML community has been experiencing a network architecture engineering era afterwards. However, now there is a massive push to establish theoretical foundations for all the methods proposed over the last years, which goes beyond proposing new (deeper) architectures. Besides the standard machine learning topics and trends therein, social topics in ML have started to be part of the conference agenda. This can be attributed to the emergence and integration of machine learning services into our daily lives including all the pros . As a result, there were a lot of talks on ethics, bias, privacy as well as a workshop addressing ML for the developing world.

- Generative Adversarial Networks

Similar to other conferences, the topic of generative models has seen quite some interest at NIPS, where particularly Generative Adversarial Networks (GANs) made up a significant amount of the contributions. However, GANs remain difficult to train. First, we could observe rather theoretical work determining and analyzing reasons for the instability of the learning process. Then, there were numerous modifications of GANs to further improve this process. Additionally, we saw many contributions that used GANs for various types of applications (e.g. semi-supervised learning).

- Reinforcement Learning

Reinforcement Learning (RL) has seen a lot of progress for many applications, as well as various research trends. However, RL still requires huge amounts of data and a very long time to learn, while at the same time suffering from poor generalization. In order to alleviate this problem, much research is conducted in the domain of meta reinforcement learning approaches that can adapt quickly to new environments. An interesting paper in this domain is Learning to reinforcement learn, as well as Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks.

Similarly, the notion of Hierarchical Reinforcement Learning tries to breakup tasks into smaller reoccurring manageable components, each controlled by sub-policies instead of one monolithic policy that might be harder to train. One paper trying to combine the idea of Meta RL and Hierarchical RL is [Meta Learning Shared Hierarchies. The topic of RL and especially Hierarchical RL was also prominent in the workshop on conversational applications. The basic idea is to have several domain specific agents that are orchestrated by a manager agent. Another interesting extension was presented in [Neural Map: Structured Memory for Deep Reinforcement Learning where the authors extend the standard memory architectures (usually an RNN) with dynamic memory networks. The topic of interpretability and ‘white boxing’ has gained traction in RL. An intriguing application can be seen in Natural Language Policy Search. In her talk, Joelle Pineau provided somewhat shocking insights on reproducibility: The difference in outcomes for two Reinforcement Learning agents were statistically significant depending on the random seed of the initialisation.

- Bayesian Deep Learning and Deep Bayesian Learning

Naturally, Deep Learning suffers from several shortcomings that limit wider applicability such as lack of interpretability, notion of confidence in prediction, as well as missing mathematical foundation. Bayesian Deep Learning tries to compensate these issues by means of combining Deep Learning with Bayesian probability theory. For more details, check out Uncertainty in Deep Learning, as well as the white paper Deep Learning: A Bayesian Perspective.

- Meta-Learning

In meta-learning, which is often also referred to as “learning to learn”, the deep learning model is itself a learning algorithm. The notion behind it being that when (meta-)training such a model, one will be able to learn a procedure that can learn in a more efficient manner through better generalization. Exemplary papers in this field are [Learning to learn by gradient descent by gradient descent and [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. Not surprisingly, meta-learning is a promising approach for few-shot learning, which seeks to learn new concepts from just a handful of training examples.

- Theoretical Foundations of Deep Learning

Modern deep learning architectures have enormous capacity, which theoretically allows them to easily memorize the data sets they were trained on. Yet, the models deliver state of the art results. Thus, one important topic, which up to now remains poorly understood, is to find explanations for the generalization performance of deep neural networks. More details can be found in the work Understanding Deep Learning requires rethink generalization presented at ICML this year. The lack of theoretical foundations in deep learning has also been criticized by Ali Rahimi in his acceptance speech for the test of time award. As within other conferences in the field, there were many papers focussing on the various efforts to move towards a better understanding of the generalization performance of deep neural networks.

- Fairness in Machine Learning

Fairness in machine learning is a research subfield concerned with the notion of avoiding unintended discrimination. This topic has been actively debated in various media reports in recent times. The concern arises from the fact that there may exist (implicit) bias in training data or in the formulation of algorithms, which may negatively impact society. As the research community has realized the need to address these concerns, the topic has received great attention throughout the whole conference. In their excellent tutorial, Moritz Hardt and Solon Barocas brought together the legal and the technical perspective of fairness in machine learning. This was followed up by an Invited Talk of Kate Crawford, in which she spoke of the impact of unfair machine learning methods on society. She pointed out that fairness is not only a technical problem but also a social one. Different papers were presented during the conference proposing methods (e.g. counterfactual machine learning) to address the different sources of bias.

- Privacy in Machine Learning

In many scenarios, practitioners and scientists have to work on sensitive data. Thus, preserving privacy in a machine learning context has become an essential topic for numerous tasks. This was also well reflected in the conference’s contributions, as many researchers proposed modifications to introduce privacy measures into various established machine learning methods. Differential privacy has become the method of choice for measuring privacy risks. If you are interested in a paper with the elementary definitions and formalisms of this topic, please check out Deep Learning with Differential Privacy.

- Explainable and Interpretable Machine Learning

Many currently successful deep learning methods are black-box methods. Even though practitioners and domain experts have identified that explaining a model’s outcome is highly important, there is an ongoing discourse about the necessity of explainable machine learning in practice. Yet, opening this black box through explainable and interpretable machine learning was well represented throughout the conference in various formats and it was noticeable that different approaches were presented or improved. As many proposed methods fail to live up to expectations, there is also a discussion about short comings and limitations. Due to this, there is a move towards a better definition of what such methods should achieve and how they can be evaluated in a better way.

- Learning on Graphs and Manifolds

Various lines of work, summarized in an interesting tutorial, in recent years have pushed towards extending the success of convolutional neural networks to graphs. Graphs are generalizations of the regular lattices found in text (1D grids) or images (3D grids) and are relevant to many domains e.g. social networks, molecules, and 3D shapes, which become graphs when represented as a discrete mesh. There are multiple possible generalizations of CNNs to graphs, with the two main paradigms being based on spectral and spatial interpretations of CNNs. At NIPS, the advantages and disadvantages of the two paradigms were presented, along with a number of intriguing applications ranging from recommender systems to quantum chemistry and particle physics.

Our Paper Highlights

Due to the high number of papers at NIPS reaching beyond 1000 papers, we can only present a hand full of papers that caught our attention. However, there were a lot more interesting papers presented and we therefore strongly recommend having a look at the list of papers here, as well as the workshop sites.

For more details about the papers and why we think they are interesting, please take a look at the full report.

Differentially Private Federated Learning: A Client Level Perspective (SAP ML Research contribution)

For many scenarios and applications the number of participants learning a model can be limited. Our approach shows that it is possible to train ML models under privacy constraints with a minor decrease in accuracy; even in settings where the number of participant clients is not extremely large. Interested to hear more? Our work will soon be discussed in a blog post!

NIPS remains the top conference for machine learning by keeping very high quality standards in terms of papers, talks and workshops, as well as covering a very broad range of domains. This is also reflected by the record number of attendees and sponsors in Long Beach, CA. As machine learning is gaining traction in a variety of domains, we expect this trend to continue in following years!