Insights from ICLR 2018

Tassilo Klein, Shachar Klaiman, Konrad Schenk, Marius Lehne and Steven Jaeger

For its sixth consecutive year, the International Conference on Learning Representations (ICLR), one of the main conferences in the field of deep learning, took place at the Convention Centre in Vancouver, Canada between April 30th — May 3rd, 2018.

ICLR enables those in the field to review recent techniques and trends in learning representations and addresses an international community of researchers, students and industry partners. Much like the AI industry as a whole, ICLR is experiencing exponential growth. This year’s conference received 935 submissions compared to 430 in 2017, from which 23 (2%) were accepted for oral and 314 (34%) for poster workshops. Similarly, attendance to ICLR has nearly doubled from around 1100 visitors to roughly 2000 compared to the previous year.

As official sponsor of ICLR’18, the SAP Leonardo Machine Learning team showcased our newest research projects and participated in the large spectrum of talks and workshop sessions offered at the conference. In this blog post, we will summarize general trends and paper highlights we discovered that are worth revisiting.

For more details, please refer to our team’s full conference report.

Vancouver Convention Center

Main Topics and Trends

- Generative Adversarial Networks (GANs)

With over 40 conference contributions in this area, Generative Adversarial Networks (GANs) remain one of the hottest topics in current machine learning research; a trend which can also be observed in other conferences. There were numerous contributions that focused on making the training procedure more stable through improved loss functions and architectures. Further, there is ongoing research on more fundamental and theoretical questions.

Improvements were made on the side of applying GANs, such as in a conference contribution that used GANs to break cyphers in an unsupervised way. Also in the vision domain, improvements have been made such that GANs are now able to generate realistic high resolution images.

- Adversarial Attacks and Defenses

Deep neural networks can be manipulated into predicting incorrect classes through malicious input data. Those adversarial examples are characterized through the application of minimal perturbations to the original data such that a previously correct classification leads to an arbitrary classification result with a high confidence.

Adversarial attacks pose a severe risk to the safety of various types of systems using state-of-the art machine learning methods. For example an attacker could fool an autonomous driving system by manipulating a traffic sign. Depending on the intention of the attacker, this could lead to catastrophic results. Thus, increasing the robustness against adversarial attacks is an active subject of research. The topic was present through various talks and poster presentations. Continuous research is performed on finding and improving methods to generate adversarial examples, on strategies to defend against such attacks as well as getting a more in-depth understanding of adversarial attacks in general.

- Compressing Neural Networks / Deep Learning on the Edge / Architecting resource efficient DNN

The notion of memory-efficient models as well as models that allow fast inference is getting more and more attention. This can be attributed to the fact that current focus was put on pushing for accuracy by devising ever deeper architectures that have large memory footprint and are slow. Therefore, operating these architectures in a cost-efficient manner or on portable devices is hardly possible. To alleviate the problem without sacrificing accuracy, the community is working on a series of approaches. Among the most prominent ones are quantization or binarization of network weights, selective removal of neurons to reduce over-capacity of networks and knowledge distillation, i.e. emulating a deep network by a shallow network and smart architectures that allow skipping computations, e.g. for easy data points.

- Deep Reinforcement Learning

While the first ideas for Reinforcement Learning (RL) dates back several decades, a lot of progress has been made with the advent of deep learning. The main concept of a RL system is to have an agent learning a policy to change a system’s state towards a target state through actions. The agent initially does not know which action is the most feasible for the currently given state but gets occasional rewards to optimize the policy of the agent.

Initial ideas of leveraging the power of deep learning for RL were to estimate the Q-values, representing the total expected reward for the agent after taking a particular action, with neural networks. Soon, deep learning networks were used to replace other parts of conventional RL like policies or even to extend the RL framework e.g. with actor-critic networks.

Even though RL still does not generalize well and shows difficulties in reproducibility, as stated in one of the invited talks, more than 70 related papers were published at ICLR’2018. The scope ranges from applying Deep RL to solve various tasks to leveraging RL in deep learning methods. The more theoretical papers focus on generalizing learned policies or on improving RL with new approaches.

A Selection of Interesting Papers

Below a selection of our favorite papers and why we think they are worth reading. For more paper highlights and details, please take a look at the full ICLR report.

• Learning and Memorization: This paper shows that a network of simple lookup tables is capable of solving the MNIST task with surprisingly good test accuracies. It shows that memorization can lead to generalization.
• Learning to represent programs with graphs: The authors show how it is possible to represent a program in a neural network. They observe that programming languages enforce a graph structure and therefore make direct use of graph-based neural network architectures. This work showcases some fascinating applications that can improve code quality and reveal previously undetected errors in source code on real-world data sets.
• On the importance of single directions for generalization: The paper shows that class selectivity of neurons is a poor predictor of task importance. It also reveals that networks that generalize well and show good performance have little dependence on individual units, overall reducing selectivity. Another interesting finding is that the commonly used batch normalization seems to implicitly discourage reliance on single unit neural selectivity. Besides, although dropout is known to serve as an effective regularizer to prevent memorization of randomized labels, the authors point out that it is unable to prevent over-reliance on single activations past the dropout fraction. This can be attributed to dropout leading to an implicit creation of redundant representation copies within the network in order to compensate loss of units.

Overall, ICLR was a smaller counterpart of other leading machine learning conferences such as NIPS or CVPR. During the five conference days, a broad range of topics were covered and we were introduced to several new research approaches as well practical applications for industry. Besides the focus on reinforcement learning, there was an emphasis on generative models. In view of the vast progress in computer vision, the relatively slow development in healthcare was basis for discussion along with the various reasons behind such as data privacy. Lastly, the conference reinforced the issue of reproducibility in machine learning uncovering a strong need for improvement in this area.