The 2019 Neural Information Processing System (NeurIPS) conference took place recently in British Colombia. The conference brings together the world’s most esteemed artificial intelligence experts.
The conference has taken place annually since 1987. Recently, with the explosion of research and advancement in neural networks, interest and attendance in this conference has grown in tandem.
Some of the world’s most accomplished deep learning researchers have presented at this conference including Michael I. Jordan and Yann LeCun. In earlier years, the conference had a reputation for being one big show. Just several years ago, Intel sponsored a performance by Flo Rida. The conference also had a reputation for reports of sexual harassment and discrimination.
However, they have certainly cleaned up their act and the conference this year appeared to be quite a world-class event with record attendance. Additionally, a review by the MIT Technology review states how the talks were more geared towards concrete real world problems rather than theoretical ones. Fortunately, it seems like the 2019 conference had great reviews from attendees and presented some incredible new developments to the field of machine learning.
In the rest of this review, I will discuss 9 highlighted papers that seemed especially interesting. Please note that I simply cannot include all the papers at the conference since there are over a thousand, and if you are interested in more of the great work happening at NeurIPS to check out the link here.
Highlights of NeurIPS
Jeff Dean attended the NeurIPS conference. Jeff Dean is the chief of Google AI and is renowned for some incredible work throughout his career. VentureBeat was able to interview Dean independently about the directions that artificial intelligence appears to be going in 2020. The interview allowed Dean to talk extensively about the custom machine learning hardware that is being developed at Google. In addition, it highlighted the work that Dean is doing on climate change. VentureBeat was also able to discuss accessibility of conferences with the chief of Google AI and the issues that some researchers had in getting to Canada for the conference.
Besides Jeff Dean, there were plenty of other noteworthy researchers. I discuss several of them below.
Deep Equilibrium Models – Shaojie Bai, J. Zico Kolter, and Vladlen Koltun
The new technique introduced in this paper is called deep equilibrium models are able to keep the memory usage of a neural network consistent even with increasing numbers of network layers. This is a huge advancement of neural network efficiency achieved by Intel Labs. They demonstrated that they could decrease memory usage by up to 88% in several experiments as well as improved performance with the WikiText-103 benchmark.
Neural networks with limited data – Gilad Yehudai and Ohad Shamir
Another paper discusses the problem of have huge neural networks to solve problems that do not have much data. The authors show that with a large enough network, optimization can occur with randomized parameters and allows them to be fixed when a huge network is used on a problem where one is not needed.
Reproducibility of deep learning models
One of the most interesting papers was that of Edward Raff that attempted to reproduce many different computing papers dating back to 1984. Raff was able to reproduce just 64% of the papers. The finding screams for better documentation about how the work was performed in research.
Continuous 3D-Structure-Aware Neural Scene Representations Vincent Sitzmann
The main idea with Vincent’s work is that they have 2-D observations but want the network to learn things about the 3-D world. A scene is a function with xyz coordinates with a mapping from the coordinate to the object present at that xyz coordinate. They built a “neural rendering engine” that allows taking 2-D images and transforming those into 3-D. Currently, their data is all computed artificially, though their goal is to take it to the next level and use real images and scenes. Additionally, objects such as reflections are not captured in their model currently and they wish to extend the work to that in the future.
Dynamics of stochastic gradient descent — Sebastian Goldt
This work by Sebastian Goldt takes a deep dive into neural networks and why they perform so well. Specifically, he looks at simple two-layer networks through a physics lens. Fascinatingly, he applies a physics viewpoint to his work, where he runs models and reviews what is the result in real life rather from a theoretical standpoint.
He mentions in an interview that there is a large amount of work going into understanding why neural networks do well since attempts to explain them in the past have not worked. He was able to calculate the performance of a neural network in lots of different architectures and parameters with a new model that he has created. He mentions that one cannot solely look at the neural network algorithm to find out performance, other factors need to be considered as well such as input layer, data, and the algorithm used.
Goldt and the team still don’t understand how data structures affect neural networks in real datasets and will pursue this in future work.
Kernel Instrumental Variable Regression – Rahul Singh
Instrumental variables allow detangling variables and determining true effects of variables on effects. Singh comes from an economics background and uses the work for demand estimate. One of the cases that he mentions in an interview found here is that A/B testing can be used to untangle the effects of income and the performance of a drug.
Geometry-award Neural Rendering – Josh Tobin
Like Sitzmann’s research described above, Josh Tobin is helping robots to understand the scenes that they’re in. Typically, robots have state representations of the world that they are trying to understand (concrete things in time such as position of robot, etc.). However, this can’t scale to highly complex scenes as the number of objects and variables increase.
He also mentions the concept of Neural rendering where cameras can look at a scene from multiple angles and then try the model out with a test angle. The test angle is a new viewpoint that the model has never seen. If the model can impute the scene from that new viewpoint, then they can say that the model must have some understanding of the scene.
Fast and Accurate Lean – Mean – Squares Solvers – Ibrahim Jubran and Alaa Maalouf
Jubran and Maalouf presented a fascinating preprocessing step that allows everything down the machine learning pipeline to stay the same. They demonstrate a compression algorithm that allows compressed data to be pushed through the rest of the pipeline for faster processing. This resulted in lowered runtimes with the same accuracy. No knowledge of the downstream processes is required.
Brain-Like Object Recognition with Recurrent ANNs – Martin Schrimpf
The paper that was accepted from Martin Schrimpf proposed two main concepts. The first was the concept of a brain store platform which is a platform that can tell how a model can match to the brain. His research groups is trying to build networks that behave more like brains. Additionally, they presented work that allowed transforming deep networks into more shallow networks through recurrence approaches.
Concluding Remarks
The NeurIPS conference attracts the top talent in machine learning and deep learning. Some of the most influential ideas are presented at this conference for industry leaders and top academics to learn about and implement into their own work. The 2019 NeurIPS conference was a great success and the work presented will certainly be used in industry over the next year.
