Deep Learning Race: A Survey of Industry Players’ Strategies
I’ve been working for quite a while now in trying to make sense of the research developments in Deep Learning. The methodology I’ve employed is through the cataloging of Design Patterns. It’s been quite effective in disentangling this ever growing complex field. In fact, as new surprising research is published by the giants in this field, my own conceptual understanding of how it all fits together continues to be tweaked.
There are, however, certain patterns that I have observed that is actually outside that of a general understanding of Deep Learning. What I’ve observed is that the different leading research groups seem to emphasize different kinds of approaches in solving the riddle of artificial intelligence. Now, a bit of a caveat here, I’m not privy to the internal machinations inside these organizations. So if there’s some secret sauce that an organization is brewing, then obviously I wouldn’t know about it.
However, by just reading the research publications, that fortunately come out quite occasionally, one gets a sense that organizations favor one approach from the other. (Note: The approaches are not mutually exclusive) So, with that out of the way, let me give you my intuition on the biases (or rather preferences) that each of the big players in the field approach Deep Learning research.
Google DeepMind
Ever since Google bought out DeepMind after seeing their Atari game playing AI, DeepMind has always had the preference of using Reinforcement Learning in their approach. They certainly use Deep Learning as a component in most of their research but always seems to emphasize its combination with RL. The DL research tends to focus on using variational methods that are embedded as non-parametric layers in their models. They also have a focus on attention mechanisms and memory augmented networks. In terms of breadth of research, I don’t think there is any organization that is remotely close to DeepMind. DeepMind research seems to be driven by the need to discover the nature of intelligence. You can find more of their work here: https://deepmind.com/research/publications/ [DMT]
Google Brain ($GOOG)
Google has a distinct pragmatic and engineering approach in how they approach their research. You can see how they make endless tweaks on the inception architecture. They have detailed work on how they arrange their DL architectures around the compute resources that they have available. Google also combines other traditional algorithms like beam search, graph traversals and generalized linear models with DL . The approach seems to also emphasize the need for scalable solutions. Google is able to achieve impressive results as a consequence of their massive computational and data resources. You can find their research here: https://research.googleblog.com/ [GOO]
Facebook FAIR ($FB)
This is headed by Yann LeCun. It is unclear how strong this group is since it seems that most innovative research comes from LeCun’s group in NYU. LeCun’s group does an experimentation that explores the fundamentals, this is a sorely missing aspect that groups fail to perform enough. FAIR has published a couple of good open source implementations in Torch. They’ve done some good work applying DL in certain problems. It is, however, hard to see if there is any particular research preference. I find it difficult to discern an overarching theme of their research. Maybe you might have better luck: https://research.fb.com/publications/?cat=2 [FAC]
Microsoft ($MSFT)
Is similar to Google in that their approach is very pragmatic and engineering oriented. Microsoft surprisingly has top notch computer science talent that had led to the discovery of Residual networks. They have other novel approaches like the Decision Forrest that indicates that the company clearly has thought leadership in this space and is not a company that is just a follower. Their Cognitive toolkit, despite coming late in the game, is a high- quality piece of engineering. It likely is one of the better frameworks out there with respect to learning using distributed computers. I would say that Microsoft is likely second to Google in their research contributions. That’s a big statement to make considering none of the original DL researchers have joined their team. See: https://www.microsoft.com/en-us/research/research-area/artificial-intelligence/ [MIC]
OpenAI
OpenAI was founded by Elon Musk (and some other lesser dudes) driven by the fear of seeing how quickly other firms had acquired Deep Learning talent. If you can’t compete on financial terms, then offer academic freedom instead. OpenAI tends to favor the approach of using generative models, more specifically generative adversarial networks (GANs). They’ve also made a serious effort towards the Reinforcement Learning space. What’s curious though is that despite the impressive results of GANs, is that DeepMind seems to have a preference towards variational models instead. A glimpse of their research priorities can be found here: https://openai.com/requests-for-research/ [OPEN]. Microsoft has offered its Azure service to OpenAI thus potentially assuring OpenAI’s allegiance.
University of Montreal
This is Yoshua Bengio’s group that is never really short on publications. Bengio is one of the few brave researchers who has not succumbed to joining a commercial entity. Similar to Lecun’s NYU group, the approach focuses on trying to understand why DL works. The group also has plentiful work on tweaking DL with new models and learning algorithms. MILA is arguably the best academic DL research group on the planet (see: https://mila.umontreal.ca/en/ ) [MILA]. Google was pretty savvy by helping the group to some additional spare change. This move encouraged Hugo Larochelle to exit from Twitter ( quickly becoming a non-player ) and join the newly founded organization.
SalesForce MetaMind ($CRM)
Headed by Richard Socher, most of the work coming from this group has an obvious bent towards solving NLP problems using RNN based approaches. These solutions will tend towards networks that employ memory in their construction. The output of these folks are quite impressive and therefore should be taken very seriously. Here’s their work: http://metamind.io/research-list.html [META]
Baidu ($BIDU)
Andrew Ng’s group that likely was one of the first organizations to really create massive GPU systems with Infiniband networks. They’ve done a lot of work emphasizing infrastructure and have open source some of their solutions. Their research focus has been in image and speech processing. In the latter, they’ve done extremely well. Their research can be found here: http://research.baidu.com/institute-of-deep-learning/ [BAI]. Baidu was also originally involved with BMW with regards to self-driving cars.
Nvidia ($NVDA)
Nvidia is betting the bank on Deep Learning solutions. They have the best engineering team that is out there tweaking their GPUs to get maximum performance. They are completely absent in terms of innovative DL research, but you can trust them to have spent serious effort in building the computation resources required for DL. They, however, have likely one of the leading implementations on self-driving cars. Their end-to-end DL solution for self-driving cars is one of those DL research papers that I remember that’s notable.
Intel Nervana ($INTC)
Intel was in deep trouble prior to their acquisition of the Nervana startup. Their hardware solutions were nowhere near competitive to Nvidia’s. Nervana is very good at engineering the computational infrastructure used by Deep Learning. Some of the fastest implementations on GPU hardware come from Nervana. The speculation here is that Nervana was acquired for its software chops and not necessarily for the hardware they were designing. It is curious that the Nervana hardware solution ETA is mid-2017, that in the DL field is a very long time. You won’t find a lot of research publications coming out of Nervana, I don’t think they were able to amass enough DL talent while they were still a startup. However Intel is a player you cannot write off, they have technology shared with Micron ($MU) that may give them an insurmountable advantage in this space.
IBM ($IBM)
IBM got this entire AI craze going wheb Watson destroyed its human competitors in Jeopardy. They were perceived to have been way ahead of everyone back in 2011. Unfortunately, Deep Learning came along and IBM has been slow to react in this space. Watson certainly uses DL in some of their solutions, however, they are almost non-existent in terms of DL research. They have this TrueNorth neuromorphic computing thing going, but that’s mostly more hype than actual substance. IBM is not as prominent a researcher in the DL space, but if you look hard enough you’ll find some: https://arxiv.org/abs/1604.08242 [IBM].
Apple ($AAPL)
Not really much to say here other than the hiring of Russ Salakhutdinov from CMU tells you that future research would emphasize his bent toward unsupervised learning approaches. Apple is busy acqui-hiring companies. Some of the bigger hires (i.e. Dato) were in the ML space. Unfortunately, their acqui-hires were mostly in the data science and big data space. I think when Apple realized that DL was different from ML, they rushed out and bought out Salakhutdinov. Apple’s culture is extremely secretive, so I’m not sure if their lack of research publication is an indicator of lack of expertise or towing the corporate line. Update: BusinessInsider reports Apple will start publishing their work.
Amazon ($AMZN)
Not much to say other than the support of MXNet was a good thing. They had originally open sourced a framework called DSSTNE, it was unfortunately destined to receive a lot of neglect by the community. It’s hard to get attention when there are so many competing frameworks that are out there. MXNet is actually impressively good considering that it did not have a big corporate backer.
Uber
Uber has made recent acquisitions of Otto (Self-Driving Trucks) and just recently started their AI Lab through the acquisition of Geometric Intelligence. Gary Marcus, the founder, was featured previously in other publications, he was unimpressed at that time with Deep Learning capabilities. It is important to note that Gary Marcus is a cognitive psychologist and not a computer scientist. Geometric Intelligence, however, was able to hire a basket load of ML practitioners. Uber’s acquisition by all likelihood was an acqui-hire considering that is unlikely that Geometric Intelligence had any revenue, much less any product to speak of.
Just another caveat, these are just my impressions and I am certain, given the deluge of information in the DL space, that I may have missed some publications that these organizations may have put out that should have been recognized. So my apologies if you feel my review was unfair.
My point though of this article is just to point out that AI research landscape is not homogeneous. Different organizations have different priorities in what they believe as important research areas. There is definitely an AI race going on and the likely winner is the company that actually was the one lucky enough to have put resources in the winning research approach! It is indeed very odd, that given the high stakes involved, that there’s an implicit gamble that every firm seems to be making.
An early overview of ICLR 2017 gives an idea of the DL research activity of the above organizations.
I am certainly surprised how low Baidu and Salesforce.com are on this list. Nevertheless, quantity does not necessarily always reflect quality. Let’s wait till the submissions get reviewed. Also worth pointing out are the universities without a corporate affiliation, you will find that some of them have startup spinoffs that are ripe for corporate acquisition (just like Geometric.AI).
References
[DMT] https://deepmind.com/research/publications/
[GOO] https://research.googleblog.com/
[FAC] https://research.fb.com/publications/?cat=2
[MIC] https://www.microsoft.com/en-us/research/research-area/artificial-intelligence/
[OPEN] https://openai.com/requests-for-research/
[MILA] https://mila.umontreal.ca/en/
[META] https://metamind.io/index.html
