Building A Deep Learning Neural Network Startup
The most fantastic startups of the next decade would be starting now, curating a specific dataset which the big tech giants don’t have, and using it to train their own neural network and making their product vastly better as a result.
80s was the age of PCs, 90s was about the Internet, mid 2000 till date has been about Smartphones. Now, for the next decade or so, we are entering the age of Deep Learning Neural Networks, until the next paradigm-changing, mass-market technology breakthrough comes around. Not just AI or machine learning — but specifically, deep learning neural networks. If there is only one post you read about it, I recommend skipping this article and reading this fantastic New York Times in-depth story about it. It is very, very long — but we are talking about a significant trend for the next decade:
If it can be digitized (example — images), if it is data which can be captured in a database, then large samples of that data can be used to train a deep learning neural network. Here is an overview:
How does the human brain learn ?
The human brain is comprised of ~100 billion neurons which have ~100 trillion interconnections between them called synapses. Each of these neurons — for a given input, is either in an off or an on state. The interconnections work on a concept of positive reinforcement — a set of inputs leads to a certain output and the “brain” correctly remembers that path and “learns” to associate this way. Positive feedback is remembered, and these interconnections fire faster in such cases. With further breakthroughs in computing power and other research, perhaps 2030s will be a lot more interesting for brain-machine interface and related technologies. For now, we have deep learning neural networks to contend with. They are here, today.
What is a deep learning neural network ?
It is a mathematical formula, represented in a computing model — a best case approximation we have today of trying to replicate the human brain. Although nowhere even close, because the human brain is a lot more complex and powerful than any machine humans have been able to devise. We are however able to devise such neural networks which are good at one/or a few tasks only. While these machines are able to learn and excel and be better than humans in selected tasks, this is not “intelligence” as such. It does not transform or has an impact from one task to another. This is a specific technology, an application of machine learning, which has transcended from research into implementation to a form where anybody can dive in and create/train their own neural network.
How does a deep learning neural network learn ?
Artificial neurons (called perceptrons, and sigmoid neurons…huh), are arranged in a computer model which takes a set of input data, key parameter values and works through by building patterns of patterns to form a positive co-relation with a path which leads to the expected output. For example, to identify a keyboard from just looking at its picture — the core, basic element is a pixel of a key which is either in an off or an on state. Just a small tiny little pixel. A pattern of pixels in an on state would then lead to the pattern of one key. And a pattern of keys would lead to a positive association with the object being a computer keyboard.
Such a neural network is trained by feeding it pictures of tens of thousands of such objects which are already labelled as such (keyboards in this case). The better the data the neural network is trained with, the better results can be obtained. If the data itself is biased, then it is still a garbage in-garbage out system. There is no “intelligence” here — simply the ability for a system to acquire the ability to learn what it is taught.
Why now ? Haven’t neural networks been talked about for ages ?
Big Data, Machine Learning and significant increase in computing power have been key enablers of deep learning. Without data to train the neural network, there is no AI, no magic here.
While the concept of neural networks has been around for a while, back in 2012, all of this came together along with fundamental research done by Geoffrey Hinton and others. The magic happened at Google, where the Google Brain team converted this theory to implementation. First came the “cat experiment” demo, where researchers fed still-image pictures of cats from cat videos on Youtube to a neural network, and it was able to identify cats in pictures where they were not labelled so. Pass the champagne moment for researchers.
Google Translate — switch from programmatically expressed rules to a deep learning neural network
Then, the real turning point came Nov 2016 when Google switched to using a deep learning neural network for its Google Translate service, making a drastic switch from prior 10 years of work building algorithms programmatically as the results from the deep learning neural network were orders of magnitude superior.
TensorFlow
Google has also open sourced its machine learning platform called TensorFlow which it uses for its own products. Using pre-learned neural network models which Google makes available, anybody can get started with it. A fellow in Japan used TensorFlow to implement a machine which can automatically sort cucumbers based on certain criteria — his parents were farmers and training an employee to sort cucumbers has always been cumbersome for them.
All big tech giants are deeply invested in this
With the level of research going on at Google, Microsoft, Facebook, Baidu and others — these are the “Bell Labs” of our generation — where a lot of good, hard tech is coming out from. The PhDs are loved again, and I read on a deep learning VC’s blog that a newly minted Deep Learning PhD commands compensation in the millions of dollars. In the New York Times article, there is note of how Facebook’s CEO, Mark Zuckerberg, personally takes the time to meet and court the exceptional talent in this emerging field.
Over a $1 billion has been invested by VCs into AI startups.
Where can it be used ?
Such a neural network can not only identify/classify objects, but also predict.
For instance, looking at pictures partially, and filling in the rest — what could have been. A version of this is actually being tested extensively by Google in a project called RAZR — where a lower resolution image is downloaded to the user’s phone and machine learning is used to enhance it afterwards. This saves on bandwidth significantly.
How about improving upon the function of a radiologist — a neural network trained based on feedback provided by the best radiologists around the country, which can read your reports and offer better, more reliable feedback than a single radiologist can going forward ?
How about a neural network which can learn the most convenient path to travel from point A to point B, going over and beyond the current generation of programmed map applications which provides the fastest route by default ? In fact, Google’s Waze, which crowd-sources traffic conditions data from its users, must be a prime user of this technology — where deep learning neural networks can be trained with that data set to provide better routes.
Sports — helping coaches plan out the best strategy by having a neural network trained by “watching” competitor’s video archives.
Combined with AR tech — imagine scanning your phone over your restaurant dinner plate and seeing a quick identification of what are the food items in the plate and what is their nutritional content. This could help people eat better and be healthier. For folks with food allergies, can’t this provide an added level of comfort, even though not 100% surety ?
But wait, what about chatbots ?
There is a lot of momentum for chatbots currently. However, they are a user interface for consumers. While deep learning is the foundation of the Natural Language Processing (NLP) needed to infer what the customer is saying and to respond like a human as much as possible — the real deal is the inference derived by a neural network from massive amounts of data which can actually be used to make the product better. This is like driving up through a McDonald’s drive-through, and interfacing with the most awesome menu selection ever, but what you care fundamentally about is the actual product — the food. Has deep learning been used to make that better — now that would be interesting.
Deep learning neural networks are no fad — this tech is here, has made the jump from the lab to consumers, and now begins a rapid growth phase where we would experience it in every aspect of our lives. For ambitious startups looking to get ahead of the curve, there is no shortcut here — you need to curate data — more specific it is, less chance the big tech giants will be able to replicate it.
Modelling graphs with neural networks
This is a personal preference — but one of the most intriguing concepts to me is that of training a neural network on graphs. If you friend Mark on a social network, then you are more likely to friend Maria and Jenny which will make your engagement higher and experience happier as a user…modelling human communities — the possibilities are intriguing.
What’s next for AI ?
David Quail wrote a great comment in another forum on this post which I am cross-posting here as I think it is very relevant to this discussion:
….
Deep neural nets are cool. But they themselves don’t create machines that “learn themselves.” That’s the problem with deep learning (and supervised learning in general.) They need a teacher. A human expert that tells them the millions or billions of right answers a single question. It’s obviously incredibly effective. But it’s rote learning, it doesn’t scale to other predictions. And it’s hardly learning on it’s own.
This is why people are so excited about reinforcement learning paired with deep learning. True experience based learning of general predictions without requiring a human expert to tell the system millions and millions and millions of times, what the right answer is.
Yann LeCun, Facebook’s AI guru, keynoted NIPs this year talking about this type of learning being at the heart of what’s next in AI. It’s worth a read for anyone interested in this area. https://drive.google.com/.../0BxKBnD5y2M8NREZod0tVdW.../view
A world where the AI is advanced enough to have “common sense”, and can for example predict the next several frames in a video will bring with it a new wave of opportunities and challenges — but I wonder how far out that is ?
I think even with the current implementation of deep learning neural networks which require training with a large data set and are special to a singular task — that is still “good enough”, today, for a variety of uses. We need large digital datasets assembled through crowdsourced applications (such as what Waze has done), or datasets available through other means (eg sports video archives of a particular player or team). The world to us is inherently visual — if we take an image, it can be digitized, analyzed and used to train a neural network. There will be startups who either already have a well-defined path to collect this data or which start with the express purpose of collecting data so as to train their neural network to provide significant more benefit to their users.
Augment humans, instead of replacing them outright from every major profession ?
In a world of 7 billion+ people, with life expectancies longer than ever, how far advanced would AI get before the sentiment becomes hey wait a minute, we need to regulate it to prevent massive job losses. If everyone in retail, taxi driving, customer service, coaches, radiologists, even artists etc etc is to be replaced by AI, because it will be better and eventually cheaper, what will people do to earn their living ? What skills do they need to acquire ? With the combined forces of Globalization, Immigration and Automation hitting at locals jobs — there needs to be a balance or we will have a very unhappy population.
Even as a computer programmer and a data analyst — where I have a level of competence, I am not sure if that will be a viable profession 10 years from now — given a neural network should be able to write better code and do better analysis that I do. There are patterns in how I code and how I analyze data. The AI could analyze my work and analyze the work of others within a subset (say the enterprise) and write it itself. Or better still — it can learn/predict in a different,better way altogether. Even management is a function of automated codified rules at the world’s largest hedge fund according to this WSJ article.
What would I do ? Heck, start a crowdsourced app/marketplace, acquire massive amounts of data (in a specific vertical which no one has), and build my own deep learning neural network which would feed back into the product with massive improvements. If not then open up a coffee shop or something! AI is not replacing the need to eat or drink anytime soon — but would people have continued spending power if they have just lost their job to AI ?
What’s your plan ?
Since machines can’t think, my blogging is safe for now. Although — the Internet being flooded with good-quality machine written content is not far away.
But, wait. Here is the best part:
“AI software learns to make AI software”
In an interesting twist of fate — researchers at Google are already designing neural networks which can learn to build their own neural networks which can learn so that they are not reliant on human experts only. Uh…what an exciting future lies ahead!
How would humans fit in such a world — or would we ? Would we out-innovate ourselves’ out of existence (“cheez, it seemed like a good idea at the time”), and some have called this the “last invention of humans” ? Or maybe it will free us up to use our powerful brains for more advanced analysis. A new age of Renaissance perhaps ?
Some machine learning algorithm would be reading this blog post several years from now, motivated by gamification as I am, to write excellent content like this and earn rewards in the form of clicking on that green little heart icon below. Maybe it would have learnt to snicker. Here is to you, deep learn this:
Resources
- Learn TensorFlow and deep learning, without a Ph.D. https://cloud.google.com/blog/big-data/2017/01/learn-tensorflow-and-deep-learning-without-a-phd
- Deep Learning online book: http://neuralnetworksanddeeplearning.com/index.html
- http://www.deeplearningbook.org/ — by Ian Goodfellow and Yoshua Bengio and Aaron Courville
- Matt Turck: Deep Learning VC http://mattturck.com/2016/09/29/building-an-ai-startup/
- AMA with Geoffrey Hinton on Reddit: https://www.reddit.com/r/MachineLearning/comments/2lmo0l/ama_geoffrey_hinton/
- Geoffrey Hinton’s Machine Learning course on Coursera: https://www.coursera.org/learn/neural-networks
- A CS PhD student at Stanford’s take on deep learning: http://karpathy.github.io/neuralnets/
- AMA with Google Brain team on Reddit: https://www.reddit.com/r/MachineLearning/comments/4w6tsv/ama_we_are_the_google_brain_team_wed_love_to/
- Paper published by a Stanford University prof describing Deep Learning: http://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00239
- https://cloud.google.com/blog/big-data/2016/08/how-a-japanese-cucumber-farmer-is-using-deep-learning-and-tensorflow
- Graph Convolutional Networks: https://tkipf.github.io/graph-convolutional-networks/
- TensorFlow: https://www.tensorflow.org/tutorials/
- https://matthewearl.github.io/2016/05/06/cnn-anpr/
- https://bamos.github.io/2016/08/09/deep-completion/
- https://research.googleblog.com/2016/03/train-your-own-image-classifier-with.html
- http://a16z.com/2016/02/22/whats-next-in-computing/
If you liked this post, please click the recommend button below (“heart icon”) to refer it to others on your network. Thank you!
