The Down Sides of Deep Learning

I came across an interesting post on Quora yesterday where a user asked about the “cons and disadvantages of deep learning.”

When I first saw the question, I was curious to see what the answers would be. Thus far, deep learning has proven to be an incredibly effective tool at solving problems that were previously only in the realm of human expertise, so I didn’t think that the answers would relate to anything around its performance.

For the most part, that was indeed the case (one of the points given was that deep learners don’t generalize well, which makes sense if you aren’t giving it the data to generalize from!).

The answers that were given related to our ability as humans to manage and understand what’s going on as a deep learner.

We still struggle to give explanations about what deep networks are truly doing under the hood. We can talk about how they start learning, but what they start learning is still difficult to analyze and parse out of the network. On top of that, you never really know when a deep learner has gotten to be as good as it’s going to get.

As the rest of the answers mentioned, they also take a lot of resources in the form of time, hardware, and data, which makes developing them a bit of a pain.

I’d also throw in there that deep learners don’t always deal well with combining structured numerical data with unstructured hierarchical data like images and text . We’ve had a lot of trouble supplementing some of our deep learners with external, numerical, structured information. They’ll frequently go to NaN or become unstable in their training process. Things like that are extremely hard to debug, as well, which is another problem with deep learners.

From a practical standpoint, the biggest hurdles that my team is currently trying to solve when trying to use deep learning from a practical standpoint includes the following:

  • Developing a methodology for creating the “best” deep network structure (in terms of layers, layer composition, and hyperparams), and knowing whether that will actually be the best as we add more data
  • Taking deep learners that are good at parsing unstructured data and supplementing them with structured numerical data such that they’re stable and their performance improves
  • “Helping” our deep learners learn more by doing feature pre-processing (i.e. there are things that deep learners should be able to learn if we had an infinite amount of data, but since we don’t, we need to structure our inputs in a way that allows them to glean patterns more quickly)
  • Trying to come up with some way, as roundabout as it may be, to explain what the deep network is learning and how that allows it to pick up on patterns that allow it to predict our objective

There are a lot of hurdles, that’s for sure. But the great thing about deep learning today is that there have been some fantastic tools created to ease the use of developing deep networks for real world applications.

Those sorts of tools (which, I should note, don’t seem to be as polished in the world of reinforcement learning) make it extremely easy to apply standard software development principles to a machine learning application.

Ultimately, I suspect that as machine learning algorithms get more and more complex, and as they start to resemble the human brain, both in form and in function, the problems I’ve mentioned above will only amplify.

I don’t think that’s necessarily a bad thing. Humans have known for a long time that A.I. will ultimately be better than us at many things, and as that happens, we should be okay with letting algorithms do what they’re best at doing without having the need to step in and explain or manage them. Of course, I’m talking short-to-mid term here. If Elon Musk’s biggest fears ever do come true, then we better be ready to step in and pull the plug.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.