Dense layers explained in a simple way

A part of series about different types of layers in neural networks

Assaad MOAWAD
Oct 8, 2019 · 2 min read
Image for post
Image for post

After introducing neural networks and linear layers, and after stating the limitations of linear layers, we introduce here the dense (non-linear) layers.

In general, they have the same formulas as the linear layers wx+b, but the end result is passed through a non-linear function called Activation function.

y = f(w*x + b) //(Learn w, and b, with f linear or non-linear activation function)

The “Deep” in deep-learning comes from the notion of increased complexity resulting by stacking several consecutive (hidden) non-linear layers. Here are some graphs of the most famous activation functions:

Image for post
Image for post

Obviously, we can see now that dense layers can be reduced back to linear layers if we use a linear activation! But then as we proved in the previous blog, stacking linear layers (or here dense layers but with linear activation) will be redundant.

Intuitively, each non linear activation function can be decomposed to Taylor series thus producing a polynomial of a degree higher than 1. By stacking several dense non-linear layers (one after the other) we can create higher and higher order of polynomials. For instance, let’s imagine we use the following non-linear activation function: (y=x²+x). By stacking 2 instances of it, we can generate a polynomial of degree 4, having (x⁴, x³, x², x) terms in it. Thus the more layers we add, the more complex mathematical functions we can model.

Limitations

Dense layers add an interesting non-linearity property, thus they can model any mathematical function. However, they are still limited in the sense that for the same input vector we get always the same output vector. They can’t detect repetition in time, or produce different answers on the same input.

If we are in a situation where we want that:

  • If the first input = 2 the output will be 9
  • But if the next input is 2 again the output should be 20 now.

We can’t model that in dense layers with one input value. Because if f(2)=9, we will always get f(2)=9. If we want to detect repetitions, or have different answers on repetition (like first f(2) = 9 but second f(2)=20), we can’t do that with dense layers easily (unless we increase dimensions which can get quite complicated and has its own limitations). That’s where we need recurrent layers.

If you enjoyed reading, follow us on: Facebook, Twitter, LinkedIn

DataThings

DataThings blog is where we post about our latest machine…

Assaad MOAWAD

Written by

Interested in artificial intelligence, machine learning, neural networks, data science, blockchain, technology, astronomy. Co-founder of Datathings, Luxembourg

DataThings

DataThings blog is where we post about our latest machine learning, big data analytics and neural networks experiments. Feel free to visit our website: www.datathings.com

Assaad MOAWAD

Written by

Interested in artificial intelligence, machine learning, neural networks, data science, blockchain, technology, astronomy. Co-founder of Datathings, Luxembourg

DataThings

DataThings blog is where we post about our latest machine learning, big data analytics and neural networks experiments. Feel free to visit our website: www.datathings.com

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store