# PyTorch Deep Learning Nano degree: Neural Networks

## A second part of the Nano degree: neural networks

*Neural Networks*

Generative Adversarial Networks

# General

In the first lesson we learn about neural nets, implement gradient descent, train simple neural nets written from scratch and in Pytorch. Also we get a taste of sentimental analysis and complete our first project in this course.

## Introduction to Neural Networks

In this lesson, we go through the basics of neural nets: perceptron, error functions, losses, forward- and backpropagation. I’ll skip this section. Why? Because it is available to free to anyone. There is a free course by Udacity: Introduction to Neural Networks. Lesson 2 of this course contains this introduction, so you can go through it if you are interested.

## Implementing Gradient Descent

In this lesson, we go through the implementation of basic gradient descent.

Here is a basic explanation of what gradient descent is.

Now here is the list of videos of this lesson:

This is a sample code:

*# Defining the sigmoid function for activations*

**def** **sigmoid**(x):

**return** 1/(1+np.exp(-x))

*# Derivative of the sigmoid function*

**def** **sigmoid_prime**(x):

**return** sigmoid(x) * (1 - sigmoid(x))

*# Input data*

x = np.array([0.1, 0.3])

*# Target*

y = 0.2

*# Input to output weights*

weights = np.array([-0.8, 0.5])

*# The learning rate, eta in the weight step equation*

learnrate = 0.5

*# the linear combination performed by the node (h in f(h) and f'(h))*

h = x[0]*weights[0] + x[1]*weights[1]

*# or h = np.dot(x, weights)*

*# The neural network output (y-hat)*

nn_output = sigmoid(h)

*# output error (y - y-hat)*

error = y - nn_output

*# output gradient (f'(h))*

output_grad = sigmoid_prime(h)

*# error term (lowercase delta)*

error_term = error * output_grad

*# Gradient descent step *

del_w = [ learnrate * error_term * x[0],

learnrate * error_term * x[1]]

*# or del_w = learnrate * error_term * x*

import numpy as np

from data_prep import features, targets, features_test, targets_testnp.random.seed(21)def sigmoid(x):

"""

Calculate sigmoid

"""

return 1 / (1 + np.exp(-x))# Hyperparameters

n_hidden = 2 # number of hidden units

epochs = 900

learnrate = 0.005n_records, n_features = features.shape

last_loss = None

# Initialize weights

weights_input_hidden = np.random.normal(scale=1 / n_features ** .5,

size=(n_features, n_hidden))

weights_hidden_output = np.random.normal(scale=1 / n_features ** .5,

size=n_hidden)for e in range(epochs):

del_w_input_hidden = np.zeros(weights_input_hidden.shape)

del_w_hidden_output = np.zeros(weights_hidden_output.shape)

for x, y in zip(features.values, targets):

## Forward pass ##

# TODO: Calculate the output

hidden_input = np.dot(x, weights_input_hidden)

hidden_output = sigmoid(hidden_input)output = sigmoid(np.dot(hidden_output,

weights_hidden_output))## Backward pass ##

# TODO: Calculate the network's prediction error

error = y - output# TODO: Calculate error term for the output unit

output_error_term = error * output * (1 - output)## propagate errors to hidden layer# TODO: Calculate the hidden layer's contribution to the error

hidden_error = np.dot(output_error_term, weights_hidden_output)# TODO: Calculate the error term for the hidden layer

hidden_error_term = hidden_error * hidden_output * (1 - hidden_output)# TODO: Update the change in weights

del_w_hidden_output += output_error_term * hidden_output

del_w_input_hidden += hidden_error_term * x[:, None]# TODO: Update weights

weights_input_hidden += learnrate * del_w_input_hidden / n_records

weights_hidden_output += learnrate * del_w_hidden_output / n_records# Printing out the mean square error on the training set

if e % (epochs / 10) == 0:

hidden_output = sigmoid(np.dot(x, weights_input_hidden))

out = sigmoid(np.dot(hidden_output,

weights_hidden_output))

loss = np.mean((out - targets) ** 2)if last_loss and last_loss < loss:

print("Train loss: ", loss, " WARNING - Loss Increasing")

else:

print("Train loss: ", loss)

last_loss = loss# Calculate accuracy on test data

hidden = sigmoid(np.dot(features_test, weights_input_hidden))

out = sigmoid(np.dot(hidden, weights_hidden_output))

predictions = out > 0.5

accuracy = np.mean(predictions == targets_test)

print("Prediction accuracy: {:.3f}".format(accuracy))

## Training neural networks

I’ll skip this part also, as Lesson 2 of the aforementioned course also contains it.

## Project: Predicting Bike-Sharing Patterns

You can see the code for this project in the official repo: https://github.com/udacity/deep-learning-v2-pytorch/tree/master/project-bikesharing

This is a first project in this course.

In this project we build a neural net which will predict daily bike rental ridership. We have various data and it will be an input to our neural net:

We transform categorical variables using one hot encoding, normalize target variable and thus the data is prepared. Last 21 days are taken for validation.

We write the neural net as a class in the script. To tell the truth, it took me a long time to write in correctly because I messed up with the dimensions :)

If everything is done correctly, then the result will look something like this:

We can see that the predictions are better at the beginning and worse at the end. It seems that our net can’t work for cases when the target values become lower. Well, this is a very basic neural net, it isn’t surprising that it works only in a simple scenario. Still, it works! And so the project is completed.

## Sentiment analysis

Code is here: https://github.com/udacity/deep-learning-v2-pytorch/tree/master/sentiment-analysis-network

This is a lesson by Andrew Trask!

It was very exciting to me, because I still remember his article A Neural Network in 11 lines of Python — it was my first experience with neural nets and it took me several attempts to understand it. So I wondered what cool things would I learn in this lesson :)

Here we build a neural net for text classification. The lesson is split into 6 mini-projects. Let’s see how it is done!

Transforming Text into Numbers

Understanding Inefficiencies in our Network

Analysis: What’s Going on in the Weights?

In this video you could see the visualization of words in the embedding space. I had a different plot, let’s see:

The shape of the figure is quite different, but in essence it is the same:

Green color shows “good” words.

And black color shows “bad words”.

That’s it. I hope you liked this lesson as much as I did :) We implemented a lot of things from scratch, which is quite useful and did some fun things.

## Deep Learning with PyTorch

And this lesson is equivalent of lesson 4 in Introduction to Neural Networks.

So this was the first part of Deep Learning Nanodegree. It gave us fundamental knowledge of neural nets and PyTorch. Things will start becoming more interesting in the next part — CNN: transfer learning, style transfer, autoencoders and more!