Deep Learning Book Notes: Numerical Computation Part 1

Starting with it!

Because the post is incomplete without the glorious cover page!

Topic 1: Numerical Error

Underflow:

Subsequent problems which arise:

An example in Python:

Underflow in Python

Overflow:

An example of Softmax Function:

Case 1: c is a large negative number

An example in Python

Case 2: c is a large positive number

An example in Python

How to solve it?

  1. The largest argument in exp will be 0, ruling out the possibility of overflow.
  2. At least one term in the denominator has a value of 1, which rules out the possibility of underflow in the denominator leading to a division by zero.
  1. Defining Softmax Function in Python

Topic 2: Poor Conditioning

2. For Non-Linear single variable functions (Extra Stuff, can skip!)

An interesting example:

Why so?

3. For Non-Linear multi-variable functions (Extra Stuff, can skip! At the end of Part 2)

References

  1. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT press.
  2. https://github.com/roatienza/Deep-Learning-Experiments
  3. https://en.wikipedia.org/wiki/Condition_number#One_variable

--

--

--

The logic in Mathematics. The creativity in Art. The beauty in Code.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Tackling Climate Change with Machine Learning

FAIR Paper Review: A ConvNet for the 2020s

Evaluate Machine Learning Model Accuracy

“Utility is in the Eye of the User: A Critique of NLP Leaderboards” Paper Summary & Analysis

Use your eyes and Deep Learning to command your computer — A.I. Odyssey part. 2

Mapping Nomads with Machine Learning

What Is Machine Learning?

“The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks” Paper Summary & Analysis

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Shubhi Sareen

Shubhi Sareen

The logic in Mathematics. The creativity in Art. The beauty in Code.

More from Medium

Memory

A Stacked Ensemble Technique for DFU detection

Introduction to Neural Networks Part-1

A comprehensive introduction to GNNs — Part 3