An Introduction to AdaGrad

We have discussed several algorithms in the last two posts, and there is a hyper-parameter that used in all algorithms, i.e., the learning rate (η). To refresh again, a hyper-parameter is a parameter that has to be chosen manually before training. The prefix ‘hyper-‘ is to…

Cross-entropy and Maximum Likelihood Estimation

Jenkins for Node.js App on AWS EC2 Part 3: Jenkins — Node.js App Integration

Curious with Jenkins…

Momentum Method and Nesterov Accelerated Gradient

In the previous post, Gradient Descent and Stochastic Gradient Descent Algorithms for Neural Networks, we have discussed how Stochastic Gradient Descent algorithm can improve plain Gradient Descent algorithm in terms of complexity and…

Jenkins for Node.js App on AWS EC2 Part 2: Creating a Node.js App

Create Node.js App to be deployed…

Jenkins for Node.js App on AWS EC2 Part 1: Installing Jenkins on EC2

What are we going to build?

In this article series, we will build a simple Node.js app that hosted on AWS EC2 instance. Furthermore, we are going to configure…

Understanding Dropout

One particular layer that is useful, yet mysterious when training neural networks is Dropout. Dropout is created as a…

Continuing on Adaptive Method: ADADELTA and RMSProp

In our last post, we have discussed the difficulties of setting the learning rate hyper-parameter, which can be mitigated by using the Adagrad algorithm. However, this algorithm has a serious flaw, i.e., the effective learning rate…

Modifying Adam to use Nesterov Accelerated Gradients: Nesterov-accelerated Adaptive Moment Estimation (Nadam)