Become a member
Sign in
Rajneesh Soni
Rajneesh Soni

Rajneesh Soni

10 Following
1 Followers
  • Profile

  • Claps

  • Highlights

Highlighted by Rajneesh Soni

See more

From Yes you should understand backprop by Andrej Karpathy

Another non-obvious fun fact about sigmoid is that its local gradient (z*(1-z)) achieves a maximum at 0.25, when z = 0.5. That means that every time the gradient signal flows through a sigmoid gate, its magnitude always diminishes by one quarter (or more). If you’re using basic SGD, this would make the lower layers of a network train much slower than the higher ones.

Claps from Rajneesh Soni

See more

Yes you should understand backprop

Andrej Karpathy

VMAF: The Journey Continues

Netflix Technology Blog

FileNation, but what about the basics?

David Redondo