Debugging Neural Network code
A little about me. Fortunately my job requires me to move out of conventional deep learning libraries and write straight up code for deploying deep networks in languages like c and c++. And as simple as this process sounds, it is if you get it right in fast pass. There are multiple level of problems. Some of them are personal like not being a good enough c coder. The others are in part how lot of open source deep learning libraries are too encapsulate. And I am not complaining, if these libraries were too exposed. I would have never been able to experiment. This post is not about bickering, it’s about sharing experiences while converting code from one library to own code.
Know your library well: Half of my problems arise because I don’t know all the nut’s and bolt’s of the libraries I use. I play around with them at a very high level. While implementing new architectures I just go around fitting the right built in function at right place. I completely ignore the facts how one function interacts with another. What goes under the hood of each function, how is it optimized which in turn effects the value returned by each of them. And this has always burnt me. I live in delusions. Assuming stuff, excuse for this is that the native code is hidden inside. It took me two full days worth of work to find out what convolution code goes underneath. So before jumping in with all hands in you should go through the functions you use in your code. Like till the native c code. It will always save you time. Think like this if you write your code and then try to find out where it is going wrong, it’s going to be pretty tough.
Keep your network code modular and repeatable: So this is very important thing I have learnt. Use the same structure as your library. Have it simple. Easy to read and modular. This helps a lot while debugging. The good thing about lot of Deep Learning Libraries is that they have followed a very good hierarchy. This also helps to move code around.
Don’t Look at open source implementations: Sounds weird yes. They are good to look. May seem as any easy copy paste solution. But it never is. Trust me I have been burnt so many times because of this. You follow the author, his methodology and forget about your stuff.
Ask for help: The golden rule. All libraries out have great user group. People are more than willing to help in case you are stuck. So go and ask for help. No matter who you are one can always use some help.