Thanks a lot for this article. Unit tests (and more types of tests) are sometimes an underappreciated tool for software engineering in general and it's awesome to start sharing ideas of how to unit test such beasts as nn.
Have you tried keras? It fixes some of your issues "simply" by design, which is an even better way to avoid bugs IMHO. I'm thinking about the sequential model and the batchnormalization mainly. I still had to assert that my weights were not changing with gans, and obviously, they were at first.