Matthew Phelps
Sep 3, 2018 · 2 min read

I didn’t read the rest of your comment. Your first argument against the article rests on the fact that your Google query of “continuous subset of the program space” did not yield any meaningful reference. That turns out to be a terribly weak argument and is not a valid critique.

Why? Firstly, because the author is freely expressing his own mathematical representation of what a neural network does, at it most abstract operation. I do not know your background, but if you spend any significant time in mathematics and science, you will begin to understand that nearly all processes and interactions can be modeled mathematically, and the mathematical description of a process is not bound to a single representation. In physics, for example, one may formulate quantum mechanics via the Schrodinger picture or the Heisenberg picture — they are both isomorphic.

Secondly, arguing upon the basis that if something is not “Googleable” then such a concept is not valid, is complete nonsense. You would be surprised how very little you have to advance towards expertise in a topic before Google’s abilities to help you become significantly limited. More often than not, your best hope is to find a research paper that is remotely related the idea you are pursuing.

Finally, my take on “continuous subset of the program space”. Continuous — as in the resolution of the space of possible solutions is nearly infinite (I imagine the weights may in principle occupy any real number, but are probably only precise up to the machine’s precision). Subset — we will always be bound to be within some subset of possible programs. To not be in a subset would suggest we are searching within the entire space of possible program solutions, which is implausible for a number of reasons. And lastly, program space- those two words alone do not provide much mathematica rigor, but we have no need to expect such. This is really just a most general, abstract description of the computational space spanned by all possible neural networks of a given dimension and configuration (layers and cost function).

    Matthew Phelps

    Written by