A Reality Checklist for your Deep Learning Project
Where is Deep Learning applicable? This is one of the more fleeting ideas to understand about Deep Learning and related A.I. technologies. It is all too easy to fall in the trap that a “Artificial Intelligence” application can solve your problem.
The usual coverage of this problem involves the question of “do you have enough data?” Unfortunately, that is too vague in that to answer this you have to at least understand your problem domain. In the academic sense, you want to understand the “boundary conditions.” Said differently, you want to understand the intrinsic constraints of your problem. What exactly are “boundary conditions” of Deep Learning or A.I. learning problem?
Earlier, I wrote about “What is Knowable” and “What is the current state of knowledge?”. To understand a problem domain, you have to understand what the current state of knowledge is. Uncertainty is a measure of what is unknown relative to what is knowable. The strange assumption of the word uncertainty is that it assumes that everything can be eventually known. That is we assume determinism, in the real world this is rarely ever the case.
Nevertheless, it is a useful term that we will use to identify to boundaries of a problem. Here is the checklist. It is written in the form of questions and a answer of “no” implies the existence of uncertainty. For some odd reason, I can not phrase it in the other way that makes it simple! Contrapositive statements always seem complicated to parse! I will also use the terms actors, activity and environment to describe the entire context.
Execution uncertainty — Does the sequence of actions of the actors, from the environment’s initial state, always lead to the same final state?
Observational uncertainty — Do actors have complete information of the environment ?
Duration uncertainty — Do the actors know how long the activity will last?
Action uncertainty — Are the effects of the actor’s actions known exactly?
Evaluation uncertainty — Is there an evaluation criteria to measure the successful completion of the activity?
Training uncertainty — Is there knowledge or data of previous successful solutions that can be used as a guidance to learning?
In most real world problems, the answers to these question are most likely to be in the negative. Automation requires that a majority of the above be answered in the affirmative. Deep Learning based automation however equips the practitioner with a little more wiggle room here. The nature of Deep Learning automation is that they are approximation machines. However, DL can only learn a good approximation of an uncertain situation if there exists information that already removes the uncertainty (i.e. supervised learning).
No machine can know the unknowable (the unsupervised problem needs to be solved first). More specifically, systems learn by using knowable knowns to increase certainty of knowable unknowns. The checklist above guides one to identify what is unknown and thus is a list of “known unknowns”. Let’s be real here, you can’t solve a hard problem if you don’t know what you don’t know! Strive to break that “reality distortion field” by identifying exactly what not only uncertain but also what is unknowable in your problem domain.
Deep Learning cannot do magic even if it is practiced like alchemy. However, understanding the fuzzy boundaries of what is reality and what is science fiction can give you a tremendous leg up from the competition.
This list explains the success of DeepMind’s AlphaGo as explained by Andrej Karpathy in his previous post “AlphaGo in Context”. This post was inspired by his analysis. The above checklist is applicable for systems that interact with its environment. An example of this is AlphaGo, however most Deep Learning systems only need to perform classification.
