Human or Vulcan: Should Data Scientists Rely on Intuition?
Daniel Tunkelang
181
Many data sets are too sparse to permit the development of models from scratch. However, they permit the verification of intuited models.
More specifically, a finite data set can only “curve fit” a surface in a certain number of dimensions ( and as we get closer to that maximal dimension of the model, the model becomes more sensitive to noise ).
However, if we had an oracle that told us the right dimensions to look at ( said another way : provided a kernel to reduce the dimensionality of the model ), then we could develop such a model.
Intuition is that kernel development process.