On “Randomly Wired” and “Optimally Wired” Feed Forward Neural Networks — Part 2

Part-2 of this blog-post series is concerned with the use of an ensemble of randomly wired neural networks (RWNNs) to quantify prediction uncertainty.
For demonstration reasons a simple 1 dimensional case (figure 1) will be used (though the method described is general to any problem dimension).

Figure 1: Training data; A simple y=x² + noise training data set is generated and presented above

First step (sanity check) is generating a single RWNN suitable to the problem in hand; comprising of 1 input, 2 biases, 10 hidden and 1 output node (Figure 2) and training it on the generated data.

Figure 2: Initial RWNN architecture; with 1 input node (red) 2 bias nodes (purple) 1 output node (blue) and 10 hidden nodes

In Figure 3 a visualization of the trained RWNN can be seen as well as the results of testing the model in 1000 points spanning the training data input range.

Figure 3: Left; a visualization of the trained RWNN. Right; Testing the model on 1000 points spanning the training data input range (red line is the model prediction blue dots the training data)

RWNN Ensemble based uncertainty quantification

After performing the sanity checks above, its finally time to train an ensemble of (100 in this case) RWNNs aiming at capturing not only a “good” prediction but also, an at least usable, quantification of our models uncertainty.

To do so 100 RWNNs were generated, each with their one unique wiring and weight initialization. This RWNNs were subsequently trained on the, randomly selected, 90% of training data (borrowed for Random Forests). Figure 4 shows the prediction of all 100 RWNNs (red lines) together wit the training data (blue notes). Observe how the ensemble of models nicely spans the training data.

Figure 4: Predictions of 100 RWNNs (red) and training data (blue)

Figure 5 further analyzes the statistics of our ensemble. The thick green line demonstrates the P50 prediction. Light green shading the P10-P90 range and with red shading the Min-Max range.

Figure 5: Statistics of ensemble. Green Line is the P50 prediction. Light green shading the P10-P90 range and red shading the min-max range.

As a comment on the Interpolation/Extrapolation behavior of this models we can clearly see, as expected, that a number of models (figure 4) do really bad while trying to predict on regions outside of the training data input span (extrapolation regions). Using the ensemble of models this uncertainty can be correctly (and clearly) quantified (Figure 6). Observe how the min-max range drastically increases on the extrapolation regions.

Figure 6: Extrapolation (light yellow) vs Interpolation (light green) regions

Epilogue

We’ve seen that we can indeed use an ensemble of RWNNs to quantify prediction uncertainty.
Next post will initiate our investigation of Optimally Wired Neural Networks (OWNNs) instead of simple RWNNs.

P.S.

Below please find the representations of every RWNN in the esnemble trained in this post.

Thumbnail