Learning parameters of 3D simulations

I had this brief go at some data provided by http://ailivesim.com/. AIliveSim built a simulation tool to train and test agents in very realistic 3D environments.

I was really impressed by some of their maritime simulations. Snapshots from their simulated 3D environments look really real. This includes realistic looking sea surfaces with waves, as well as reflexes and eventual splashes of water on the camera lens.

Snapshot of a simulation taken at daytime.

Snapshot of a simulation taken at dusk.

Predicting parameters out of a picture

There are many parameters that define a particular simulation. You can see some of them below:

[SeasonDefaultSettings]
FoliageDensity=1.707774
Season=Summer
EnableRain=false
Randomize=true
Rain=NoRain
EnableLeaves=false
GroundWetnessType=Dry
…

Among these parameters, there exists numerical ones such as: time of the day, amount of fog, amount of rain, sun brightness, etc. I decided to analyse if predicting any parameter out of a single picture is possible at all. This might be useful to estimate parameters for a new simulation from any real picture. I started with predicting what time it is out of a single picture.

I was provided with a data set of images taken from simulations set at different times: from 5am to 6pm. For each time, there is roughly the same amount of pictures. In other words the target label is uniformly distributed. This might make training a classifier easier.

Transfer learning on images

I found this handy tutorial for transfer learning using Keras and Tensorflow 2.0: https://www.tensorflow.org/tutorials/images/transfer_learning. In this tutorial, transfer learning is used to customise a model pre-trained on ImageNet. Their target task is discriminating between “cat pictures” and “dog pictures”. Similarly, I used the pre-trained model to discriminate pictures taken at day-time against pictures taken at dawn or dusk. My proxy for day-time is “pictures taken between 11am and 3pm”. More specifically, my binary classification problem has two classes:

• Class 1: pictures taken between 11am and 3pm;
• Class 0: pictures taken between 5am and 11am, as well as pictures taken between 3pm and 6pm.

The main modification to the notebook above consisted in loading my own images from Google Drive. To this end, I made a script to resize images to 160x160 pixels as well as to add their label to the file name. For example, d4321a23_raw_13.0.png is the output of a resized image taken at 1pm. I made the script available here: https://github.com/simoroma/PythonMLUtils. Once images are resized to 160x160, they do not occupy much space and can be conveniently transferred to Google Drive. I also modified the notebook above to load images directly from Google drive. This is available here: https://github.com/simoroma/TransferLearningOnImages. You might try it out on Google Colab (just click on Open in Colab) if you provide your own images. Google Colab offers also basic GPUs for this type of experiments.

I used 1000 images for training and 500 images for validation. Images in the training set are obtained from different simulations than the validation set. Validating results on a different set of simulations should be a better estimator of the generalisation ability of the classifier.

Initial results

The plot on the top shows training and validation accuracy for each epoch.

Results from the first epochs, are obtained using the pre-trained network for feature extraction, the second part of the plot shows instead results with fine-tuning.

Feature extraction: the pre-trained network is not re-trained for this task but just used to extract features. A classifier trained on the extracted features is added on top of the network to customise it to our specific task.

Fine-tuning: the whole network, or some parts of it, are fine-tuned using the new images.

We see that the prediction accuracy on the training set increases quite a bit when we start fine-tuning. On the other hand, validation accuracy increases slowly. The task of predicting time of the day seems a challenging one for the network, nonetheless the model is definitely picking up some patterns to predict it.

Insights

This is just a starting point for analysis. I have just a few initial insights to share before continuing with in depth analysis:

• Defining Class 1 as “pictures taken between 11am and 3pm” seems to help. Other settings consists in choosing a different time range. Nonetheless the model seems to perform worse;
• I tested also another set of images with less fog and sea colour set to constant. This does not seem to actually improve the model;
• Of course it would be better to treat this task as a regression problem. The network could learn time of the day as a numerical value. It it not too difficult to change the notebook for this purpose;
• As previously said, the simulations in the validation set are not the same as the ones in the training set. Using the same simulations did not really help;
• There is quite high variability in the accuracy results between different runs on the same notebook. I did not expect this much variability and this is definitely something to look into;
• We can use model.evaluate to double check the performance of our model on the different sets. The results do not seem to match the output of model.fit though. I didn’t dig into it, but it still seems to be an open issue in Keras: https://github.com/keras-team/keras/issues/6977.

Summary

I showed a preliminary analysis about extracting parameters out of a single picture. Pictures were obtained from 3D simulations by AIliveSim. It seems definitely possible to learn parameters out of pictures if transfer learning is used.

The following scripts could also be a good starting point to apply these techniques to your own images:

• Python scripts to resize images and add their labels the their file name https://github.com/simoroma/PythonMLUtils;
• The notebook I used in Colab to load images from Google Drive: https://github.com/simoroma/TransferLearningOnImages;
• The original Google tutorial on transfer learning for images: https://www.tensorflow.org/tutorials/images/transfer_learning.