Building Data Science Capability at UKHO: our January 2022 Research Week
This blog was co-authored by Dr Susan Nelmes, Andrew Smith, Jiri Klic and Dr Thomas Redfern.
Hello and welcome to our first data science roundup of 2022! I’m here as your guide to our regular quarterly blog post in which we share our ‘research week’ findings, as we explore the interesting bits of data science and machine learning. This month, the team look at digital twins, platforms for model evaluation, detecting subtle changes in images and finding structure in unstructured data.
Firstly, what is a research week?
Members of the Data Science team at the UKHO come up with excellent ideas for how we can apply the latest in machine learning or big data research to the marine data domain. Sometimes these ideas are outside the scope of our current project deliverables so, to push the boundaries of what is possible, every quarter we conduct a ‘research week’ — an allocated time for data scientists to experiment with new ideas, techniques and technologies.
In this blog, we outline our work, how it might tie into our future projects, and how the lessons learned will help with future projects.
Andrew Smith: Qualitatively evaluating computer vision models
Developing systems that efficiently utilise any machine learning capability is often quite a complex process. It typically involves many steps like defining a problem to be solved, collecting and cleaning data, training a model, deploying the model to production and monitoring performance. Each of these steps themselves can be quite complicated with many moving parts. Consequently, evaluating and debugging a machine learning system can be quite a time-consuming process, with compute vision problems often being additionally cumbersome given the nature of the data. Fortunately, there are many open-source tools that are available to help us with this process — the openness of data science communities is one of the key reasons I believe this field moves so fast, and I am very grateful to these for all the fantastic tools (and associated research, documentation, tutorials, blogs, videos,…).
PyViz is a very useful resource for learning about a large number of python based visualisation tools. These tools are useful for a wide range of purposes including geospatial visualisation. I have found many of the libraries listed here very useful for building more custom applications regarding qualitatively evaluating machine learning pipelines (computer vision and otherwise).
Another tool I have recently come across is FiftyOne. This tool is more specific to the purpose of evaluating computer vision models and building high-quality data sets. It has many useful features like the ability to visualise (and validate) labelled data, to add tags to images (e.g. to mark images that are labelled incorrectly), to sort and filter images by some numeric score (which is very helpful, for example visually comparing the most correct and most incorrect images), functions to generate a wide range of plots of your data (including to create interactive geospatial plots, see image below), the ability to explore a number of large publicly available data sets like the Open Images Dataset, and much more.
Dr Susan Nelmes: Spot the difference
My research week involved looking into the question of how we detect changes between images. In particular, by comparing screenshots before and after a code change in software production, can we check if it results in undesirable changes to the visual layout of its user interface.
A pixel-by-pixel comparison may seem the obvious choice, but differences in the way images are smoothed and rendered by different systems can result in changes that we might not want to count as a test failure if the essential content and layout remains the same.
The structural similarity index measure (SSIM) tries to better mimic how humans would assess images by identifying the visual information gained from the image. Applying this to a screenshot where I have modified the position of one of the buttons, shows it identifies the change, shown in cyan, easily.
Image hashing algorithms are another method for comparing the similarity of images. Normally applied to whole images, I investigated gridding an image up and applying them to individual parts. The algorithms reduce an image, or part of it, down to a hash which can then be compared with another image hash, with similar hashes implying similar images. Using this, parts of the screenshots can be assessed for changes. To do this I used the Python package ImageHash.
Finally, I also looked at feature detection, descriptors and matching. A good explanation can be found here. Both the Python packages opencv and scikit-image offer functions for this.
All these methods worked well on a simple modification to the layout, but further investigation is required to see if they could be optimised to produce good results when differences in smoothing and rendering are applied to the screenshots.
Dr Thomas Redfern: Finding structure in un-structured data
In this research week I investigated the use of Auto-encoders and dimensionality reduction techniques to find structure in un-structured imagery data. Imagine the scenario where you have a large volume of unlabelled imagery data and limited resource available for labelling. What’s the best way of splitting the imagery data up so that you can spread your labelling effort across different types of images? Can we find structure in the imagery, to best design a labelling approach that efficiently covers the variety and complexity within the imagery data?
An autoencoder is a machine learning algorithm that is trained to reproduce the input as its prediction. An autoencoder “learns” a representation of the input image, that is simpler than all of the input pixels. It finds structure. I trained a fully convolutional neural network autoencoder (based on the UNET architecture) to reproduce the MNIST data set (a series of 28x28 pixel images of handwritten digits). I then used the TSNE dimensionality reduction algorithm to produce a 2-dimensional plot of the features that the encoder had learnt. Without any labels to help, I wanted to see if the autoencoder was able to tell the difference between different hand drawn digits. This was the result:
As Figure 1 shows, there is a good degree of separation between most of the different hand drawn digits with some overlap between 5s and 3s and 9s and 4s. The next steps of this work would be to transfer this process to satellite imagery, a topic that is being studied by our sponsored PhD project at the University of Swansea.
Jiri Klic: Digital twins
The digital twin concept has been known for over two decades and is used primarily in engineering and manufacturing. By creating a detailed digital replica of a physical object, we can simulate its behaviour under different conditions. This process can provide valuable insight that would be too difficult or dangerous to obtain with real-life experiments.
There is an increasing trend to use digital twins to simulate and analyse complex interconnected systems. This approach can be instrumental in supply chain management and maritime shipping. By using historical AIS data to model the global marine traffic, we could use a digital twin to run simulations of different scenarios that never even happened. Combining it with other data science techniques like multi-agent simulation or reinforcement learning can be a promising approach for voyage optimisation and analysis of unprecedented events like the Suez Canal obstruction in March 2021.
During my initial investigation into the subject, I came across two large scale implementations of digital twins on a national and international level:
The National Digital Twin
The National Digital Twin programme (NDTp) aims to create a family of digital twins that could communicate via secure data sharing. It’s run by a partnership between the University of Cambridge and the Department for Business, Energy and Industrial Strategy in United Kingdom called the Centre for Digital Built Britain (CDBB). As part of NDTp, the CDBB published the Gemini Principles report in 2018, a collection of nine guiding principles to ensure the purpose, trust and practical functionality of interconnected digital twins. They also created the DT Hub, an online space for digital twin developers to collaborate and exchange experiences.
Destination Earth
Destination Earth (DestinE) is the initiative of the European Commission to develop the Digital Twin of the Earth to monitor and predict an environmental change from human impact and support sustainable development. DestinE will allow users to test various strategies to tackle climate change and explore their potential consequences. The key contributors to this project are ECMWF, EUMETSAT, and ESA. An inspiring project is the Digital Twin of Antarctica, developed in cooperation with the University of Edinburgh.
