Deep Learning in Water Resources

Part 1 of 3

Here, I present the first part of 3 posts that aim to provide a fast start to Deep Learning. Feel free to jump in if you want direct hands-on examples, Part 3, but if you want to understand how the model is set up and details about the reasoning of the algorithm, then Parts 1 and 2 would be interesting.

Part 1. Theory

Part 2. Building models

Part 3. Hands-on example

Here, I will cover the theoretical basic concepts needed to apply DL in modelling water resources.

Due to the large variety of network architectures in deep learning, the focus will be on MLPs, convolutional neural networks, and LSTM. This work deals directly with all the elements needed to run one example on the code and understand the parameters used along the example. For this, we will use TensorFlow in Python. In general, even if you’re starting with a minimal background in math or programming. You’ll learn to navigate these tools through practical examples and hands-on exercises and contribute meaningfully to solving water-related issues.

How to learn the use of Deep Learning

Understanding how to use a Deep Learning model sometimes appears complex due to the terminology and level of evolution in algorithms solving matrices of high dimensions (Figure 1).

Definition

Deep learning is a subset of machine learning methods based on artificial neural networks (ANNs) with representation learning. The term “deep” refers to using multiple layers in the network, which can be supervised, semi-supervised, or unsupervised. Deep-learning architectures, such as deep neural networks and convolutional neural networks, have been applied across various fields, achieving results comparable to or surpassing human expert performance in some cases. For a detailed definition and further information, you can visit Wikipedia’s page on deep learning (https://en.wikipedia.org/wiki/Deep_learning).

Origin of the Definition: Deep learning is not new; it draws inspiration from the artificial neural networks designed in the mid-20th century. However, the term “deep learning” gained prominence through the work of Geoffrey Hinton and colleagues, who applied it to large-scale neural networks capable of feature detection through deep layers of processing.

Deep learning models come in many different shapes and sizes, with convolutional neural networks (CNNs) used for image recognition and recurrent neural networks (RNNs) for sequential data like time series or natural language. More recently, transformers have shown great promise for understanding the context of text data.

The beauty of modern deep learning is its accessibility in many open libraries. Libraries such as TensorFlow and Keras offer pre-built models and easy-to-use interfaces that allow you to apply complex models to your data with just a few lines of code. These sessions will guide you through using these tools effectively, enabling you to apply deep learning to tackle pressing water-related challenges with minimal technical background.

Evolution

The terms artificial intelligence is quite common nowadays and since seems an overarching terminology, everyone use it even if is Machine learning or Deep Learning. However, if you look at the evolution and start to dig a bit into the concepts, there are important differences.

Source: NVIDIA difference between Machine Learning and AI

Artificial Intelligence (AI): This is the big umbrella term. AI is all about creating machines or software that can think and learn like humans. Imagine you have a smart robot friend who can solve puzzles, play games, and maybe even hold a conversation — that’s AI in action. It’s not just about mimicking humans, though; it’s about enabling machines to handle tasks that typically require human intelligence, such as understanding natural language or recognising objects in a photo.

Machine Learning (ML): This is a subset of AI. If AI is the goal, machine learning is one of the methods to get there. Think of ML as teaching your computer to learn from data without having to programme it with specific rules. For example, instead of teaching your computer every single rule to differentiate between cats and dogs, you feed it thousands of cat and dog pictures, and it starts to figure out the differences on its own. So, machine learning is all about the computer learning patterns and making decisions based on data.

Neural Networks (NN): These are a bit more specific, still under the machine learning umbrella. Imagine your brain as a network of synapses connecting neurons. This biological network is the model for neural networks in computing. They’re like a web of simple, interconnected units (think of them as digital neurons) that process data. By adjusting the connections between these units, the network learns to perform tasks, like translating languages or recognising objects in pictures.

Deep Learning (DL): This is a specialised type of machine learning. You can think of deep learning as “deep” machine learning because it involves more layers of these digital neurons. The “deep” part refers to the number of layers through which the data is transformed. More layers allow the network to learn really complex patterns. This is what powers the most advanced AI tasks, like interpreting complex speech or identifying objects in an image with superhuman accuracy.

So, in a nutshell: AI is the big dream of machines thinking like humans. Machine learning is a way to achieve AI where machines learn from data. Neural networks are a structure used in machine learning, kind of mimicking how our brains work. And deep learning is a more advanced version of this, with deeper layers of neural networks, enabling even more complex learning and problem-solving.

Modelling Concept

Regardless of whether we’re discussing Machine Learning, Neural Networks, or Deep Learning models, they all stem from the same motivation: to learn from data. Essentially, instead of crafting a specific programming algorithm, we aim to design a system that can ‘learn’ by itself. This means setting up the machine with parameters in such a way that it can understand and respond to data based on the input and output we provide. In other words, we’re not just programming it in the traditional sense; we’re teaching it to make sense of data and make decisions, mimicking the way we humans learn.

As the concept of machine learning has evolved and been adapted to other fields of science, it has become clear that machine learning models were not intended to replicate phenomena. Many have criticised the idea of building a machine learning model for an area of science where we already understand most of the physical principles. It seems unnecessary to build a model if we already have a solid physical equation. Additionally, it is evident that understanding physics is more valuable than having a black box that reproduces a couple of linear equations. The figure below shows how machine learning was initially used for knowledge discovery, then improved or extended forecasting, the estimation of uncertainty, and then tracking stochastic behaviour of phenomena, among other problems. In general, we now build the model, check what will happen in different scenarios, and then try to understand the logic and the way the system can be extrapolated or not into other situations.

Use of machine learning in applied sciences

Types of Artificial Intelligence (AI) Algorithms

Understanding AI algorithms can be approached similarly to how one categorises water models. Just as water models are classified based on their purpose, structure, and the processes they simulate, AI algorithms are grouped by their learning paradigms, network architectures, problem domains, inference methods, and adaptability. This categorisation helps clarify the diversity and utility of these algorithms in solving various problems, much like how different water models are chosen based on the specific hydrological process being simulated.

Taxonomy Based on Learning Paradigms

In the same way water models can be classified by the type of processes they simulate — such as surface runoff, groundwater flow, or water quality — AI algorithms are categorised by how they learn from data some people would say the way they are calibrated.

• Supervised Learning: This is a kind to the calibration process in a hydrological model, where the model is trained using historical data with known outcomes (like river discharge data) to predict future behaviour. Algorithms such as Support Vector Machines (SVMs) or Decision Trees adjust their parameters during this training process, similar to how a water model adjusts its parameters to fit observed data.
• Unsupervised Learning: This parallels exploratory data analysis in water modelling, where you might examine patterns in a dataset without predefined outcomes, much like identifying clusters of similar hydrological behaviours in a basin. K-means clustering, for example, groups data points into clusters without prior labels, like grouping catchments based on their hydrological responses.
• Semi-supervised Learning: Imagine a situation where you have sparse monitoring data in a catchment. Semi-supervised learning combines the limited observed data (labelled data) with a larger amount of unobserved data, similar to how hydrological models might interpolate or infer missing data points from sparse monitoring networks.
• Reinforcement Learning: This is like running multiple scenarios in a water management model where the model ‘learns’ the best strategy by trial and error, such as optimising reservoir operations to maximise water supply while minimising flood risk. The algorithm adjusts its strategy based on feedback, much like a model might refine its operation rules based on simulated outcomes.

Taxonomy Based on Network Architecture

The structure of an AI algorithm can be likened to the architecture of a water model.

• Feedforward Neural Networks (FNNs): These are like simple rainfall-runoff models where input data (e.g., rainfall) passes through the model to produce an output (e.g., runoff) without feedback loops. The process is straightforward, much like how FNNs process data in one direction.
• Convolutional Neural Networks (CNNs): These networks are kind of spatially distributed hydrological models that capture complex spatial relationships, such as how rainfall distribution over a watershed influences runoff. CNNs similarly process data to detect spatial patterns, such as identifying features in images.
• Recurrent Neural Networks (RNNs): RNNs are comparable to models that deal with time series data, like a river flow model that uses previous states (e.g., past flow rates) to predict future states. These networks have memory elements, much like how state variables in a hydrological model retain information about past conditions.

Taxonomy Based on Problem Domain

Different problem domains in AI correspond to different applications in water modelling.

• Natural Language Processing (NLP): In water resources, this could be likened to interpreting and synthesising large volumes of textual data, such as reports or research papers, to extract meaningful insights—similar to how NLP models process and generate human language.
• Computer Vision: This is similar to remote sensing applications in hydrology, where algorithms interpret satellite images to identify water bodies, land cover changes, or flood extents. Computer vision models analyse visual data to detect patterns, much like how remote sensing tools are used to monitor and analyse water systems.
• Metaheuristic Optimisation Algorithms: These are similar to algorithms used in water resources management to find the best operational strategies for reservoirs, allocation of water resources, or design of hydraulic structures. Algorithms like Particle Swarm Optimisation (PSO) or Genetic Algorithms search for optimal solutions, similar to how optimisation techniques are applied in water management.

Taxonomy Based on Inference Methods

Inference in AI can be compared to how water models make predictions or estimations based on different methods.

• Probabilistic algorithms: These are similar to stochastic hydrological models that account for uncertainty in inputs or model parameters, providing probabilistic forecasts. Algorithms like Naive Bayes operate under uncertainty, similar to how probabilistic models estimate the likelihood of different hydrological outcomes.
• Deterministic Algorithms: These correspond to deterministic hydrological models, which produce a specific outcome given a set of inputs, such as how a rainfall-runoff model predicts streamflow based on a known rainfall event. Decision Trees and k-Nearest Neighbours (k-NN) operate deterministically, producing explicit outcomes based on the input data.

Taxonomy Based on Adaptability

Adaptability in AI algorithms can be compared to how hydrological models might adapt to new data or changing conditions.

• Static algorithms: These are like traditional hydrological models with fixed parameters and structures that don’t change over time. They produce consistent results given the same inputs, but they don’t adapt to new data or changing conditions.
• Dynamic algorithms: These are more akin to models that can adjust or recalibrate as new data becomes available, such as real-time flood forecasting models that update predictions based on the latest rainfall data. Most machine learning algorithms fall into this category, continuously learning and improving over time.

Modelling

In many situations, it is best to start with a concept when using AI and machine learning. Learning the fundamental concepts underlying the model is a good first step towards comprehending the plot of the entire story. This I felt was appropriate to discuss in part two.