Deep Learning in Water Resources

Published in

Hydroinformatics

7 min readMar 22, 2024

Part 2 of 3

Building Models

It is common to need clarification on where to start, how my problem fits a model or how a model can be identified to solve it. Most of the time, experience plays a role, which starts with paper publications. You will see that the key questions are

“What is the model for”
“Which physical laws are important to understand the problem”

With this information, we can start to build a combination of input and outputs, but we can also understand the boundaries and what to expect and look for in terms of success and failure. This will help us to know what to validate in the model. In deep learning, the algorithm is considered a feature learning, which implies little work or less work is done in identifying the variables that could be very important in the system. In deep learning, it is still necessary to do some preprocessing and work on setting the possible training elements. From these two questions, it is possible to derive more important questions: “What accuracy is required?” and “When to stop training?”.

I suggest the following links to start to understand more the models and be able to have a simple and straightforward start.

Intro to Machine Learning (ML Zero to Hero — Part 1) — YouTube

Say hello to the “Hello, World” of machine learning (google.com)

Learn Python, Data Viz, Pandas & More | Tutorials | Kaggle

Learn Feature Engineering Tutorials (kaggle.com)

Neural Network Architectures & Deep Learning — YouTube

To understand this building model process, we can think about a normal model; in water resources, we have conceptual process-based models, like hydrological tank models or hydraulic models. In general, a mathematical equation has some parameters to represent a problem. An easy way to see it is like a tank model. You can see an excellent example in the work of Mostafa Farrag:
HBV lumped conceptual hydrological model | by Mostafa Farrag | Hydroinformatics | Medium
For this type of model, you need to optimise and find the proper parameter testing. The same principle applies to machine learning and deep learning models, just that the equations of the model are adaptable. Thus, we can say that part of the optimization process is the identification of the proper model or structure of the model (Hyperparameter optimization).

The diagram illustrates the process of building a machine learning (ML) model, specifically focusing on how it’s trained to simulate a natural system based on measured data. Here’s how it works:

Input and Natural System Interaction:

Measured Forcing (X): These are the input variables or conditions applied to the natural system(Vector). For instance, in a water resources model, this could include rainfall, temperatures, and other climatic factors.
Natural System: This represents the real-world system you’re modeling, like a watershed or a river basin. The system responds to the measured forcings in complex ways.

2. ML Model and Its Outputs:

ML Model f(θ,…): The machine learning model, defined by a set of parameters (θ), tries to mimic the natural system’s behavior. It takes the measured forcings as input and produces outputs or predictions (P).
ML Output (P=Predictions): These are the model’s predictions of how the natural system responds to the input forcings, such as simulated water levels or flow rates in response to the input climatic conditions.

3. Comparing Model Outputs to Reality:

Measured Response (Y): The actual, observed responses of the natural system to the input forcings. For a water resources model, this could be the actual measured water levels or flow rates.
Error Analysis E=D(P,Y): This step involves comparing the model’s predictions (P) with the actual, measured responses (Y). The difference function (D) calculates the error (E) between the predictions and the real measurements. This error quantifies how well the model simulates reality.

4. Training the Model:

Training: Based on the error analysis, the ML model’s parameters are adjusted to minimize the error (E). This process involves using results from error analysis to update the model’s parameters, aiming to reduce the difference between the model’s predictions and the actual measurements.
The model might undergo calibration (adjusting parameters to better match observed data) or use reinforcement learning (a technique where the model learns to make better predictions through trial and error).

A cyclical process of building and refining a machine learning model to accurately simulate a natural system is a very common representation in water resources. The goal is to continually improve the model’s accuracy by minimizing the discrepancy between its predictions and actual observed data from the natural system.

Uncertainty present in the model building

This diagram illustrates the various uncertainties in building a model, specifically focusing on the stages where these uncertainties can manifest. It breaks down the process into four primary areas of concern:

Input Uncertainty:

This pertains to the data used to feed the model. The key questions here are about the relevance and interpretability of the input variables. What if the model is being fed data that isn’t actually useful for making predictions? Or what if the data is difficult to understand or interpret in the context of the model?

2. Sampling Uncertainty:

This relates to the variability in the data samples used during different stages of the model’s lifecycle, such as training, verification, and testing. It questions the stability of the model’s performance if the samples it’s trained on were to change. Would the model still perform reliably if fed a different subset of data?

3. Model Calibration Uncertainty:

Calibration involves adjusting model parameters to improve its accuracy based on known outputs. This section asks what would happen if the starting conditions for the model were altered or if a different method for searching the best parameter settings (optimisation) was employed. How significantly would these changes affect the model’s performance?

4. Model Performance Uncertainty:

This final area deals with the evaluation metrics used to gauge the model’s effectiveness. It raises questions about the impact of changing these metrics or incorporating multiple performance measures. How does altering how we measure the model’s success influence our perception of its accuracy or usefulness?

The fluidity and complexity of modelling seem complicated when you consider how changes at any stage in the process can introduce uncertainty. Addressing these uncertainties is crucial for developing robust, reliable models.

Some algorithms, structures, and explanations

Slide from the presentation of what is deep learning (IHE Delft Lectures)

Many algorithms can be used in Deep Learning, but depending on the type of problem, it is easy to see that some apply more to specific type of problems:

Time series forecasting and large language models (LLM), which in water resources are forecasting rain and flows in a river basin.

For LSTM:

How the LSTM input is prepared
How the LSTM model receives the first input
How the second layer connects
How all the layer are interconnected at the end of the process
What should be consireded during the exercise
How the DropOutone works
How the regularization works
How the performance measure work
How the LSTM works for Classification and for Regression.

Long Short term memory networks (LSTM), Generalize Adaptive Networks (GANs), overal eplanation can be found here:

What is LSTM (Long Short-Term Memory)? — YouTube

Deep Learning: Long Short-Term Memory Networks (LSTMs) — YouTube

Illustrated Guide to Recurrent Neural Networks: Understanding the Intuition — YouTube

C5W1 Recurrent Neural Networks — YouTube

Tips and tricks to train LSTM

https://danijar.com/tips-for-training-recurrent-neural-networks/

GRUs A step by step explanation

Crash Course in Recurrent Neural Networks for Deep Learning — MachineLearningMastery.com

A Crash Course in Sequential Data Prediction using RNN and LSTM | by Asel Mendis | bitgrit Data Science Publication | Medium

Illustrated Guide to LSTM’s and GRU’s: A step by step explanation | by Michael Phi | Towards Data Science

GitHub — E3-JSI/ml-rapids: Fast implementation of incremental learning methods (C++), which are exposed in Python and NodeJS.

GitHub — E3-JSI/FASTENER: Feature selection genetic algorithm implemented in Python.

Recurrent Neural Network Guide: a Deep Dive in RNN — neptune.ai

Applications in Python

Time Series Forecasting With RNN(LSTM)| Complete Python Tutorial| — YouTube

Multivariate Time Series Modeling using Facebook Prophet — YouTube

#lstm #timeseries #multivariate LSTM Multivariate Time Series Forecasting in Keras — YouTube

Pytorch

Pytorch Tutorial — Setting up a Deep Learning Environment (Anaconda & PyCharm) — YouTube

PyTorch Tutorial — RNN & LSTM & GRU — Recurrent Neural Nets — YouTube

Pytorch Bidirectional LSTM example — YouTube

Understanding the units in the LSTM

Tung website — Units in LSTM (tung2389.github.io)

Counting No. of Parameters in Deep Learning Models by Hand | by Raimi Karim | Towards Data Science

Animated RNN, LSTM and GRU. Recurrent neural network cells in GIFs | by Raimi Karim | Towards Data Science

Vision pattern recognition or in water resources pattern in space. Convolutional neural networks (Explained here), Autoencoders (here), U-Nets (Here)

Requires for Convolution

What is a kernel
How the pooling works
How it shifts the kernel
How the next line received the size of the previous line
How the model uses a parameter to store each iteration
How the model reach a final MLP
How the final MLP is trained
Where does the RLU should go and how it looks in the Convolution

How Convolution Works — YouTube

Convolutions in image processing | Week 1 | MIT 18.S191 Fall 2020 | Grant Sanderson — YouTube

Machine Learning Zero to Hero (Google I/O’19) — YouTube

A friendly introduction to Convolutional Neural Networks and Image Recognition — YouTube

How convolutional neural networks work, in depth — YouTube

Practice

A Neural Network Playground (tensorflow.org)

Spatiotempora models with CONV LSTM

I think when the problem goes to integrating space-time features, the combined ConvLSTM is quite powerful. An example of this application can be found in this paper Spatiotemporal convolutional long short-term memory for regional streamflow predictions — ScienceDirect.

These are some sources of code that are very interesting for applying this algorithm: