Deep Learning in Water Resources
Part 2 of 3
Building Models
It is common to need clarification on where to start, how my problem fits a model or how a model can be identified to solve it. Most of the time, experience plays a role, which starts with paper publications. You will see that the key questions are
- “What is the model for”
- “Which physical laws are important to understand the problem”
With this information, we can start to build a combination of input and outputs, but we can also understand the boundaries and what to expect and look for in terms of success and failure. This will help us to know what to validate in the model. In deep learning, the algorithm is considered a feature learning, which implies little work or less work is done in identifying the variables that could be very important in the system. In deep learning, it is still necessary to do some preprocessing and work on setting the possible training elements. From these two questions, it is possible to derive more important questions: “What accuracy is required?” and “When to stop training?”.
I suggest the following links to start to understand more the models and be able to have a simple and straightforward start.
Intro to Machine Learning (ML Zero to Hero — Part 1) — YouTube
Say hello to the “Hello, World” of machine learning (google.com)
Learn Python, Data Viz, Pandas & More | Tutorials | Kaggle
Learn Feature Engineering Tutorials (kaggle.com)
Neural Network Architectures & Deep Learning — YouTube
To understand this building model process, we can think about a normal model; in water resources, we have conceptual process-based models, like hydrological tank models or hydraulic models. In general, a mathematical equation has some parameters to represent a problem. An easy way to see it is like a tank model. You can see an excellent example in the work of Mostafa Farrag:
HBV lumped conceptual hydrological model | by Mostafa Farrag | Hydroinformatics | Medium
For this type of model, you need to optimise and find the proper parameter testing. The same principle applies to machine learning and deep learning models, just that the equations of the model are adaptable. Thus, we can say that part of the optimization process is the identification of the proper model or structure of the model (Hyperparameter optimization).
The diagram illustrates the process of building a machine learning (ML) model, specifically focusing on how it’s trained to simulate a natural system based on measured data. Here’s how it works:
- Input and Natural System Interaction:
- Measured Forcing (X): These are the input variables or conditions applied to the natural system(Vector). For instance, in a water resources model, this could include rainfall, temperatures, and other climatic factors.
- Natural System: This represents the real-world system you’re modeling, like a watershed or a river basin. The system responds to the measured forcings in complex ways.
2. ML Model and Its Outputs:
- ML Model f(θ,…): The machine learning model, defined by a set of parameters (θ), tries to mimic the natural system’s behavior. It takes the measured forcings as input and produces outputs or predictions (P).
- ML Output (P=Predictions): These are the model’s predictions of how the natural system responds to the input forcings, such as simulated water levels or flow rates in response to the input climatic conditions.
3. Comparing Model Outputs to Reality:
- Measured Response (Y): The actual, observed responses of the natural system to the input forcings. For a water resources model, this could be the actual measured water levels or flow rates.
- Error Analysis E=D(P,Y): This step involves comparing the model’s predictions (P) with the actual, measured responses (Y). The difference function (D) calculates the error (E) between the predictions and the real measurements. This error quantifies how well the model simulates reality.
4. Training the Model:
- Training: Based on the error analysis, the ML model’s parameters are adjusted to minimize the error (E). This process involves using results from error analysis to update the model’s parameters, aiming to reduce the difference between the model’s predictions and the actual measurements.
- The model might undergo calibration (adjusting parameters to better match observed data) or use reinforcement learning (a technique where the model learns to make better predictions through trial and error).
A cyclical process of building and refining a machine learning model to accurately simulate a natural system is a very common representation in water resources. The goal is to continually improve the model’s accuracy by minimizing the discrepancy between its predictions and actual observed data from the natural system.
This diagram illustrates the various uncertainties in building a model, specifically focusing on the stages where these uncertainties can manifest. It breaks down the process into four primary areas of concern:
- Input Uncertainty:
- This pertains to the data used to feed the model. The key questions here are about the relevance and interpretability of the input variables. What if the model is being fed data that isn’t actually useful for making predictions? Or what if the data is difficult to understand or interpret in the context of the model?
2. Sampling Uncertainty:
- This relates to the variability in the data samples used during different stages of the model’s lifecycle, such as training, verification, and testing. It questions the stability of the model’s performance if the samples it’s trained on were to change. Would the model still perform reliably if fed a different subset of data?
3. Model Calibration Uncertainty:
- Calibration involves adjusting model parameters to improve its accuracy based on known outputs. This section asks what would happen if the starting conditions for the model were altered or if a different method for searching the best parameter settings (optimisation) was employed. How significantly would these changes affect the model’s performance?
4. Model Performance Uncertainty:
- This final area deals with the evaluation metrics used to gauge the model’s effectiveness. It raises questions about the impact of changing these metrics or incorporating multiple performance measures. How does altering how we measure the model’s success influence our perception of its accuracy or usefulness?
The fluidity and complexity of modelling seem complicated when you consider how changes at any stage in the process can introduce uncertainty. Addressing these uncertainties is crucial for developing robust, reliable models.
Some algorithms, structures, and explanations
Many algorithms can be used in Deep Learning, but depending on the type of problem, it is easy to see that some apply more to specific type of problems:
Time series forecasting and large language models (LLM), which in water resources are forecasting rain and flows in a river basin.
For LSTM:
- How the LSTM input is prepared
- How the LSTM model receives the first input
- How the second layer connects
- How all the layer are interconnected at the end of the process
- What should be consireded during the exercise
- How the DropOutone works
- How the regularization works
- How the performance measure work
- How the LSTM works for Classification and for Regression.
Long Short term memory networks (LSTM), Generalize Adaptive Networks (GANs), overal eplanation can be found here:
What is LSTM (Long Short-Term Memory)? — YouTube
Deep Learning: Long Short-Term Memory Networks (LSTMs) — YouTube
Illustrated Guide to Recurrent Neural Networks: Understanding the Intuition — YouTube
C5W1 Recurrent Neural Networks — YouTube
Tips and tricks to train LSTM
https://danijar.com/tips-for-training-recurrent-neural-networks/
GRUs A step by step explanation
Crash Course in Recurrent Neural Networks for Deep Learning — MachineLearningMastery.com
GitHub — E3-JSI/FASTENER: Feature selection genetic algorithm implemented in Python.
Recurrent Neural Network Guide: a Deep Dive in RNN — neptune.ai
Applications in Python
Time Series Forecasting With RNN(LSTM)| Complete Python Tutorial| — YouTube
Multivariate Time Series Modeling using Facebook Prophet — YouTube
#lstm #timeseries #multivariate LSTM Multivariate Time Series Forecasting in Keras — YouTube
Pytorch
Pytorch Tutorial — Setting up a Deep Learning Environment (Anaconda & PyCharm) — YouTube
PyTorch Tutorial — RNN & LSTM & GRU — Recurrent Neural Nets — YouTube
Pytorch Bidirectional LSTM example — YouTube
Understanding the units in the LSTM
Tung website — Units in LSTM (tung2389.github.io)
Counting No. of Parameters in Deep Learning Models by Hand | by Raimi Karim | Towards Data Science
Vision pattern recognition or in water resources pattern in space. Convolutional neural networks (Explained here), Autoencoders (here), U-Nets (Here)
Requires for Convolution
- What is a kernel
- How the pooling works
- How it shifts the kernel
- How the next line received the size of the previous line
- How the model uses a parameter to store each iteration
- How the model reach a final MLP
- How the final MLP is trained
- Where does the RLU should go and how it looks in the Convolution
How Convolution Works — YouTube
Convolutions in image processing | Week 1 | MIT 18.S191 Fall 2020 | Grant Sanderson — YouTube
Machine Learning Zero to Hero (Google I/O’19) — YouTube
A friendly introduction to Convolutional Neural Networks and Image Recognition — YouTube
How convolutional neural networks work, in depth — YouTube
Practice
A Neural Network Playground (tensorflow.org)
Spatiotempora models with CONV LSTM
I think when the problem goes to integrating space-time features, the combined ConvLSTM is quite powerful. An example of this application can be found in this paper Spatiotemporal convolutional long short-term memory for regional streamflow predictions — ScienceDirect.
These are some sources of code that are very interesting for applying this algorithm:
2. To predict the next frame using single encoder https://colab.research.google.com/drive/125ViO5BqmII-b8rQ9wzqcQa1DVteOa5W?usp=sharing
3. To predict the next frame using data augmentation https://code.earthengine.google.com/c7bc5ab5bf3aca4bfc8341197d1a446e?noload=true
4. Similar: https://www.kaggle.com/kcostya/convlstm-convolutional-lstm-network-tutorial
5. GEE: https://code.earthengine.google.com/c7bc5ab5bf3aca4bfc8341197d1a446e?noload=true