Part III: Deep Liquid Time Neural Network in NeuralStart
In this third part, let’s delve deep into the Liquid Neural Network (LNN), a key component of the NeuralStart project. We will explore the data processing process with LNN in detail, its exceptional advantages, and why it is considered a revolutionary innovation in the field of neurotechnology.
Liquid Neural Network: A Revolutionary Concept
The Liquid Neural Network, or LNN, is a deep learning approach heavily inspired by the functioning of the biological brain. This network is designed to mimic how biological neural networks interact and adapt to incoming signals. Here’s an in-depth explanation of how it works and its relevance to NeuralStart:
Concept and Continuous Learning: The central idea of the LNN is to simulate an artificial neural network in a liquid manner. In an LNN, connections between neurons are constantly changing, enabling continuous learning. At each moment, the network adapts based on incoming data, making it incredibly flexible and capable of tracking changing brain activity patterns in NeuralStart users.
Model Architecture: We used a specific model architecture to implement the LNN in NeuralStart. The architecture includes the following components:
- An input layer that takes into account the shape of the input data.
- A 1D convolutional layer that extracts important features from the data.
- An RNN layer based on the LTC (Liquid Time-Constant) Cell, which represents the core of the LNN. This layer is responsible for continuous learning and adaptability.
- A Global Average Pooling layer to aggregate information.
- An output layer with a “softmax” activation function for multi-class classification.
X_train = X_train.reshape(X_train.shape[0], 1, X_train.shape[1])
X_test = X_test.reshape(X_test.shape[0], 1, X_test.shape[1])
from ncps.wirings import AutoNCP
from ncps.tf import LTC
import tensorflow as tf
from ncps import wirings
import tensorflow as tf
from sklearn.utils.class_weight import compute_class_weight
from keras.callbacks import EarlyStopping, ModelCheckpoint
import kerasncp as kncp
from kerasncp.tf import LTCCell
#from ncps import LTC, AutoNCP
num_classes = 6 # Remplacez par le nombre de classes dans vos données
timesteps = 1
input_shape = (timesteps, X_train.shape[2])
#arch = kncp.wirings.Random(32, 1, sparsity_level=0.5) # 32 units, 1 motor neuron
#rnn_cell = LTCCell(arch)
#wiring = kncp.wirings.FullyConnected(10)
wiring = wirings.AutoNCP(100, 25)
rnn_cell = LTCCell(wiring)
model = keras.models.Sequential(
[
keras.layers.InputLayer(input_shape=input_shape),
keras.layers.Conv1D(filters=82, kernel_size=3, activation='relu', padding='causal'),
keras.layers.RNN(rnn_cell, return_sequences=True),
keras.layers.GlobalAveragePooling1D(),
keras.layers.Dense(num_classes, activation="softmax"),
]
)
model.compile(
optimizer=keras.optimizers.Adam(0.01),
loss='sparse_categorical_crossentropy',
metrics=['accuracy'],
#run_eagerly=True
)
model.summary()
# Définissez les callbacks
early_stopping = EarlyStopping(monitor='val_accuracy', patience=100, verbose=1, mode='max')
model_checkpoint = ModelCheckpoint('test.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
y_train_encoded = to_categorical(y_train)
y_test_encoded = to_categorical(y_test)
# Entraînez le modèle
history = model.fit(X_train, y_train, epochs=1000, batch_size=30, validation_data=(X_test, y_test), shuffle=True, callbacks=[early_stopping, model_checkpoint])
Model: "sequential_10"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv1d_10 (Conv1D) (None, 1, 82) 84460
rnn_10 (RNN) (None, 1, 25) 73314
global_average_pooling1d_10 (None, 25) 0
(GlobalAveragePooling1D)
dense_10 (Dense) (None, 6) 156
=================================================================
Total params: 157,930
Trainable params: 157,930
Non-trainable params: 0Model Training: The LNN model was trained with patience and determination. We used the Adam optimizer to minimize the “sparse_categorical_crossentropy” loss function. The model was evaluated on metrics such as accuracy, which measures the proportion of correct predictions compared to the test set
Epoch 1/1000
812/814 [============================>.] - ETA: 0s - loss: 1.0425 - accuracy: 0.5146
Epoch 1: val_accuracy improved from -inf to 0.59027, saving model to test.h5
814/814 [==============================] - 37s 29ms/step - loss: 1.0425 - accuracy: 0.5148 - val_loss: 0.9636 - val_accuracy: 0.5903
Epoch 2/1000
814/814 [==============================] - ETA: 0s - loss: 0.9033 - accuracy: 0.6210
Epoch 2: val_accuracy improved from 0.59027 to 0.65383, saving model to test.h5
814/814 [==============================] - 23s 29ms/step - loss: 0.9033 - accuracy: 0.6210 - val_loss: 0.8530 - val_accuracy: 0.6538
Epoch 3/1000
812/814 [============================>.] - ETA: 0s - loss: 0.8015 - accuracy: 0.6771
Epoch 3: val_accuracy improved from 0.65383 to 0.68496, saving model to test.h5
814/814 [==============================] - 23s 29ms/step - loss: 0.8010 - accuracy: 0.6772 - val_loss: 0.7829 - val_accuracy: 0.6850
............................................................
Epoch 262/1000
814/814 [==============================] - ETA: 0s - loss: 0.0174 - accuracy: 0.9939
Epoch 262: val_accuracy did not improve from 0.97625
814/814 [==============================] - 24s 29ms/step - loss: 0.0174 - accuracy: 0.9939 - val_loss: 0.1261 - val_accuracy: 0.9740
Epoch 263/1000
813/814 [============================>.] - ETA: 0s - loss: 0.0114 - accuracy: 0.9967
Epoch 263: val_accuracy did not improve from 0.97625
814/814 [==============================] - 24s 30ms/step - loss: 0.0114 - accuracy: 0.9967 - val_loss: 0.1362 - val_accuracy: 0.9718
Epoch 263: early stoppingExceptional Results: The results achieved with the LNN model are impressive. We achieved an accuracy of approximately 96.94%, meaning the model could correctly classify the data almost 97% of the time. These high-performance levels are the result of the continuous learning of the LNN and its ability to adapt to changing data.
import matplotlib.pyplot as plt
# Afficher l'historique de l'exactitude
plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('Model Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
# Afficher l'historique de la perte
plt.subplot(1, 2, 2)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Model Loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
plt.tight_layout()
plt.show()
Evaluation and Results Analysis: We evaluated the restored model and obtained an accuracy of approximately 93.14%. This confirms the model’s robustness even after being saved and loaded again. We also compared the model’s predictions to the true class labels, demonstrating the model’s ability to make accurate predictions.
restored_model = tf.keras.models.load_model("test.h5")
restored_model.evaluate(x=X_test, y=y_test)
191/191 [==============================] - 2s 8ms/step - loss: 0.4350 - accuracy: 0.9314
[0.4350038766860962, 0.9314191937446594]
# Évaluez le modèle
loss, accuracy = model.evaluate(X_test, y_test)
print(f"Test accuracy: {accuracy}")
191/191 [==============================] - 2s 8ms/step - loss: 0.4189 - accuracy: 0.9226
Test accuracy: 0.9225594997406006
predictions = model.predict(X_test)
predicted_classes = np.argmax(predictions, axis=1)
191/191 [==============================] - 2s 8ms/step
predicted_classes
array([4, 3, 1, ..., 5, 5, 3])
y_test
array([4., 3., 1., ..., 5., 5., 3.])Conclusion: A Revolutionary Innovation for NeuralStart
In conclusion, the Liquid Neural Network is a cornerstone of the NeuralStart project, enabling seamless interaction between users’ brain activities and applications. Its continuous learning and adaptability make it an exceptional technology for neurotechnological applications. The high-performance achieved with the LNN opens the door to fascinating possibilities in the field of neurotechnology, offering significant potential to improve the quality of life for people with disabilities and explore new frontiers of brain-computer interaction.
However, to achieve such high results, two key aspects need to be taken into account. Firstly, the transformation of initial data using various technologies like ICA, PCA, regression, correction, and Kalman filters played a critical role. Secondly, data augmentation was crucial as well, as without it, the class imbalance would have led to a mere 33% learning efficiency for both the Random Forest and the LNN models.
This overview has been presented briefly, but it required extensive research and testing on various models and different data processing styles. Now, I am proud to present our results alongside my colleague ROSIE for this remarkable team effort. The tasks were divided into two parts: I handled all aspects of data science, and ROSIE managed the architecture, including the connection between the headset and the game creation, as well as the Python program for live data processing.
This project was immensely exciting, and it has significantly enhanced my experience in neurotechnology, deep learning, and the processing of temporal and sequential series. Thank you, dear reader, for your patience. I hope the shared concept and code will be useful to you. Have a fantastic day ahead.
The research on Liquid Time-Constant (LTC) Networks is detailed in a paper titled “Liquid Time-constant Networks,” authored by Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, Radu Grosu, and others. This paper is a significant contribution to the field of time-continuous recurrent neural network models and is inspired by the dynamic behavior of biological neural networks. In this paper, the authors introduce the LTC network, which incorporates a nonlinear input-dependent term in its time-constant module. This feature allows the network to adapt its dynamics to different tasks and inputs, making it particularly suitable for processing spatiotemporal data.
The paper discusses the universal approximation theorem, which explores the expressive power of neural network models. The authors emphasize the need for a more rigorous measure of expressivity to compare models, especially those specialized in spatiotemporal data processing like LTCs. They propose a measure of expressivity based on trajectory length, which considers the complexity a learning system can compute, given the network’s capacity such as depth, width, type, and weights configuration. This approach helps to evaluate how a deep model transforms a given input trajectory into a more complex pattern.
Furthermore, the research was supported by several notable institutions, including the AI2050 program at Schmidt Futures, the Boeing Company, and the United States Air Force Research Laboratory. The authors of the paper include experts from the Massachusetts Institute of Technology, Institute of Science and Technology Austria, Aalborg University, and the University of Vienna.
For more detailed information and to access the full paper, you can refer to the publication on arXiv: “Liquid Time-constant Networks”. Additionally, for further insights into the broader applications and implications of this research, you can also refer to the related publication in Nature Machine Intelligence: “Closed-form continuous-time neural networks”.
