Kolmogorov Arnold Networks (aka KANs), a game changer or just a hype.
After reading the paper on KANs, I was really stunned by knowing the accuracy of a KAN with compared to MLPs (Multi-Layer perceptron), we can’t deny the fact that MLPs are very proven ways to reach to any machine learning problem quickly, we know that if we have any high dimensional dataset and some complex patterns, then we can simply add more hidden layers to MLPs to make it learn better in terms of accuracy for understanding the underlying pattern for the targeted solution, and with adding more layer it requires more utilization of resources and more training time.
So, let me come to point, KANs claimed very big results, by showing that it is better than MLPs, where instead having a fixed activation functions and other fixed parameters on neurons, KANs use multi-variate functionalities on edges itself.
KANs introduced as replacement of MLPs, KANs have Learnable Activation Functions which MLPs don’t have.
In paper Authors claimed that KANs are promising in solving Naïve-strokes equations and for PDE solving, authors of the paper also claimed that a 2-Layer width-10 KAN is 100 times more accurate than a 4-Layer width-100 MLP (10^–7 vs 10^−5 MSE) and 100 times more parameter efficient (10² vs 10⁴ parameters).
The above results are taken from the KANs Paper, first decode the terminologies, KAN [2, 10, 1] means KAN network with input of 2 values, 10 hidden Layer, and 1 output values. MLP [2, 10, 1] is a MLP with input of 2 values, 10 hidden Layer, and 1 output values. MLP [2,100,100,100,1] means MLP of 2 inputs, first hidden layer with 100 neurons, second hidden layer with 100 neurons, third hidden layer with 100 neurons, and final layer with 1 neuron as output.
Results says that MLP [ 2, 10, 1] a simpler MLP is not performing well, but if we make it complex MLP [ 2, 100, 100, 100, 1] shows that results were improved a lot, but as per the results when KAN [ 2, 10, 1] is performing better than even complex MLPs.
But are KANs really promising for the claims given in their paper?
As per the George Karniadakis Professor of Applied Mathematics & Engineering, Brown University, says the statement given below:
The validity of the claims in the KAN paper is under scrutiny. An author of the paper initially agreed to give a talk at CRUNCH but later canceled to re-evaluate their claims. Members of the group attempted to verify these claims and found several issues. The KAN network failed to correctly solve problems such as the cavity flow using Navier-Stokes equations, a two ODE system for PK dynamics, the Helmholtz equation, and the Poisson equation, resulting in significant errors. Although the paper reported high accuracy, this was achieved by switching from learning activation functions to using sine functions, which converge quickly like spectral methods. KAN was also found to be slow. A systematic investigation is planned to rigorously evaluate the claims, but current findings suggest KAN is not reliable.
In response to Prof. Karniadakis’ constructive suggestions, the first author Ziming Liu of the KAN paper has provided clarifications and updates regarding their research findings. The author acknowledged the criticisms and offered the following key points for the public:
1. Grid Update and Extension Tricks:
— These techniques were utilized in the numerical experiments but were not included in the tutorial to keep it simple.
— Incorporating these tricks reduced the error for the 2D Poisson equation from 24% to 0.05%.
— The author expressed gratitude for the feedback on the 24% error, noting it reflected typical user experience due to the lack of clear instructions on grid extension in PDE contexts.
— The tutorial has been updated accordingly: [Example_6B_PDE.ipynb](https://github.com/KindXiaoming/pykan/blob/master/tutorials/Example_6B_PDE.ipynb).
2. Navier-Stokes Equation:
— The author thanked GitHub user chaous for their contribution, despite the initial errors in the equations.
— The errors have since been corrected, and the author admitted to not thoroughly checking the equations when approving the pull request.
— Prof. Karniadakis was thanked for identifying this issue.
— The corrected tutorial is now available: [physics_informed_kan.ipynb](https://github.com/KindXiaoming/pykan/blob/master/tutorials/physics_informed_kan.ipynb).
The first author expressed appreciation for Prof. Karniadakis’ valuable feedback and the opportunity to improve the KAN network’s implementation and documentation.
Link to that LinkedIn thread is given here.
“Although KANs still need to provide many examples to verify their claims, I am looking forward to exploring this interesting topic further.”
