Geek Culture
Published in

Geek Culture

Why Tree-Based Models Beat Deep Learning on Tabular Data

A much-needed reality check for AI Researchers and Engineers caught up in the hype around Deep Learning

Don’t show this graph to all the people with ‘Deep Learning Expert|Podcaster|Blockchain|Software’ in their bio. They will probably start screeching and get violent.

Points to note about the Paper

Reason 1: Neural Nets are biased to overly smooth solutions

The better performance of RFs can be attributed to the more precise decision boundaries they generate.

Finding 2: Uninformative features affect more MLP-like NNs

  1. Removing a lot of features reduced the performance gap between the models. This clearly implies that a big advantage of Trees is their ability to stay insulated from the effects of worse features.
  2. Adding random features to the dataset shows us a much sharper decline in the networks than in the tree-based methods. ResNet especially gets hammered by these useless features. I’m assuming the attention mechanism in the transformer protects it to some degree.
Tree Supremacy. One thing to note is that they used only the Random Forest feature importance. Involving more protocols to create a better feature accuracy score would make things much better.

Finding 3: NNs are invariant to rotation. Actual Data is not

…there is a natural basis (here, the original basis) which encodes best data-biases, and which can not be recovered by models invariant to rotations which potentially mixes features with very different statistical properties

Reach out to me



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Devansh- Machine Learning Made Simple

Deep Insights about Artificial Intelligence (AI), Machine Learning, Software Engineering, and the Tech Industry. Follow me to come out on top