Analytics Vidhya
Published in

Analytics Vidhya

Google AutoML Tables vs XGBoost

Google is the prototypical machine learning company. As a young and hungry startup, they first used “statistical modelling” to correct the spelling of search terms, and they’ve never looked back. These days, they even use machine learning to invent the new the machine learning, mechanising the invention of new deep neural network topologies that achieve state-of-the-art results in terms of accuracy, while often reducing total model size. If you’ve trained any deep neural networks at all, you’ve likely used Google’s Tensorflow framework — not bad for an advertising firm!

So, when such a legendary force in machine learning announced a new product — AutoML — it wasn’t a matter of if I would try it out, but when. My first opportunity came as my team reached our “feature complete” milestone on a machine-learning based microservice we are developing for a non-profit.

The training set for that project is structured or tabular data — close to the sort of thing you might find stored as a table in a database. Since I already had code to train an effective model using, this was a perfect chance to compare “traditional” (models from open-source libraries tune using grid search) to “cutting edge” — Google’s AutoML Tables.

Getting started was super-easy — just upload a CSV of my data to a Google Cloud bucket, something AutoML Tables has UI support for, keeping the workflow simple.

Freshly imported data summarized in AutoML Tables

Data import was slick and accurate, with numeric and categoric columns handled automagically. All I had to do was select the target column (“correct_location”), and I was ready to click the enticing blue “TRAIN MODEL” button. Data import will be even easier if you are already using GCP.

Model training comes in 1-hour increments, and at the time of writing, costs $19.32 per hour — this is paying for a cluster of 92 virtual machines (n1-standard-4) which will team up for an hour to get the best possible results.

Training results — nice accuracy, AUC looks usable

An hour later, my model was ready, and a very presentable dashboard was full of good news — acceptable F1 score, AUC, log loss. The model itself ways in at a comparatively chunky 235MB.

While my 92 node cluster was doing its Google-Fu, I also rented a tiny VM to run our traditional, artisanal, hand-crafted non-Bayesian-optimised training code — a single n1-standard-1 machine, which is a single Haswell core with just under 4GB of memory, renting for just $0.04 per hour. Our code loaded the same CSV, then ran a grid-search, our brute-force hyperparameter tuning, training 108 XGBoost models (gradient boosting trees are highly recommended for structured or tabular data). Our 1-core minnow completed its work in about 15 minutes, crossing the finish line a healthy 45 minutes ahead of the cluster, and costing us just $0.01 of compute. The finished model was about 1.3MB in size and scored a log loss of about 0.2, significantly better than the AutoML Tables results of 0.36 (for log loss, a larger score is worse, indicating less accurate class probabilities).

Google AutoML Tables vs. XGBoost + Grid Search

Of course, training a model isn’t much use unless you deploy it. AutoML tables really shines in this area — with a few clicks, you can batch-process data from a Google Bucket, or deploy your model as a REST service. There’s definitely a big difference between this and our hand-crafted human-dockerized Flask app.

It’s an exciting time for machine learning engineering. For our specific customer, cloud deployment isn’t where we will deploy, but it’s inspiring to see how easily we could have done so. However, AutoML Tables, at least for this dataset, definitely disappointed in terms of model quality, model size, and time train, all distinctly worse than our fairly bare-bones baseline.

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Edward Dixon

Edward Dixon

#AI guy, Principal/Founder @ Rigr AI, co-author of ‘Demystifying AI for the Enterprise’.

More from Medium

Why Data Science Experts Should Adopt AutoML

Top 5 Model Training and Validation Challenges That Can be Addressed with MLOps

Deploy Machine Learning Model using Amazon SageMaker (Part 1)

Regression in RAPIDS vs Sklearn