Book Summary — Data Smart

You can find all my book summaries — here.

This book was the intellectually hardest book to comprehend, but incredibly stimulating. It goes through machine learning and AI models at incredible speed, as well as detail with very specific examples.

Absolutely loved it, but so hard to follow every step.

Data Smart gives you a full insight into statistics by running the reader through basic machine learning, AI, forecasting and optimisation models in Excel, but then also provides a bridge into how to solve all those problems in R.

Everything you ever needed to know about Spreadsheets

The first chapter kicks off with a great list of every Excel command you will need to build models — I thought it was an incredibly comprehensive list of all things you need to know.

Once you know them you’re sorted.

• shortcuts (cmd + right, etc)
• pane freezes
• copy paste (transpose, values only etc)
• fixed references
• dynamic formatting
• Formulas — Index, Match, Offset, Vlookup, Sumproduct
• Filtering / Sorting
• PivotTables
• Array Formulas
• Solver (which is essential for those models)

Then the book dives straight into examples and theory.

Cluster Analysis

Unsupervised analysis

Instead of blasting your whole database with emails, a simple cluster analysis can identify what different segments have in common.

The idea is you pick a couple K-means and you check which observations/customers/users (whatever you’re analysing) are closest — that’s the makeup of clusters

Silhouette → ( avg distance to those in the nearest neighbouring cluster minus avg distance to those in my cluster) / Maz of those two averages

K-median → modify clustering to only use values present in customers’ deal vectors

Measuring distances:

• Euclidean — take Pythagoras’ distance, as the “crow flies”
• Manhattan/Hamming —giving points when you match with someone else on a characteristic — you assign cosine distance to make it even better

Naive Bayes Model

Supervised Analysis

You train a model so that it recognises patterns in datasets — “bags of words”

• remove punctuation, capitalised and short words
• crosscheck words between dataset and bag

Optimisation Models

An artificial intelligence model predicts the result of a process by analyzing its inputs. That’s not what this is about.

Min / Max optimisation with conditions → use Solver

Artificial Intelligence Model

Training the model to identify the best predicting factors “via stumps”.

Train linear regression model by minimising squared error.

• F test to show statistical significance
• Check model coefficients
• T test to show individual statistic significance

Ensemble model — build a couple AI models and let them vote on the most likely outcome

Forecasting

Exponential smoothing — smooth impact + most recent higher weight

Check for autocorrelation — do more smoothing ie seasonality (2x12 months moving average)

Systemise it

The last chapter does everything the book has showed step by step in Excel in a couple of simple lines of code in R.