Agile Machine Learning for Classification — Introduction
Hello everyone !
Several years ago, when I started my journey in AI, the vast majority of the articles and examples were written from an algorithm perspective rather than starting with the business problem statement. For example, an introductory course in data science will take the academic approach and will tend to walk you through the intricacies of Logistic Regression, Random Forests, XGBoost, etc. After I read them, I was always left wondering: “When do you use what?” Understanding how something works is not the same as knowing when to use the right technique
If all you have is a hammer, everything looks like a nail !
Knowing how long a hammer is, how much it weighs, what material its made of gives no actionable insight on whether it can be used to hammer a nail or other kinds of objects. In Machine Learning, the ‘nail’ that I am referring to are the simple datasets like titanic or iris dataset so often seen in example tutorials. After the end of the tutorial, you are left wondering if the machine learning workflow can be generalized to other datasets.
Additionally, I noticed a dearth of materials on end-to-end machine learning starting with the business problem statement. There are indeed very good articles on specific topics like CNN or RNN but very few resources that illustrate how data scientists in the industry actually go about solving business problems to achieve the highest impact. In fast-paced jobs, the business stakeholders rarely care about how your latest & greatest Deep Learning algorithm outperforms everything else if it took several months to develop. What the stakeholders are interested in are regular updates to see things are on track. Wait too long and the project may get the funding cut.
This created a dilemma for me. How should I go about structuring my Machine Learning notebooks? Should I do Feature Engineering right from the very beginning? When should I do hyperparameter tuning? What about Ensemble methods like Stacking? Should I ask for more resources to do Deep Learning? Or should I just stick with Tree-based models?
To help other Data Scientists who are on the same situation as I was a few years ago, I have created a series of notebooks that increases in complexity from a baseline model to more advanced methods like hyperparameter tuning through hyperopt during Stacking. These notebooks cover the majority of the steps and are comprehensive enough to be used as Starter notebooks for common types of classification problems. Sometimes, business is happy to get anything to work and may even ask you to wrap up the modelling part after the very first week because the performance maybe good enough to get the project going. Therefore the intent of these articles is to be able to stop at any week and present the results to the business
What I wont be covering are the initial steps such as setting up your local Python environment because there are adequate number of these articles online. The following notebooks are aimed at those who have already worked through quite a bit of previous Machine Learning tutorials and would like to level up their skills in techniques that are frequently used by the top data scientists. In the interests of time, I will also be skipping EDA because again there are sufficient materials online and the assumption is you have already done some preliminary due diligence of the data and have a fair understanding of it. While the techniques introduced primarily use rich Kaggle datasets on binary classification data, you can reuse much of the code for Regression Problems, Anomaly Detection, etc.
Lets get going with the Kaggle Data! The next in the series is available here
