Modern NLP
Published in

Modern NLP

10 Useful ML Practices For Python Developers

Fast bug-free coding

Photo by DeMorris Byrd on Unsplash

→ ML in production series

Come join Maxpool — A Data Science community to discuss real ML problems!

Sometimes as a data scientist we forget what are we paid for. We are primarily developers, then researchers and then maybe mathematicians. Our first responsibility is to quickly develop solutions which are bug-free.

Just because we can make models doesn’t mean we are gods. It doesn’t give us the freedom to write crap code.

Since my start, I have made tremendous mistakes and thought of sharing what I see to be the most common skills for ML engineering. In my opinion, it’s also the most lacking skill in the industry right now.

I call them software-illiterate data scientists because a lot of them are non-CS Coursera baptised engineers. And, I myself have been that 😅

If it came to hiring between a great data scientist and a great ML engineer, I will hire the later.

Let's get started.

Get all of my blogs topic wise here

1. Learn to write abstract classes

Once you start writing abstract classes, you will know how much clarity it can bring to your codebase. They enforce the same methods and method names. If many people are working on the same project, everyone starts making different methods. This can create unproductive chaos.

2. Fix your seed at the top

Reproducibility of experiments is a very important thing and seed is our enemy. Catch hold of it otherwise it leads to different splitting of train/test data and different initialisation of weights in the neural network. This leads to inconsistent results.

3. Get started with a few rows

If your data is too big and you are working in the later part of the code like cleaning data or modeling, use nrows to avoid loading the huge data every time. Use this when you want to only test code and not actually run the whole thing.

This is very applicable when your local PC config is not enough to work with the datasize but you like doing development on local on Jupyter/VS code/Atom.

df_train = pd.read_csv(‘train.csv’, nrows=1000)

4. Anticipate failures(sign of a mature developer)

Always check for NA in the data because these will cause you problem later. Even if your current data doesn’t have, it doesn't mean it will not happen in the future retraining loops. So keep checks anyways 😆

print(len(df))

df.isna().sum()

df.dropna()

print(len(df))

5. Show the progress of processing

When you are working with bigdata, it definitely feels good to know how much time is it going to take and where are we in the whole processing.

Option 1 — tqdm

Option 2 — fastprogress

6. Pandas can be slow

If you have worked with pandas, you know how slow it can get some times — especially groupby. Rather than breaking our heads to find ‘great’ solutions for speedup, just use modin by changing one line of code.

import modin.pandas as pd

7. Time the functions

Not all functions are created equal.

Even if the whole code works doesn’t mean you wrote a great code. Some soft-bugs can actually make your code slower and it's necessary to find them. Use this decorator to log time of functions.

8. Don't burn money on cloud

Nobody likes an engineer who wastes cloud resources.

Some of our experiments can run for hours. It's difficult to keep a track of it and shutdown the cloud instance when it's done. I have made mistakes myself and have also seen people leaving instances on for days.

This happens when we work on Fridays and leave something running and realise it on Monday 😆

Just call this function at the end of execution and your ass will never be on fire again!!!

But wrap the main code in try and this method again in except as well — so that if an error happens, the server is not left running. Yes, I have dealt with these cases too 😅

Let’s be a bit responsible and not generate CO2. 😅

9. Create and save reports

After a particular point in modeling, all great insights come only from error and metric analysis. Make sure to create and save well formatted reports for yourself and your manager.

Anyways, management love reports right? 😆

10. Write great APIs

All that ends bad is bad.

You can do great data cleaning and modeling but still you can create huge chaos at the end. My experience with people tells me many are not clear about how to write good APIs, its documentation and server setup. I will be writing another post on this soon but let me get you started.

The below is good methodology for a classical ML and DL deployment under not too high load — like 1000/min.

Meet the combo — Fastapi + uvicorn

  • Fastest — Write the API in fastapi because its the fastest for I/O bound as per this and the reason is explained here.
  • Documentation — Writing API in fastapi gives us free documentation and test endpoints at http:url/docs → autogenerated and updated by fastapi as we change the code
  • Workers — Deploy the API using uvicorn

Run these command to deploy using 4 workers. Optimise number of workers by load testing.

pip install fastapi uvicorn
uvicorn main:app --workers 4 --host 0.0.0.0 --port 8000

Conclusion

I hope you had a good time reading this. Please hit like if you enjoyed and share with friends who can benefit as well 😃

Want more?

Subscribe to Modern NLP for latest tricks !!! 😃

--

--

--

All the latest techniques in NLP — Natural Language Processing

Recommended from Medium

Why and How of Hypothesis Testing ( Part — I )

Motivation: The Time is Right

Creative Technologist/Data Scientist 🤓 at Decoded (New York)

Classifying environmental audio recordings

Costing the Count: What’s the real cost of a Census

Internal Audit: Top Software and Tools for Harnessing Data

The best data quality framework for senior platform engineers

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Pratik Bhavsar

Pratik Bhavsar

NLP & Semantic search engineer | Now writing on pakodas.substack.com | maxpool.club | @nlpguy_

More from Medium

Testing software — a practical guide to write tests for machine learning pipelines

How data scientists can leverage object oriented programming (OOP) design pattern to write better…

Train Python Code Embedding with FastText