My Python boilerplate and a little Python-Fu

Nicky Reinert
Feb 24 · 6 min read

Let me show you my Python boilerplate and some little nifty standard features I’m working with. You can download the full template from github. This post will probably be extended in the future.

Virtual environment

I like working with virtual environments. It helps me to work with individual clean development environments. How do you do that? Switch to your main development folder, create a subfolder for your new project and initiate it:

cd /development/
mkdir project1
python3 -m venv project1
cd project1
source bin/activate

There you go. If you install packages, they are located in this folder only. Wait, packages? Right, let’s get to the next step:

This post comes without any images, that’s why I’m showing you this beautiful image that I took in Athens, a couple of years ago — a wonderful city!

Manage your libraries

As your Python scripts grows, by the amount of lines but also by the amount of features, the amount of used libraries will grow, too. But how to run the script on a different machine, in a differente environment? How to easily share you script? The answer is pipreqs:

pip install pipreqs

After you installed this little tool, run it against your script (or a folder of scripts):

pipreqs yourscript.py # one script only
pipreqs . # all scripts in the folder

Pipreqs will scan your script for required libraries and list them in a file called requirements.txt (suprise!). You need to share this text file with your Python script. Everyone who wants to run the script needs to prepare the environment using this line of code:

pip install -r requirements.txt

Et voilà. All requirements met.

Telegram logging

Please be adviced: Using a centralized messaging service like telegram, where your content is hosted on foreing computers, may be a risky plan. I’m fully aware of that, you should be too. That’s why I’m only sharing non-senstive information with telegram. It’s just my perfect notification service.

How does telegram help? You may ask. Well, imagine you have a script running hours or even days! Actually, you don’t really know. Wouldn’t it be nice to get a message, when the script is done, or about it’s progress.

First let’s create your own bot for Telegram. Open the app and look for “BotFather”. He will allow you to create a new bot. Name doesn’t matter, what does matter is the secret(!) token. Take it and start a chat with your bot. Then browse to this url (replace the token part with your secret(!) token):

https://api.telegram.org/bot<bot token>/getUpdates

The page will return a JSON object with a couple of information, amongst it is your chat-id (it’s secret, too, everything is secret!). Grab it and put it with your secret (!!!) token into your telegram_notifier.py:

import requestsbot_token = 'secret token'
bot_chat_id = 'secret chat id'
def send_text_to_telegram(bot_message):
bot_token = bot_token

send_text = 'https://api.telegram.org/bot' + bot_token + '/sendMessage?chat_id=' + bot_chat_id + '&parse_mode=Markdown&text=' + bot_message
response = requests.get(send_text)

return response.json()
def send_image_to_telegram(image): bot_token = bot_token

data = {'chat_id': bot_chat_id}

url = 'https://api.telegram.org/bot' + bot_token + '/sendPhoto?chat_id=' + bot_chat_id + '&parse_mode=Markdown'

with open(image, "rb") as image_file:

response = requests.post(url, data=data, files={"photo": image_file})

return response.json()

Please note, that there’s no error handling with this script.

Congratulations, you now have two functions to send text and images to your telegram bot. This will help you to get a notification from your Python scripts. Import the two functions with this line in every of your actual scripts:

from telegram_notifier import *

In media res

Command line parameters

Enough prepping, let’s talk about the script itself. The first lines look like that:

import time # for logging purposes
import argparse # for command line arguments
parser=argparse.ArgumentParser()parser.add_argument('--limit', help='Source files in CSV format', required=True, type=int)
parser.add_argument('--loop_length', help='Source files in CSV format', required=False, type=int)
parser.add_argument('--message', help='Give me a message', required=False, default="Hello World", type=str)
args = parser.parse_args()

You’ll find it helpful to pass arguments to your script from the command line. That’s what the argparse library is for. This library supports loads of other parameters and features, I’m just showing you some basic ones to read two parameters limit and loop_length. You can read those parameters using args.limit or args.loop_length. So easy. Example?

loop_length = 1000000 if args.loop_length == None else args.loop_length
limit = args.limit

That’s the shortend version of a if-condition. If the loop-length is set, take it. Otherwise use the default value of 1.000.000. The limit is always set to the one passed by the user — because it’s a mandatory parameter and therefore always present. Next, please.

Logging

Why is logging so important to me? Why sould it be important to you? You will get to that point, when a customer asks you:

How long will it take to process this dataset with 2 Mio. rows? How much will it costs?

~a customer, somewhen

Or if you want to compare two different methods. And so on. I’m a huge fan of (extensive) logging. It’s very helpful. Trust me. So, how do we start?

feedback_frequency = 10                                               count = 1  
start = time.time()

Frequency? Why? Imagine you have a script that crunches 1 Mio. rows. And depending on the surrounding parameters, it’s either pretty fast or pretty slow. When do you create a log message? With the feedback_frequency you define, how many times you want to see a log message. Set it to 10 to get a feedback every 1 Mio. / 10 rows. Increase it to get even more feedback.

Colors

This is really just some Python-fu. If you’re working with data, colors are important. I’m making this quick. You need color palettes with custom lenght? Her you go:

import seaborn as sns
count_colors = 42
color_palette = sns.color_palette("Paired", count_colors)

This will get you a palette of 42 different colors? Need less or more? Change the value of count_colors. I love it!

If you need more color palettes, look there.

A simple loop — comprehension

The list or dict comprehension feature is a feature you either love or don’t understand. I’m not even sure, if I really understand the feature. Neverthelesse, here’s an example that hopefully explains a lot.

List comprehension works for dicts, too. It allows you to loop through data without writing a fully fledged loop.

The following example goes through a range of numbers from 1 to loop_length=1.000.000 (second last line).

The last line in this comprehension says: Only process the current row, if index is greater than 10.

The first line just takes the index value. Work with it. Add it, round it, calculate it. That’s the value, that goes to our list “a_list”.

The second line is the second way to work with conditions: Depending on the acutal value of index, it defines an outcome. If index is less than 50, pass it on. Else pass on the value defined in default_value.

default_value = 'x'
a_list = [
index
if index < 50 else default_value
for index in range(1, loop_length)
if index > 10
]

I know it’s tricky. You will get to it, as soon as you need it. ;P

Logging, actually

The next lines will send one message to telegram and to the standard output.

duration = time.time() - start            message = '\r\nDone with list comprehension after %s seconds ' % (      '{:.2f}'.format(duration) # two decimal places
)
send_text_to_telegram(message) print(message, flush=True)

You should understand this one from the code, except the flush-Parameter? This one forces the Python-script to send the message to the output, without waiting for the script to end. That’s your real-time-update-guarantee!

The fully-fledged Loop

This is a full loop, that allows you to add as many conditions and calculations as you want.

for i in range(1, loop_length):   if i > limit - 1 and limit > 0: break

count += 1
if i % (limit / feedback_frequency) == 0 or i % (round(loop_length / feedback_frequency, 0)) == 0:

progress = i / limit if limit > 0 else i / loop_length
message = (
'%s%%...' %
('{:.1f}'.format(100 * progress))
)
print(
message,
end = '', # no line break
flush=True # don't buffer output
)

The first lines are easy. Break the loop, if the limit is hit. The most exciting thing is the logging-feature. As mentioned above, it will only logg a message if it meets the frequency requirements. Also check out the end-Parameter of print. It removes the line break from the print-command.

Final message

Finally, after our script logic is finished, we send a message. This time to telegram, too.

duration = time.time() - startmessage = '\r\nDone with processing %s iterations after %s seconds ' % (
'{:0,.0f}'.format(count), # formatting integers with thousand separator
'{:.2f}'.format(duration) # formatting decimals with two decimals
)
send_text_to_telegram(message)print(message)

There’s nothing special happening here. The only interesting thing is, that Telegram also accepts images. This is good if you’re creating some sophisticated statistcial analysis and want to see the output. Again, a word of advise: Remember that Telegram is storing your messages on foreign servers. Keep that in mind.

Last words

I’m always learing, and I hope your learned something, too. I’m going to extend this post in the future. If you have further tricks for me, drop me a comment.

Nerd For Tech

From Confusion to Clarification

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store