A Cheap Introduction to Comprehension In Python

One of the coolest features I have found in python is comprehension. I will share my learning experience about comprehension for the little time I have written python.

Python has three types of comprehension namely;

1. List comprehension

2. Dictionary comprehension

3. Set comprehension

When to use comprehension?

Comprehension comes in handy when creating a new built-in data structure like a list, a dictionary or a set from an existing data structure. When doing so the following steps are usually followed:

1. Begin with a new empty data structure to create for example a list, set or a dictionary.

2. Loop through an existing data structure and process each item in the iterator

3. Populate the new data structure with the processed data.

If you find yourself doing the above, think of a comprehension. Let’s see it in action.

List comprehension

Syntax.

new_list = [expression for each_item in existing_data_structure]. 

The outer [] brackets indicate that you are creating a new list for each item in the iterator. The expression can be any valid python expression resulting into a value to include in the newly created iterator.

Examples

Suppose you have a list of letters and you are required to capitalize each letter in a list.

Solution 1: Using a for loop

output: ['A', 'B', 'C', 'D', 'E', 'F']

Solution 2: Using list comprehension

output: ['A', 'B', 'C', 'D', 'E', 'F']

Solution 2 uses less code as compared to solution one. It as well executes faster as we shall see later.

However you should not rush for comprehensions as better solutions might exist. For example, If you care about performance, the lines of code below will be better for the above solution. The solution will perform better than the previous solutions as this will be demonstrated after.

Solution 3: Using a combination of join and list built in functions

output: ['A', 'B', 'C', 'D', 'E', 'F']

A closer look at the performance of the three solutions

Performance of the three approaches can be compared using the code snippet below.

Note that the code below will run better on python 3.6 since f-strings are used to print out the times of execution of the individual functions.

The output differs from machine to machine and depending on the running processes on the machine but the third solution will always win.This is an example of an output I got from my machine.

Time taken by:
1. For loop code is: 1.4384246520003217s
2. Comprehension code is: 1.1827780300000086s
3. Join code: 0.6697403460002533s

The times printed out correspond to time taken to run each function 1,000,000 times as this is the default value of the `timeit` function from the `timeit` module. This can be overridden by passing a keyword argument `number` and assigning it the number of times of execution of a particular block of code.

For more information on how to determine performance of small bits of python code you can look at the awesome timeit module inbuilt into python.

What if we are given a list of numbers and we are told to pick odd numbers only in our second example?

Solution 1: Using a for loop

Solution 2: Using comprehension

Output for solutions 1 and 2: [1, 35, 67, 81, 45, 89]

The above example introduces something new that is,

If each_number% 2 == 1. 

This checks whether the number is an odd number before it’s added to the created list. Such expressions are called filters. They are just normal python conditional expressions used to filter out values to include in the new list or data structure.

In our last example lets make a list, from dictionary keys using list comprehension.

Suppose we are given a dictionary of students, containing their names as keys and age as values. We are as well told that a student can only vote if he/she is above 18 years. We are required to generate names of students who can vote. Let’s see some of the available solutions.

Solution 1: Using the for loop

Solution 2: Using comprehension

Output for both solutions: ['dennis', 'david', 'daniel']

Comprehensions will save you from writing a lot of code and they are optimized for running by the python interpreter. The above example also filters out students whose age is greater than 18.

What if you are required to generate a list of lists from a dictionary, i.e. a list which has a lists of students who can vote and their age? How can you go about that? Below is unexplained code.

Solution 1: Using a python for loop

Solution 2: Using list comprehension

Output for solutions 1 and 2: 
[['dennis', 23], ['david', 21], ['daniel', 25]]

Hope you were able to figure out how the code works.

From our previous examples, we note that list comprehension always arises when you are creating a new list from an existing data structure which can either be a dictionary, set, tuple or a list its self among others. Why don’t we look at set comprehensions to see how they come about too.

Set comprehension

This also arises if we are creating a set from an existing data structure. If you know about list comprehension then you already know about set comprehension. However you have to substitute [] with {} in set comprehension.

Below is the syntax.

{ expression for each_item in existing_data_structure }

The opening and closing curly braces that is `{}` mean that you are telling the computer to start with an empty set and create it from an existing data structure.

Examples

In our first example,we are given a `list` of uppercase words and we are required to get a `set` of capitalized words.

Solution 1: Using a for loop

Solution 2: Using list comprehension

Output for solutions 1 and 2: {'Men', 'Queen', 'King', 'Dreams'}

In our second example we are required to get a set of even numbers from a list of numbers. This example also demonstrates filtering as we filter out even numbers.

Solution 1: Using looping

Solution 2: Using set comprehension

Output for both solutions: {24, 2, 12}

There is a difference in the two examples we have seen in set comprehension. In the first one we started with a list which we converted to a set (words) before iteration and in the second example we just loop through the given list which has duplicate entries. The latter is bad if the list has many duplicate entries and you do not need to keep track of them. So better convert it to a set to eliminate duplicate entries before looping through them. This will save you some PC cycles and reduce execution time of your code.

Dictionary comprehension

This also arises if we are creating a new dictionary from an existing data structure. The existing data structure doesn’t need to be a dictionary that is, it can as well be a list or a set or a tuple.

Lets see its syntax.

{ key:value for key,value in existing_data_structure }

Note the difference between the dictionary comprehension and set comprehension. The colon between the `key` and `value` at the beginning only exists in dictionary comprehension. The `key` will be the `key`of the new dictionary and the `value` its corresponding `value` in the new dictionary. Let’s see some examples of this in practice.

Example 1: Demonstrates making a new dictionary from an existing dictionary

Supposing we are given a dictionary of students with their names in lowercase as keys and age as values. We are required to write the names in uppercase. Let’s see how we can go about this.

Solution 1: Using a for loop

Solution 2: Using comprehension

Output for solutions 1 and 2: {'DENNIS': 23, 'DAVID': 21, 'MARY': 9, 'DANIEL': 25, 'DARIUS': 17, 'JIM': 10, 'MARVIN': 8}

Do you notice the colon (:) between key.upper() and value in the comprehension solution?. If you forget it, python won’t forgive you. It’s the main difference between dictionary comprehension and set comprehension since both of them use curly braces.

Example 2: Demonstrates filtering values to include in a new dictionary basing on a value or a key in the existing data structure.

Let’s use the students dictionary in example one above to create a new dictionary of students who can vote. A student can vote if he/she is above 18 years.

Solution 1: Using a for loop

Solution 2: Using list comprehension

Output for solutions 1 and 2: {'dennis': 23, 'david': 21, 'daniel': 25}

Example 3: Demonstrates nesting comprehensions

Suppose we have a list containing a list of a student’s name and his test score;

scores = [["dennis", 10], ["david", 12], ["joseph", 13]]

We are required to generate a dictionary containing the student name as a `key` and his score as a `value` if the student scored a mark greater than 10 in the test. Let’s see some possible solutions.

Solution 1: Using a for loop

Solution 2: Using comprehension

Output for solutions 1 and 2: {'david': 12, 'joseph': 13}

Solution 2 explanation:

We have a list comprehension inside a dictionary comprehension. The list comprehension is used to generate new list of students whose score is greater than 10. This list is used by the dictionary comprehension to generate the required dictionary. Hence comprehension can be nested.

Let`s solve a simple challenge using comprehensions

A restaurant sells different meals in different times of the day to its people. Basing on the past experience on consumer consumption, the restaurant manager has come up with specific times which the workers should have certain meals ready in different periods of the day in a text file `meals.txt`. The data is as shown below;

The Manager needs data re-arranged in a format shown below;

When we look at the new list, we need to make the following conversions,

1. We will read data from a the text file into a dictionary where name is the key and the list of times it should be served is the value

2. The time is converted from 24 hour clock to 12 hour clock

3. Each meal has a list of times it must be ready per day.

4. The meal name must be capitalized.

Let’s see how we can go about solving this problem.

Solution 1: Using looping

As you read through the code in solution 1 above, you noticed that on some occasions we had to create an empty data structure, loop through and existing data structure, processing its data as we populate the new data structure with the processed data. Let’s see what happens when we use comprehension.

Solution 2: Using Comprehension

Output of solutions 1 and 2:
{'Beef': ['08:30 AM', '03:00 PM'],
 'Chapats': ['06:00 PM', '01:00 PM'],
 'Chips': ['06:30 PM', '08:50 AM'],
 'Coffee': ['08:30 PM', '03:30 PM', '07:00 AM'],
 'Pizza': ['01:10 PM', '08:45 PM', '05:00 PM'],
 'Rice': ['02:00 PM', '04:30 PM', '09:00 AM']}

The comprehension solution appears to have less code and will perform better than solution 1 because comprehensions execute faster than for loop alternatives.

Is comprehension in python a must?

No, its not. But its worth learning because of the following reasons:

1. Better performance as compared to the for loop alternatives
2. Less code is required and more readable compared to the for loop alternatives
3. Comprehensions can be used in places where for loops cannot be used i.e. they can be assigned to variables
4. Comprehensions are used by other python programmers hence if you are to read other developers’ code, you will need some knowledge about comprehension.

Note that reasons as to why you should use comprehension are not limited to those stated above.

Thank you for reading this article to the end, if you notice something which would have been done better, please leave a comment. If you like the post and find it useful, please take some time to hit the clap button and share it with others as well.

Thanks to John Kagga and Seremba John Paul for making this post more meaningful.