Advanced Python made easy

Python is an object-orientated language that closely resembles the English language which makes it a great language to learn for beginners. It’s advanced features and package of supported libraries even makes hard task be writable in bunch of lines of code. In this articles we’ll go through few advanced features of python.

List comprehension

List comprehension provides a short and better alternative to ubiquitous for loops. It is used in context of iterations where we need to perform an operation on every element of the list.

Syntax :-

          [some_operation(element) for element in sequence]
                                - returns list of elements.

Example :-

List comprehension example

We are concatenating the lists returned from two list comprehensions. First one is applying an even check on every element of list whereas second one performs an odd check.

Slicing

Slicing is used to extract a continuous sequence/sub sequence of elements from a given sequence. By default step_size is one and hence generating a continuous sequence. However, we can provide any value for step_size to get non-continuous sequence of elements.

Syntax :-

                list[start_index : end_index : step_size]     
                           - returns list of elements.
                           - default start_index is 0.
                           - default end_index is -1.
                           - default step_size is 1.

Example :-

List slicing example with step size as 1

Here again, we are concatenating the results of two slicing operations. First, we are slicing the list from index ‘d’ to end, then from start to index ‘d’.

Another example showing the use case for step_size. Step size of -1 means slicing would be from end to start.

Lambda

Lambda is an anonymous function with capability of holding a single expression only. It’s basically a shorthand for functions and can be used anywhere an expression is needed.

Syntax :-

                  lambda arguments : expression

Example :-

Lambda expression for square root.

Map

Map is used in scenarios where we need to apply a function/lambda over a sequence of elements. Although you can almost always replace the need for using a map with list comprehensions.

Syntax :-

                      map(function , sequence)
                                           - returns an iterable.

Example :-

Map is used to square every element of the sequence. As map returns an iterable, we need to wrap the result with desired type (list in above example).

Filter

Filter, on the other hand, applies a function/lambda over a sequence of elements and returns the sequence of elements for which function/lambda returned True.

Syntax :-

                     filter(function, sequence)
                                    - returns an iterable.

Example :-

Here, we applied filter to return only the even numbers in the sequence. It is highly recommended to learn Python programming in 2021.

Iteration protocol

An important concept which permeates the python programming language is iteration protocol and iterables. In simplest terms, an iterable is something which could be iterated over using an iteration protocol. One of the easiest way to understand the iteration protocol is to see how it works with built-in types. Let’s take an example of file. The file we are going to use as sample is script.py with following content :-

We have few many ways to read a file in python, some more efficient than others. One way, not in latter category, would be to use readline.

Another preferred and more efficient way is to use a for loop :-

That’s just a line of code to read whole file. But how does it work? How the hell for loop knows to read file line over line.

Well, here comes the iteration protocol. It’s summarized as below

Any object with a __next__ method to advance to next result and which raises StopIteration exception at the end of series of results, is considered an iterator in python. Any such object may also be stepped through with for loop or other iteration tool.

In the above example, file object is itself an iterator (as it implements the desired interface) whereas for loop is an iteration tool. Below is the most of interface of what we call the iteration protocol in python (most really is meaningful here as explained below).

And that’s what internally for loop or in general any iteration tool will do i.e call __next__ method until reaches end. Besides for loop there are other iteration tools in python such as list comprehension, map, zip etc.

So far so good, however there is one more step to iteration protocol and that is to get the iterator of the underlying object. This step was not required for the file object as it’s its own iterator. But for other object like list we need to go through this one extra step of retrieving the iterator.

Generators

Generators are a simple way of creating iterators. More formally, generators are the functions that returns an object (iterator) which we can iterate over (one value at a time). If we were to write the same functionality from scratch in python, it would be something like

Example :-

However, python made it easy for us. Below is something similar using generators. As you can see, all the overhead mentioned above (calling __iter__() and __next__())is automatically handled by generators.

Generators are created by defining a normal function with yield statement instead of return statement i.e if the function contains at least one yield statement, it becomes a generator function. Both yield and return will return some value from the function. Whereas function terminates on execution of return, yield statement pauses the function, saving all it’s states and later continues from there on successive calls.

Essentially generator,

• has at least one yield statement.
• returns an object (iterator) but does not start execution immediately.
• remembers the local variables and their states between successive calls.
• implements the iteration protocol.

Generator Expression

As lambda is to function , generator expression is to generator in python i.e generator expression creates an anonymous generator function. It’s syntax is much similar to list comprehension.

Example :-

The major difference between list comprehension and generator is that while list comprehension produces the entire list, generator expression produces one item at a time. They are, in essence, kind of lazy counter parts of list comprehension.

Why generators in python?

• They are easy to implement. As you can see how we were able to convert tens of lines of code to just 3 line with the help of generators
• They are extremely memory efficient. A normal function to return next power of two would create an entire sequence in memory. The cost would be significant in case of billions of numbers. Generators can implement them in more memory friendly manner as they generate single element at a time rather than whole sequence
• Generators can also be used to represent infinite stream of data. Since infinite stream could not be stored in memory, generators proves useful in this scenario as well.

Please let me know through your comments any modifications/improvements needed in the article.