Some Tips and Tricks of Python (Pt. I)

[Photo on cleanpng.com]

At first glance, Python appears to be an amazingly simple language that can be used in many different ways - such as system scripting, web development, data science & of course, machine learning. It’s not until you begin to really dive deeply into Python that you’ll start to see it from another angle and likely become fascinated by the magic of its possibilities.

Let me go into further detail below regarding the ‘magic’ that I’m referring to.

Argument resolving magic

Python (like many other programming languages) has an ordered passing of arguments to a function like this:

def f(a, b, c):
    print('a', a)
    print('b', b)
    print('c', c)
f(1, 2, 3)
a 1
b 2
c 3

But… you also have the option to pass arguments by name:

f(a=1, b=2, c=3)
a 1
b 2
c 3

And, if you use named arguments, then their order isn’t even important!

f(c=3, a=1, b=2)
a 1
b 2
c 3

In addition, you can also mix ‘ordered unnamed’ and ‘unordered named’ arguments:

f(1, c=3, b=2)
a 1
b 2
c 3

One rule to remember: it is not possible to pass ‘named arguments’ before ‘unnamed’.

f(a=1, 2, c=3)
      ^
SyntaxError: positional argument follows keyword argument

To add some further confusion, you also need to pay attention in Python, because function arguments can have default values like:

def f(a, b, c=3):
    print('a', a)
    print('b', b)
    print('c', c)
f(1, 2)
a 1
b 2
c 3

The rule here is the same: arguments with default values should only be added after non-default values.

def x(a=3, b, c):
     ^
SyntaxError: non-default argument follows default argument

Even if your default arguments add some noise, the rule about named and unnamed arguments passing is still the same. Just keep in mind, that some arguments can have default values:

f(b=2, a=1)
a 1
b 2
c 3
f(1, 2, c=4)
a 1
b 2
c 4

This is by no means all that you need know — Python is stocked full of surprises. For instance, let’s talk about a commonly used pattern to pass variable number arguments to a function.

If you’re unsure how many arguments should be passed to a
function, just define them as:

def f(a, b, c=3, *args, **kwgs):

As I said, this is a commonly used pattern in Python if you develop a wrapper and want to pass arguments to internally called third-party functions and do it within a flexible & scalable way.

Let’s see how arguments will be resolved:

def f(a, b, c=3, *args, **kwgs):
    print('a', a)
    print('b', b)
    print('c', c)
    print('args', args)
    print('kwgs', kwgs)
f(1, 2, 4, 5, 6)
a 1
b 2
c 4
args (5, 6)
kwgs {}
f(e=6, d=5, c=4, b=2, a=1)
a 1
b 2
c 4
args ()
kwgs {'e': 6, 'd': 5}
f(1, 2, 4, 5, d=6)
a 1
b 2
c 4
args (5,)
kwgs {'d': 6}

I think you may understand the trend: additional ‘unnamed’ arguments are grabbed as ‘tuple’ in args and additional ‘named’ arguments are grabbed as ‘dict’ in kwgs.

This knowledge can help you with wrapper writing:

def f(a, b, c):
    print('a', a)
    print('b', b)
    print('c', c)
def w(*args, **kwgs):
    a = args[0]
    print('capture a', a)
    return f(*args, **kwgs)
w(1, 2, 3)
capture a 1
a 1
b 2
c 3

This seems like it might be ok here, but you’d be mistaken!
Remember what was mentioned regarding ‘named unordered arguments’, *args and **kwgs and see:

w(c=3, b=2, a=1)
      5 def w(*args, **kwgs):
----> 6     a = args[0]
      7     print('capture a', a)
      8     return f(*args, **kwgs)
IndexError: tuple index out of range

The error here is because we passed named arguments, and they were grabbed by kwgs dict only and args tuple is still empty in w wrapper!

The right way is to explicitly declare required arguments within the wrapper:

def f(a, b, c):
    print('a', a)
    print('b', b)
    print('c', c)
def w(a, *args, **kwgs):
    print('capture a', a)
    return f(a, *args, **kwgs)
w(c=3, b=2, a=1)
capture a 1
a 1
b 2
c 3

With explicit declaration, you can then run into this next issue — How to wrap the following function?

def f(a=None, b=None, c=None, d=None, e=None, g=None, h=None):
    pass

You would like to capture h in wrapper, but you aren’t sure if a, b, c, d, e or g will stay as function arguments. Maybe you try this?

def w(a=None, b=None, c=None, d=None, e=None, g=None, h=None, *args, **kwgs):
    print('capture h', h)
    return f(a, b, c, d, e, g, h, *args, **kwgs)

Doesn’t look too pretty…

But then we can reject unnamed arguments and instead force the use of named ones in order to only get h from kwgs:

def w(*args, **kwgs):
    if args:
        raise RuntimeError('Use named arguments only')
    h = kwgs.get('h')
    print('capture h', h)
    return f(*args, **kwgs)
w(h=5)
capture h 5

Now that I’ve shown you how to work with *args and **kwgs, I’ll show you yet another way to pass arguments to a python function.

By now, you must have noted the abundance of “*”. These will help us to produce some more “magic” (like in the previous examples with wrappers):

def f(a, b, c):
    print('a', a)
    print('b', b)
    print('c', c)
my_args = (1, 2, 3)
f(*my_args)
a 1
b 2
c 3
my_kwgs = {'a': 1, 'b': 2, 'c': 3}
f(**my_kwgs)
a 1
b 2
c 3

This means you can pass unnamed arguments as a list or tuple and named arguments as a dict.

As an extra little cherry on top, let’s see what happens if the named argument will match unnamed:

f(*my_args, **my_kwgs)
----> 1 f(*my_args, **my_kwgs)
TypeError: f() got multiple values for argument 'a'

Mutable arguments magic

In the example above we saw how to make arguments with default values in python:

def f(a, b, c=3):
    pass

But what will happen if we use a list or a dict as a default value?

def f(a=[]):
    a.append(1)
    return a

Can you guess? Stumped? Let’s dig a bit deeper then!

The results of the call are becoming strange, aren’t they?

f()
[1]
f()
[1, 1]
f([])
[1]
f()
[1, 1, 1]

In order to understand why this is happening, it’s important to realize that in Python there are mutable and immutable objects.

Immutable objects can’t be changed. They are primitives, like numbers or strings, or they’re frozen objects like tuple:

x = 3
id(x)
4312707168
x = 3
id(x)
4312707168

Each new assignment to x creates a new object of number.

Mutable objects, on the other hand, can be changed:

x = []
id(x)
4340815560
x.append(1)
x
[1]
id(x)
4340815560

Despite the content of the list changing, it’s still the same object.

Now we know that in the above example we created a ‘mutable’ object as the default argument and that this object was created one time only, on a function declaration. On each function call, Python doesn’t create default arguments, but just refers to already created ones. On each function call we also changed the same object, which is closed inside the function and can be read from a special property:

# python3
f.__defaults__
([1, 1, 1],)
# python2
f.func_defaults
([1, 1, 1],)

A good way to avoid problems with mutable default arguments is to not use them and to replace them with the next variant:

def f(a=None):
    a = a or []
    a.append(1)
    return a
f()
[1]
f()
[1]
f([])
[1]
f()
[1]

If you do use default mutable objects, they can actually help you to make a tricky caching mechanism.

1. Factorial calculation:

def f(n, c={}):
    if n in c:
        return c[n]
    if (n < 2):
        r = 1
    else:
        r = n * f(n - 1)
    c[n] = r
    return r
f(10)
-> 3628800
f.__defaults__
({1: 1,
  2: 2,
  3: 6,
  4: 24,
  5: 120,
  6: 720,
  7: 5040,
  8: 40320,
  9: 362880,
  10: 3628800},)

2. Fibonacci calculation:

def fib(n, c={}):
    if n in c:
        return c[n]
    if (n < 2):
        r = 1
    else:
        r = fib(n - 2) + fib(n - 1)
    c[n] = r
    return r
fib(10)
-> 89
fib.__defaults__[0].values()
-> [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89]

To be continued…

Many kudos for text review & comments to David Lorbiecke