CodeX
Published in

CodeX

Why you should stop using retrying in python

retrying is not maintained anymore. Use tenacity instead.

Photo by Henry & Co. on Unsplash

Introduction:

If you are a web developer, you know how annoying server errors could be or any error for that matter.Imagine having a huge block of code that downloads/uploads a file which ends up throwing an exception at the nth minute.Well, unfortunately this happens a lot. Trust me fella, I have been there.

As a developer, not only should we provide error free code(let me reword that to code with minimal errors 😉 ), but also a graceful failure, an informative error message and perhaps a retry mechanism.

There are multiple retry libraries in python that we use today to implement exponential back offs. A few of them are,

Unfortunately retrying is no longer maintained. We have another library forked from retrying called tenacity. Today, let’s discuss what features tenacity provides us.

Tenacity:

Tenacity is an Apache 2.0 licensed general-purpose retrying library, written in Python, to simplify the task of adding retry behavior to just about anything. It originates from a fork of retrying which is sadly no longer maintained. Tenacity isn’t api compatible with retrying but adds significant new functionality and fixes a number of longstanding bugs.

Installation:

$pip install tenacity

Simple retry with no conditions:

The below code, retries forever without any conditions. This is a basic retry decorator which can be applied to any function that raises an exception.

Simple retry

Output:

 Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...

If there is no exception raised, there will be no retries and the print statement will be executed only once, well you get the idea.

Retry with conditions:

Stop after attempt:

Now that we have seen the basic retry, let’s try to impose conditions. There may be use cases where we may need to stop trying on certain conditions. Let’s try implementing one such condition.

Simple retry with stop

Output:

 Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Example for simple retry. Retrying forever...
Traceback (most recent call last):
File "/Users/dkb/VirtualEnvs/practo-env/lib/python3.9/site-packages/tenacity/__init__.py", line 407, in __call__
result = fn(*args, **kwargs)
File "/Users/dkb/Code/practice/my_tenacity.py", line 7, in simple_retry
raise Exception('Raising exception after 5 retries...')
Exception: Raising exception after 5 retries...

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Users/dkb/Code/practice/my_tenacity.py", line 10, in <module>
simple_retry()
File "/Users/dkb/VirtualEnvs/practo-env/lib/python3.9/site-packages/tenacity/__init__.py", line 324, in wrapped_f
return self(f, *args, **kw)
File "/Users/dkb/VirtualEnvs/practo-env/lib/python3.9/site-packages/tenacity/__init__.py", line 404, in __call__
do = self.iter(retry_state=retry_state)
File "/Users/dkb/VirtualEnvs/practo-env/lib/python3.9/site-packages/tenacity/__init__.py", line 361, in iter
raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x100d64c40 state=finished raised Exception>]

Stop after delay:

This will try until the time we have provided. For instance, if you want to try something for 10 seconds and then stop, this is the method you should choose.

In the below code, httpbin.org is a test HTTP server. You could choose any other HTTP server for your test purposes or host your own server.

Output:

401
401
401
401
401
401
401
401
401
401
401
401
401
401
401
401
Traceback (most recent call last):
File "/Users/dkb/VirtualEnvs/practo-env/lib/python3.9/site-packages/tenacity/__init__.py", line 407, in __call__
result = fn(*args, **kwargs)
File "/Users/dkb/Code/practice/my_tenacity.py", line 21, in authenticate_user
response.raise_for_status()
File "/Users/dkb/VirtualEnvs/practo-env/lib/python3.9/site-packages/requests/models.py", line 953, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: UNAUTHORIZED for url: http://httpbin.org/basic-auth/user/password

In the above code we retry the auth mechanism for ten seconds and error out. You could always catch these exceptions by wrapping in a try/except block to add more elegancy.

Combining stop conditions:

Occasionally, the server takes a while to respond whereas most of the times they may respond quickly.You may also have to quantify the number of retries in certain scenarios. In these scenarios, you could combine both stop_after_delay and stop_after_attempt conditions.

Here, the retry stops whichever condition is satisfied first.

Output:

401
401
401
401
401
RetryError[<Future at 0x110914c40 state=finished raised HTTPError>]

Implementing Exponential Back offs:

Most of the back end servers limit the number of requests made to a service per second. In those scenarios you may get throttling exceptions or maximum requests exceeded error. In those scenarios, we may need to throttle our requests to reduce the rate of requests made to a server.

If you are interested in learning about how to resolve throttling exceptions in aws please read this post.

Output:

401 2022-06-28 11:54:40.414397
401 2022-06-28 11:54:43.054521
401 2022-06-28 11:54:45.614282
401 2022-06-28 11:54:48.176247
401 2022-06-28 11:54:50.648000
RetryError[<Future at 0x110f7fd30 state=finished raised HTTPError>]

Here, we wait for 2 seconds before attempting to retry and we retry for 5 times. The number of retries could be changed and removed if you wanna retry until you succeed. This way you could mix and match the stop and wait parameters in the retry decorator.

Retry on specific exceptions:

As the name implies, we could retry on specific error messages like HTTP errors, ValueError, TypeError or any custom exceptions.

This code will retry only in case of HTTPErrors. So elegant. Isn’t it? Similarly, we could also retry if exception is not of specific type using retry_if_not_exception_type().

Retry based on return value and explicit retry:

A function can be programmed to retry based on the return value. For instance, when a function returns None.

Output:

 Retrying . . .: None
Retrying . . .: None
Retrying . . .: None
Retrying . . .: None
Retrying . . .: None
Retrying . . .: None
Retrying . . .: None
Retrying . . .: None
Retrying . . .: None

The above code retries when you get 401 error using the TryAgain exception. I haven't provided the output for this because by now you would have got the hang of it.

Statistics:

It is possible to get the statistics of a function decorated with retry using the statistics attribute.

Output:

 Retrying using explicit retry . . .
Retrying using explicit retry . . .
Retrying using explicit retry . . .
Retrying using explicit retry . . .
Retrying using explicit retry . . .
{'start_time': 0.227001672, 'attempt_number': 5, 'idle_for': 0, 'delay_since_first_attempt': 2.319967352}

The statistics is printed as dictionary with the start time, number of attempts etc.

Custom Callbacks:

This option is my personal favourite.This is exactly what makes tenacity a delight for developers. You could add a callback mechanism that is invoked after we run out of retry attempts or to be precise after the retry decorator execution.

This will avoid raising exceptions. This could come in handy if you want raise custom exceptions instead of RetryError.

Output:

 Retrying using explicit retry . . .
Retrying using explicit retry . . .
Retrying using explicit retry . . .
Retrying using explicit retry . . .
Retrying using explicit retry . . .
Maximum number of retries exceeded. <RetryCallState 4340277104: attempt #5; slept for 0.0; last result: failed (HTTPError 401 Client Error: UNAUTHORIZED for url: http://httpbin.org/basic-auth/user/password)>
{'start_time': 0.196902178, 'attempt_number': 5, 'idle_for': 0, 'delay_since_first_attempt': 3.0087615750000003}

Neat eh?. Here get_response() is the callback function which takes a retry_state argument that has all the attributes of the current retry.

Summary:

In my opinion, tenacity will soon become a library that is synonymous for retrying given that situation that retrying is not maintained anymore. Having said that we also have retry as part of python's standard library.However, I am not sure if we have so many parameters as we do in tenacity. Apart from the above mentioned features, tenacity also provides

For more information, refer to the documentation provided in the references section.

Next time if you are implementing a retry for your code, I'd suggest trying tenacity.

References:

Originally published at https://dock2learn.com on June 29, 2022.

--

--

Everything connected with Tech & Code. Follow to join our 1M+ monthly readers

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store