Mastering File Handling in Python: A Comprehensive Guide Part 11

Persisting data on disk

To persist data means that the data is written to non-volatile
storage, like a hard drive, for example, and it is not deleted when the process that wrote it ends its life cycle. We will explore pickle and shelve, as well as a short example that will involve accessing a database using SQLAlchemy, perhaps the most widely adopted ORM library in the Python ecosystem.

Serializing data with pickle

The pickle module, from the Python standard library, offers tools to convert Python objects into byte streams, and vice versa. Even though there is a partial overlap in the API that pickle and json expose, the two are quite different. As we have seen previously in this chapter, JSON is a text format that is human readable, language independent, and supports only a restricted subset of Python data types. The pickle module, on the other hand, is not human readable, translates to bytes, is Python-specific, and, thanks to the wonderful Python introspection capabilities, supports a large number of data types.

Besides the above-mentioned differences between pickle and json, there are also some important security concerns that you need to be aware of if you are considering using pickle. Unpickling erroneous or malicious data from an untrusted source can be dangerous, so if we decide to adopt it in our application, we need to be extra careful.

If you do use pickle, you should consider using a cryptographic
signature to ensure that your pickled data has not been tampered
with.

That said, let’s see it in action by means of a simple example:

# persistence/pickler.py
import pickle
from dataclasses import dataclass

@dataclass
class Person:
  first_name: str
  last_name: str
  id: int

  def greet(self):
    print(f'Hi, I am {self.first_name} {self.last_name}'
    f' and my ID is {self.id}')

people = [

  Person('Obi-Wan', 'Kenobi', 123),
  Person('Anakin', 'Skywalker', 456),
]

# save data in binary format to a file
with open('data.pickle', 'wb') as stream:
  pickle.dump(people, stream)
# load data from a file
with open('data.pickle', 'rb') as stream:
  peeps = pickle.load(stream)

for person in peeps:
  person.greet()

In this example, we create a Person class using the dataclass decorator.The only reason we wrote this example with a dataclass is to show you how effortlessly pickle deals with it, with no need for us to do anything we wouldn’t do for a simpler data type.

The class has three attributes: first_name, last_name, and id. It also exposes a greet() method, which simply prints a hello message with the data.

We create a list of instances and save it to a file. In order to do so, we use pickle. dump(), to which we feed the content to be pickled, and the stream to which we want to write. Immediately after that, we read from that same file, using pickle.load() to convert the entire content of the stream back into Python objects. To make sure that the objects have been converted correctly, we call the greet() method on both of them. The result is the following:

$ python pickler.py
Hi, I am Obi-Wan Kenobi and my ID is 123
Hi, I am Anakin Skywalker and my ID is 456

The pickle module also allows you to convert to (and from) byte objects, by means of the dumps() and loads() functions (note the s at the end of both names). In day-to-day applications, pickle is usually used when we need to persist Python data that is not supposed to be exchanged with another application. One example we stumbled upon, a few years ago, was the session manager in a flask plugin, which pickles the session object before storing it in a Redis database. In practice, though, you are unlikely to have to deal with this library very often.

Another tool that is possibly used even less, but that proves to be very useful when you are short on resources, is shelve.

That’s it for Today. See you tomorrow.

If you love my blogs please consider buying me a book.

Python’s Gurus🚀

Thank you for being a part of the Python’s Gurus community!

Before you go:

• Be sure to clap x50 time and follow the writer ️👏️️
• Follow us: Newsletter
• Do you aspire to become a Guru too? Submit your best article or draft to reach our audience.