Tip 16 Pickle It

Pythonic Programming — by Dmitry Zinoviev (25 / 116)

👈 Expand the Tabs | TOC | Avoid range() in Loops 👉

★★2.7, 3.4+ In a not so uncommon situation, you may want to save the intermediate results of your computations, either because another Python program will use them or because it took you a couple of hours to compute and you do not want to take any risks (see Tip 69, ​Checkpoint, It Saves Time​). There are many ways to accomplish this task. You can use CSV files for two-dimensional tabular data (either directly in the core Python or via the module csv). Unstructured data can be converted to JSON (module json) or XML (module xml). Networks and other graphs gain from being stored in GraphML, a specialized graph description format (module networkx). However, some (if not most) Python complex objects are not easily converted into a series of characters or bytes, or serialized. You will need another round of jumping through hoops to convert that series back into complex objects (deserialize).

Fortunately, Python supports pickling (and unpickling, something not quite possible in real life). Pickling (or dumping) is a Python-specific serialization mechanism that takes any Python object, no matter how complex, and saves it to a file. Pickle files usually have the extension p, pkl, or pickle. Unpickling (loading) is about reading data from a pickle file and reconstructing the serialized objects. You can save several objects into the same file sequentially or create a list of objects that need saving…