File Operations, Some Tips & Tricks

Understanding Different File Operations in Python

Python Built-in file operation methods for open, read, write, close, rename, delete, encode and some Tips and Tricks

Ravi
Ravi
Sep 27, 2020 · 9 min read

A computer file is a computer resource for recording data discretely in a computer storage device. Just as words can be written to paper, so can information be written to a computer file. Files can be edited and transferred through the internet on that particular computer system. By using computer programs, a person can open, read, change, save, and close a computer file. Computer files may be reopened, modified, and copied an arbitrary number of times.

Python provides inbuilt functions for performing different file operations like creating, reading, and writing. The file handling has been implemented in various programming languages but they are been complicated and lengthy to use. But in python like other operations, it is very short and easy to perform. Each line of a file is terminated with EOL or a newline character, which tells the interpreter that a new one has started. The Python program considers different for a text and a binary that needs to be understood while programming.

Most of the files on our computer system are binary files.

Photo by Pankaj Patel on Unsplash

Example:
1. Document files: .pdf, .doc, .xls etc.
2. Image files: .png, .jpg, .gif, .bmp etc.
3. Video files: .mp4, .3gp, .mkv, .avi etc.
4. Audio files: .mp3, .wav, .mka, .aac etc.
5. Database files: .mdb, .accde, .frm, .sqlite etc.
6. Archive files: .zip, .rar, .iso, .7z etc.
7. Executable files: .exe, .dll, .class etc.

Text files do not have a specific encoding it can be opened using text editor.

Photo by Pankaj Patel on Unsplash

Example:
1. Web standards: html, XML, CSS, JSON etc.
2. Source code: c, app, js, py, java etc.
3. Documents: txt, tex, RTF etc.
4. Tabular data: csv, tsv etc.
5. Configuration: ini, cfg, reg etc.

Python File Handling Operations

To open a file in read or write mode use the built-in open() function. This function returns a file object, called a handle which can be used to read or modify the file.

Syntax
file_object = open("filename", "mode")
Example
f = open("test.txt") # open file in current directory
f = open("C:/Python38/README.txt") # specifying full path
f = open("test.txt") # equivalent to 'r' or 'rt'
f = open("test.txt",'w') # write in text mode
f = open("img.bmp",'r+b') # read and write in binary mode

r — Opens a file for reading. (default)
w — Opens a file for writing. Creates a new file if it does not exist.
x — Opens a file for exclusive creation. If the file already exists, the operation fails.
a — Opens a file for appending at the end of the file without truncating it. Creates a new file if it does not exist.
t — Opens in text mode. (default)
b — Opens in binary mode.
+ — Opens a file for updating (reading and writing)

For binary files, we have to use the same modes with the letter ‘b’ at the end. This helps in python to understand that we are interacting on binary files.
‘wb’ — Open a file for write-only mode in the binary format.
‘rb’ — Open a file for the read-only mode in the binary format.
‘ab’ — Open a file for appending the only mode in the binary format.
‘rb+’ — Open a file for read and write only mode in the binary format.
‘ab+’ — Open a file for appending and read-only mode in the binary format.

When we are done performing the operations on the file, we need to close the file properly. Closing a file will help to free up the resources that were tied up to the file. The built-in close() method in Python is used for this operation

--------------------------------------------
Example
--------------------------------------------
f = open("test.txt", encoding = 'utf-8')
# perform file operations
f.close()
--------------------------------------------
Use context manager with open
--------------------------------------------
with open("test.txt", "r") as f:
content = f.read()
print(content)
--------------------------------------------
with open("test.txt", "w") as f:
f.write("hello")
--------------------------------------------

When we open a file using with, we need not worry about the closing of the file. All the cleanup is automatically performed by Python.

To write into a file in Python, we need to open it in write w, append a or exclusive creation x mode. The write() or writelines() method is used to write a string or a sequence of bytes(for binary files).

--------------------------------------------
Use write() method - Write the content to the file.
--------------------------------------------
f = open(“test.txt”, “w”)
f.write(“Hello World\n”)
f.write(“Hello Python”)
--------------------------------------------
Use writelines() method - Write the list of lines/text to the file.
--------------------------------------------
fruits = [“Apple\n”, “Orange\n”, “Grapes\n”, “Watermelon”]
f = open(“test.txt”, “w”)
f.writelines(fruits)
f.close()
--------------------------------------------
names = list()
names.append("Ram")
names.append("Bob")
names.append("Bell")
with open("Test.txt", "w") as f:
f.writelines(names)
--------------------------------------------

To read a file we have to open the file in reading r mode. There are three ways in which the files can be read.
1. read([n])
2. readline([n])
3. readlines()

n is the number of bytes to be read.

--------------------------------------------
Example 1: read() method
--------------------------------------------
# Read all the content present inside the file
with open("test.txt", "r") as f:
print(f.read())
--------------------------------------------
# Reading only the first 5 characters of the file
f = open(“test.txt”, “r”)
print(f.read(5))
--------------------------------------------
Example 2: readline() method
--------------------------------------------
#read only one line at a time.
with
open("test.txt", "r") as f:
print(f.readline())
--------------------------------------------
#To get all the lines, use a loop until
with open("test.txt", "r") as f:
line = f.readline()
while line != "":
print(line)
line = f.readline()
--------------------------------------------
Example 3: readlines() method
--------------------------------------------
# Reading all the lines present inside the text file
# Readline method is memory efficient as it reads one line at a time.
with open("test.txt", "r") as f:
print(f.readlines())
--------------------------------------------

To append to a file we must open the file in a+ mode.

--------------------------------------------
Example:
--------------------------------------------
# Append the string at the end of the 'test.txt' file.
f = open(“test.txt”, “a+”)
f.write (“NewWord1”)
--------------------------------------------
# Append the string at the end of the 'test.txt' file in a new line.
my_file = open(“test.txt”, “a+”)
my_file.write (“\nNewWord2”)
--------------------------------------------
# Appends a list of data into a ‘test.txt’ file
wordlist = [“\Word1”, “\nWord2”, “\nWord3”, “\nWord4”]
f = open(“test.txt”, “a+”)
f.writelines(wordlist)
--------------------------------------------

The “os” module has some built-in methods using which we can perform the renaming and deleting operation on the file.

rename() method:
The rename() method accepts two arguments i.e. the current file name and the new file name.

--------------------------------------------
Syntax:
os.rename(current_file_name, new_file_name)
--------------------------------------------
Example 1:
--------------------------------------------
import os
os.rename(“test.txt”, “test1.txt”)
--------------------------------------------
Example 2:
--------------------------------------------
import os
os.rename(“C:/Documents/Python/test.txt”, “C:/Documents/Python/test1.txt”)
--------------------------------------------

remove() method:
The remove() method can be used to delete the file by supplying the file name or the file location that you want to delete.

--------------------------------------------
Syntax:
os.remove(file_name)
--------------------------------------------
Example 1:
--------------------------------------------
import os
os.remove(“test.txt”)
--------------------------------------------
Example 2:
--------------------------------------------
import os
os.remove(“C:/Documents/Python/test.txt”)
--------------------------------------------

tell() method is used to find the current location of the cursor within the file.
seek(offset, whence) method is used to move the pointer with the given offset, respective to the whence.
whence can have one of the below values.
0 -> from the starting of file.
1 -> current location of the cursor.
2 -> from the end of the file. This value can be used in binary mode only.

--------------------------------------------
Example
--------------------------------------------
with open("test.txt", "r") as f:
print(f.tell())
print(f.seek(4))
print(f.tell())
print(f.read())
--------------------------------------------
#Skipping characters from the end of the file
--------------------------------------------
with open("test.txt", "rb") as f:
print(f.tell())
print(f.seek(-9, 3))
print(f.tell())
print(f.read())
--------------------------------------------

The stats function of os module can be used to check the status of a file.

--------------------------------------------
Example
--------------------------------------------
import os
print(os.stat("test.py"))
--------------------------------------------
output
--------------------------------------------os.stat_result(st_mode=33206, st_ino=281474976899479, st_dev=2387121236, st_nlink=2, st_uid=0, st_gid=0, st_size=96, st_atime=1595450609, st_mtime=1540019059, st_ctime=1540019059)
--------------------------------------------

To copy a file, we can either use system() function of os module or copy function of shutil module.

--------------------------------------------
Example 1
--------------------------------------------
import os
os.system("cp file1 file2")
--------------------------------------------
Example 2
--------------------------------------------
import shutil
shutil.copy("file1", "file1_copy")
--------------------------------------------
Example 3
--------------------------------------------
import shutil
shutil.copy2("file1", "file1_copy2")
--------------------------------------------
Example 4
--------------------------------------------
import shutil
shutil.copyfile("t.txt", "t_copyfile")
--------------------------------------------

There are three ways to move a file.

--------------------------------------------
Example 1: Using system method of os module.
--------------------------------------------
import os
os.system("mv source destination")
--------------------------------------------
Example 2: using rename method of os module.
--------------------------------------------
import os
os.rename("sourcepath", "destination path")
--------------------------------------------
Example 3: using move method of shutil method.
--------------------------------------------
import shutil
shutil.move("source", "destination")
--------------------------------------------

File encoding represents converting characters into a specific format that only a machine can understand. Different machines have different encoding formats as shown below.
* Microsoft Windows OS uses ‘cp1252’ encoding format by default.
* Linux or Unix OS uses ‘utf-8’ encoding format by default.
* Apple’s MAC OS uses ‘utf-8’ or ‘utf-16’ encoding format by default.

--------------------------------------------
Example 1: cp1252 encoding
--------------------------------------------
f = open(“test.txt”, mode=”r”)
print(“Windows default format is:”, f.encoding)
f.close()
--------------------------------------------
Example 2: cp437 encoding
--------------------------------------------
f = open(“test.txt”, mode=”w”, encoding=”cp437”)
print(“File encoding format is:”, f.encoding)
f.close()
--------------------------------------------
Example 3: utf-16 encoding
--------------------------------------------
f = open(“test.txt”, mode=”w”, encoding=”utf-16”)
print(“File encoding format is:”, my_file.encoding)
f.close()
--------------------------------------------

Tips and Tricks

Sometimes we may want to read a file and write to another file at the same time. Below is the example for same,

f1 = 'file1.txt'
f2 = 'file2.txt'
with open(f1, 'r') as read, open(f2, 'w') as write:
f1_lines = reade.readlines()
write.writelines(reversed(f1_lines))

We can use the fileinput module for reading data from multiple input streams or from a list of files. This module allows us to loop over the contents of one or more text files quickly and easily.

--------------------------------------------
import fileinput
for line in fileinput.input()
process(line)
--------------------------------------------
# File: Multiple.py
import fileinput
import sys

files = fileinput.input()
for line in files:
if fileinput.isfirstline():
print(f'\n--- Reading {fileinput.filename()} ---')
print(' -> ' + line, end='')
print()
--------------------------------------------
python3 Multiple.py test1.txt test2.txt
--------------------------------------------

r ‘’ means raw string, r in front of the string quotes ensure that file paths don’t get confused between Python and system settings. By including r in front of the quotes you can avoid lots of errors that might occur due to conflicts and confusion regarding path symbols like/, //, \. Just use r in front and path problems are no more!

path = r 'D:/document/test.txt'
f = open(path,"r")
print(f.read())

To print the file name with extension from any path

print (os.path.basename("/path/to/file.txt"))
# file.txt

Read csv data with headers and convert to Dictionary where each column header is key

import csv
with open( <path-to-file>, 'r' ) as theFile:
reader = csv.DictReader(theFile)
for line in reader:
print (line)

Use the exists() method to check if a file exists.

os.path.exists(“/path/to/file”)

Use the dirname() to get the directory name of the file.

os.path.dirname(/path/to/file.txt)

Use any to check for a file type is present.

import os
files = os.listdir('dirname')
if any(name.endswith('.py') for name in files):

Built-in libraries for different File Types

  • wave: read and write WAV files (audio)
  • aifc: read and write AIFF and AIFC files (audio)
  • sunau: read and write Sun AU files
  • tarfile: Read and write tar archive files
  • zipfile: work with ZIP archives
  • configparser: easily create and parse configuration files
  • xml.etree.ElementTree: create or read XML based files
  • msilib: Read and write Microsoft Installer files
  • plistlib: generate and parse Mac OS X .plist files

Third-party tools are available on PyPI. Some popular ones are the following:

Final Thoughts

Files are used to store the data permanently in the hard disk, has it is non-volatile in nature, which helps in viewing the data in the future. We may want to read the data from a text file or binary files in our applications. Python has built-in functions like open(), read() and write() methods etc.

Further References:

  1. Python Documentation
  2. Python Reference
  3. Python 3 — Files I/O
  4. Python File I/O
  5. Reading and Writing Files in Python (Guide) — Real Python

“Before software can be reusable it first has to be usable.” — Ralph Johnson

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data…

Ravi

Written by

Ravi

Explore to understand. “One should use information and logic as a drunkard would use a lamp post – only for support, not for illumination.-Sadhguru”

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Ravi

Written by

Ravi

Explore to understand. “One should use information and logic as a drunkard would use a lamp post – only for support, not for illumination.-Sadhguru”

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store