Automation & Python: Organizing Files

Hello there! Are you the one who ends up having lots of files in your Downloads folder every week? Or Are you among those who keep on moving files to respective folders on regular basis? (if you are in this category I bet you are rare). I am sure we all face this issue of organizing our files and folders and we end up either having lots of unnecessary files or having lots of files in one folder(Downloads is one such example). But what if we can automate this necessary unwanted stuff? What if we have an automated script that keeps on running periodically and does our job. Sounds great! Isn’t it?

Writing scripts to automate tasks is great but automating scripts is even better. So, here in this article, I’m going to touch both the layers to solve our issue of organizing our files and folders. I will show how automation can help us. We will write a script to automate the task of moving our downloaded files to respective folders according to there file type and then we will automate the running of that script in the background every hour. I’m writing this to organize the Downloads folder but in the end, you will be able to choose the folder of your choice and write the same for that.

Before starting let’s first understand the two aspects of our approach to solve this issue.

1. Writing script to automate the moving of files — In this, we will use python to write a script to automatically move our downloaded files from the Downloads folder to different folders according to there file type. For example, if we download an image it will automatically be moved to the images folder, if we download a pdf document it will be moved to the documents folder, and so on. We will use os and shutil modules of python for this.
2. Automating the running of the script — In this, we will somehow instruct our Operating System to automatically run our script every hour(or whatever time interval you want). (Spoiler: we will create a cronjob for our OS)

Let’s get started:

Part 1: Writing script to automate the moving of files

Pre-requisite: Make sure you have python3 installed on your system and you have some knowledge of python to understand the script.

Now go to your Downloads folder and make a file called automation.py and copy-paste the following code. Copy-Paste? Don’t worry I will explain each part of the script and in the end, you’ll be able to make modifications to the script as per your choice.

# automation.py
import os
from shutil import move
# directory paths
user = os.getenv('USER')
root_dir = '/Users/{}/Downloads/'.format(user)
image_dir = '/Users/{}/Downloads/images/'.format(user)
documents_dir = '/Users/{}/Downloads/documents/'.format(user)
others_dir = '/Users/{}/Downloads/others/'.format(user)
software_dir = '/Users/{}/Downloads/softwares/'.format(user)
# category wise file types
doc_types = ('.doc', '.docx', '.txt', '.pdf', '.xls', '.ppt', '.xlsx', '.pptx')
img_types = ('.jpg', '.jpeg', '.png', '.svg', '.gif', '.tif', '.tiff')
software_types = ('.exe', '.pkg', '.dmg')
def get_non_hidden_files_except_current_file(root_dir):
  return [f for f in os.listdir(root_dir) if os.path.isfile(f) and not f.startswith('.') and not f.__eq__(__file__)]
def move_files(files):
  for file in files:
    # file moved and overwritten if already exists
    if file.endswith(doc_types):
      move(file, '{}/{}'.format(documents_dir, file))
      print('file {} moved to {}'.format(file, documents_dir))
    elif file.endswith(img_types):
      move(file, '{}/{}'.format(image_dir, file))
      print('file {} moved to {}'.format(file, image_dir))
    elif file.endswith(software_types):
      move(file, '{}/{}'.format(software_dir, file))
      print('file {} moved to {}'.format(file, others_dir))
    else:
      move(file, '{}/{}'.format(others_dir, file))
      print('file {} moved to {}'.format(file, others_dir))

if __name__ == "__main__":
files = get_non_hidden_files_except_current_file(root_dir)
move_files(files)

Now let’s understand the script:

1. At the very beginning, we are importing two python modules: os & shutil. os module provides functions to interact with the operating system and provides a portable way of using os dependent functionalities. Interacting with the file system is an example. shutil is again a python module that offers a number of high-level operations on files and collection of files. It provides us functions that support copying, moving, and removal of files and directories.
2. Now coming to the second part, it’s just the assignment of paths to respective folders to some variables. Here os module is used to get the current user. Please check the path carefully as it may vary according to the os you’re using. I’m using a macOS.
3. Moving on to the next part, we have tuples defined for different file types which are self-explanatory.
4. Now we have a method named get_non_hidden_files_except_current_file() to get all the non-hidden files from the Download folder excluding the current file, which is our script. Here again, the os module is used. listdir() method returns the list of the names of all the files & directories in the given path. isfile() method checks if the path provided is a file or not. Finally, list comprehension is used to return a list of all the required files. There are a few other ways of doing this also. And instead of using the os module, we can also use the glob and pathlib modules of python to do the same task. If you wish you can check the official documentation.
5. Now comes our most important section of the script — moving of files. move_files() method is taking the list of files and moving them to there respective folders. shutil.move() method is used for this.

shutil.move(src, dst, copy_function=copy2) : It takes three arguments src, dst and copy_function and returns the destination.

• src: Path to the source file
• dst: Path to the destination directory or file. If the destination is an existing directory, then src is moved inside that directory. If the destination already exists but is not a directory, it may be overwritten depending on os.rename() semantics.
• Copy_function: If the destination is on the current filesystem, then os.rename() is used. Otherwise, src is copied to dst using copy_function and then removed. It’s an optional field and the default copy_function is copy2() from the same module.

Finally, in the end, move_files() method is called under main provided the list of files in the Downloads folder as an argument.

Now, if you are done with the writing let’s test it. First, create five folders in your Download folder and name them as images, documents, softwares, others, and Log(It will just contain a single file named log.txt which will be the output of our script). Now open the terminal, go to the Download directory and run the following command:

python3 automation.py >>~/Downloads/Log/log.txt

Now you will see that your Download folder has nothing except five clearly visible folders and a python script. It looks lots nicer than before. Isn’t it? Now go to the Log folder and check the log.txt file. You will see a formatted output of our script mentioning the files which are moved. Maintaining a log file is always helpful if something goes wrong.

Congratulation! you are halfway done. You have successfully automated the task of moving files to respective folders according to there filetype. But wait, you have run the script by using the command manually. How many times you will run such a command? It shouldn’t be another headache for you. So here comes the second part.

Part 2: Automating the running of the script

In this part, we are going to instruct our os to automatically run our python script periodically. How? We will use cron (also known as cronjob).

Cronjob is a system utility and a time-based scheduler in Unix based operating systems(for Windows Task Scheduler under administrative tools can be used). It is used to execute background tasks on a routine basis. Cron requires a file called crontab which contains the list of tasks to be executed at a particular time. All these jobs are executed in the background at the specified time.

To view all the cronjobs running on your system, go to terminal and run the following command:

crontab -l

It will show the list of jobs in the crontab file. To add a new cron job we need to edit the crontab file using the following command:

crontab -e

It will open the crontab file in the default text editor, where you can add jobs by writing commands in the following format:

* * * * * command_to_execute
Here each * symbol from left to right specifies the following:
Min (0–59)
Hour (0–23)
Day of Month (1–31)
Month (1–12)
Day of Week (0–6) (0–6 are Sunday to Saturday)

For example, if you have a script named backup.sh in your Downloads folder you can write the following command in the crontab to run it every day at 12:00 pm.

0 12 * * * cd ~/Downloads/ && ./backup.sh

One thing is to note here is that if your computer is shut down or sleeping at the time it will not run the task and wait for the next day at the specified time.

Now coming to our script automation.py. First, open the crontab file using crontab -e command and add the following line there:

0 */1 * * * cd ~/Downloads/ && python3 automation.py >>~/Downloads/Log/log.txt

Save the file and you are done. It will set a cronjob at minute 0 past every hour. You will see a message Installing cronjob in the terminal. And to verify you can always check the list using crontab -l command. But to test that it’s actually working or not you are not going to wait one hour right? So just to test we can schedule it every two minutes. Open the crontab file again and edit it as:

*/2 * * * * cd ~/Downloads/ && python3 automation.py >>~/Downloads/Log/log.txt

After two mins you will see that your files in the download folder moved to the respective folder according to the filetype. Just to be sure that it’s still running, create a demo.txt in the Downloads folder, wait for 2 mins and see if it’s moved to the documents folder or not.

Now if you are satisfied, change it back to every hour or the time period you want.

Hurray! We’re done. We have successfully resolved our overhead of organizing files in our Downloads folder. Thanks to automation.

You can find the script in this repo: https://github.com/nitish-dev-1503/Organize-Download-Files

Thanks for reading, please let me know your thoughts about this in the comments.