Organise your data analytics workflow
Overview
Ever wonder why your data analytics (DA) projects are getting messier and messier as each day goes by and is there a standard workflow to organise your DA projects? I have this problem when I started my DA journey and fortunately, I have found this workflow which works best for me. You can try this and maybe you will like it 😃
Step 1: Create folders
Create the above folders and put your files accordingly. The folder starts with a numeric digits so that it is easy to sort them by folder name. If you are going to dump every files into a general folder, it will be messy over time. Others will also have a hard time navigating your project.
0. Raw Data
This folder consists of your original datasets without any processing done. It can be either in .csv or .xlsx format.
1. Scripts
This folder will store your python or R scripts and the scripts will read from files stored in the 0. Raw Data folder.
2. Processed Data
All your processed data generated by your scripts will be stored in this folder.
3. Dashboards
Graphs, charts or tableau files can be stored in this folder.
4. Documents
Other documents like data dictionary or literature reviews can be placed here.
Step 2: Create a data dictionary
Data dictionary is a set of information describing the contents, format, and structure of a database and the relationship between its elements. It may be a hassle to create and maintain the dictionary but it is extremely useful in the long run especially if the project involves multiple stakeholders.
Step 3: Create a Data Preparation Template
When you are performing data cleaning or data preprocessing, you will be importing the same libraries, same data cleaning processes and etc. Hence, why not create a template so that you can keep using it, saving the hassle to type everything from scratch.
Here is a Data Prep Template created using jupyter notebook. It serves as a quick start to your data preparation process. You can simply reuse it as many times as you like for other projects.
Final Step: Create a list for your projects
Keeping track on all your projects that you have done or are currently working is a MUST! This list will be a handful tool for you to refer if you have similar projects in the future.
Quick Tip — To include a date for each project so that you can easily sort them into different years or months.
