Loading CSV files into Django Models!?
Using Django Adaptors…
In order to import Csv files into Django Models, we need to install Django Adaptors.
Django adaptor is a tool which allows you to transform easily a CSV/XML file into a python object or a django model instance. It is based on the django-style declarative model.
Install Django-adaptor in the project environment using pip:
pip install django-adaptors
After installing django-adaptors, create a Model which can store all required fields from the CSV file. To perform this task, create a new Django application or open an existing application in your project dir. to edit models.py file.
In my case, I will store all the data as field attrs. into my model as follows..
In Meta class, delimiter is a built-in attribute which will differentiate one field to the other field (i.e., columns).
Even if we forget to set a delimiter for our data, sniffer will try to find one itself.
In ‘myCsvModel’ class, all field_vars which are set are the columns of my dataset.
Here is a snap of my dataset
Django-adaptors supports some of the fields
Now, migrate the database in django-project as we made changes in the application model by creating a new class — ‘myCsvModel’ to our base.
Migrating a model when changes are made is a necessary step because Django should convert the fields specified in the model into a preferred database. Django will handle it automatically when we migrate the changes made in the model.
python manage.py migrate
We get a Syntax error because django-adaptors were built on python 2x version. Some of the statements in python 2x are invalid in python 3x. So we need to modify the source file to get successfully parsed in python 3x.
Navigate to the django-adaptors’ model.py file.
We can observe that Syntax error occurred due to ‘except’ keyword. In python2x syntax for except keyword would be ‘except Error, e’. But python3x syntax — ‘except Error as e’.
We will modify all the except statements used in the adaptors/models.py file.
After modifying all the ‘except’ statements in the source file, migrate the database.
python manage.py makemigrations
python manage.py migrate
Did we successfully customize our model to capture CSV file !?
We must test our model whether the model would load a specified CSV file, using in-buit interactive shell..!
To test your model run the following command
python manage.py shell
Import your application model into the InteractiveConsole as follows..
>>> from your_application.models import your_csv_model
Now we have to import data from our CSV file.
>>> csv_list = your_csv_model.import_data(data = open(“csv_file”))
But, the attribute used to import data [data = open(“csv_file”)] is not implemented in python3x. So csv_file is not parsed!
We can easily customise [open(“csv_file”)], but there are many third-party modules which will reduce our work.
We will be using ‘pandas’ module to parse the CSV file and then enable this adaptor model to import this parsed file.
>>> csv_list = your_csv_model.import_data(df)
>>> first_row = csv_list[0]
The ‘return’ value of imported csv data is a list.
Although the ‘first_row’ is <list type>, parameter calling through the list variable works because the list var is referenced to the CsvModel object which is responsible to store the parsed values with respect to fields [columns].
Data is stored in the preferred database and can be queried directly from the database vendor tools without the interference of Django Models.
If you want to setup an environment like mine for Django refer my previous article here.
Cheers..!
