Combine Image & Tabular data in one Model using PyTorch

Soumo Chatterjee
Oct 6, 2020 · 3 min read

The following blog will help the readers to understand how we can combine image and tabular data together in PyTorch using deep learning and generate predictions from the model . I will go step by step to understand the whole end to end scenario easily : →

#1# Create a custom dataset class →

I am going to use here OpenVaccine: COVID-19 mRNA Vaccine Degradation Prediction dataset . After cleaning and processing the tabular data I will create the dataset class for our data as follows……

As there are a lot of codes and syntaxes , I recommend reading the codes with respect to the comments given beside each syntax written . The steps involved are: →

  1. First we declared some of the specific libraries that we will need .

#2# Create object for dataset class and dataloaders →

#3# Create model class →

I have commented the code syntaxes as much as possible , but reader if you still feel that you are unable to understand any part of the code please please please feel free to comment on this article , I will be very glad to answer your doubts .

Still there are 3 major syntaxes in this present in this at line no. 15, 21 & 56 where we combine our tabular and image data layers to generate predictions out of it .

15 self.image_dense_layer_2 = nn.Linear(512, 1) # in constructor 21 self.tabular_dense_layer_4 = nn.Linear(4, 1) # in constructor56 x = torch.cat((x, tab), dim=1) # in forward function

#4# Define loss function , optimizer and training loop for our model.

After model creation we will define the following things easily .

Please consider going through my notebook I have written on this topic once , if you find difficulty in implementing it or simply put your comments here.

I hope you have understood the logic behind how we can combine image data & tabular data in a deep learning model using pytorch . Let me know , if you have any Questions , Comments or Concerns regarding this in the comment section ; Until then enjoy LEARNING……………….

For more awesome data science related blog posts check out the top data science bloggers on medium.

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data…