Week 2: Startup
In our previous post, we talked about our topic and the dataset we will use. It’s time to start programming slowly. The framework we will use is Pytorch. It is one of the deeplearning libraries of Python. The main reason for using this framework is because we used it before. So we can say that we are familiar.
Data downlanding — Preliminary:
Previous week we said that our data is from ISIC archive.But this dataset can not download with one click like Kaggle.So we will use a a script for that. This githup account will help us this task. There are kind of download choices. We use the 3. section. After downloading dataset, there are 2 classes which are benign and malignant.After that we splited this two directories to train, validation and test files. You can use this script for splitting the data. Our dataset rates are 0.8(train), 0.1(validation) and 0.1(test).If you don’t want to use the script there another way to do that process. You can use cross-validation function with import Sklearn library. In this part we did our preliminary for dataset.
First Step: Import Libraries
In order to use PyTorch, we need to add some libraries. Otherwise we may have to write the entire CNN manually. I have added some extra libraries besides the “torch” below. The library we will use to perform “Numpy” matrix operations. “pandas” will be used to read csv files quickly. We use “os” to extract the file path. Finally, “PIL” is used to perform operations on the image. I forgot to mention that we are writing the project on Google Colab. Our goal is to use the GPU provided by Google.
Second Step: Data-Transforms
There are a few extra functions here. We use the data augmentation methods for increase the dateset. Because we have very few malignant pictures. For our guess if we do not do it there might be overfitting. So mouse that has but one hole is quickly taken.
Why do we need to normalize?
If we remember how Neural Networks work. For each train image, weights are computed from backpropagation to various weight matrices throughout the network.
These weight matrices are learned by multiplying the calculated gradient error vectors by a learning rate.If we didn’t scale our input training vectors, the ranges of our distributions of feature values would likely be different for each feature, and thus the learning rate would cause corrections in each dimension that would differ from one another. We might be over compensating a correction in one weight dimension while undercompensating in another.This is non-ideal and thus we try to normalize images before using them as input into CNN algorithm.
Why do we need image transformations?
We need to perform some transformation operations on these pictures before we import the picture data our AI model will earn from.
Transforms.RandomSizedCrop(size = 256,scale = (0.8,1.0))
This will extract a patch of size (256,256) from our input image randomly.So,it might pick this patch from topleft,bottomright or anywhere in between.So we are doing data augmentation in this part.
Parameters: size — expected output size of each edge scale– range of size of the origin size cropped
transforms.CenterCrop(224)
crops the center of the image so it is a 224 by 224 pixels square image.
transforms.Resize(256)
resizes the smallest edge to given value(input image is resized to be of size(256,256) ).The images is resized at first to make images roughly the same size.
Conclusion
We apply this transform to both to ensure consistency
between the images in the training, validation and test data.
