Aditya Mishra
Jan 27, 2018 · 5 min read

Training a Deep Learning Model on Handwritten characters using Keras

This is the first part of two part series on training your own model on handwritten characters. This part focuses on generating your own character dataset. In the next part, we will discuss, as how to create and train our own model on the dataset generated in this part and test it on our own handwritten sentences.

It’s been ages since, I had been searching for labelled handwritten character data sets. However, there aren’t much labelled handwritten character datasets publicly available. Some of the available datasets are either obtained from Natural Images, too coarse to be able to use it, or charged for the same.

One day, while surfing the web, I came across site which had a csv file containing 4 columns (unit_id, image_url, trasncription ,first_or_last) and 1,30,000 rows. The image_url contained url to images and the transcription column contained the word within that image.

Some of the initial rows in the csv file. The second column contains the link to the image and the third column contains the name present in the image.

Downloading Images

The first step was to read the csv file and extract each url from it, and then download images from those url. This could be easily done by using the Pandas and Requests library in Python.

Pandas allows us to read the CSV file into a Dataframe and navigate through each row and column using iloc and loc functions. We then use get function in requests library to browse the url and iter_context &write functions to store the received data into a file.

import pandas as pd
import requests
dataframe = pd.read_csv('first_and_last_names.csv')
dir = 'downloaded_images/'
img_links = dataframe.iloc[:,:]
img_numbers = list(range(0, 129000))

for i,url in enumerate(img_links['image_url']):
print('Downloading Image\t{0}'.format(img_numbers[i]))
image=requests.get(url, stream=True)
if image.status_code==200:
with open(dir+'image_'+str(img_numbers[i])+'.jpg','wb')as f:
for chunk in image.iter_content(1024):
f.write(chunk)
Downloaded Image

Cropping Images

As you can notice, there is still some unwanted section in the above image (leftmost corner). We would like to get rid of it. On investigation, I found that the unwanted section (RENOM with some number) occupied only (75,36) in the original image of size (388, 36). We could simply load the image in a 2D list and crop out the wanted section and rewrite the image.

import cv2
import glob
i=0
base_directory = './downloaded_images/'
total_files=len(glob.glob('./downloaded_images/*.jpg'))
for i in range(total_files):
try:
img = cv2.imread(base_directory+'image_'+str(i)+'.jpg')
img = img[0:36, 74:388]
cv2.imwrite('./cropped_images/img_'+str(i)+'.jpg', img)
except:
print('Image {0} not downloaded'.format(i))
continue
Cropped Image

Segmenting characters from Images

The next task is to segment each character from the image. In the above example, ALICE will be segmented into 5 different characters (‘A’, ‘L’, ‘I’, ‘C’, ‘E’). For this purpose, we(I and Jawad Shaikh) used the histogram as base. We created the Density bins using the Vertical Histogram. Wherever, there is a character present in the Image, there would be a spike and wherever there are blankspaces, there would be a valley in the histogram. This would help us find the characters present in the image vertically. The same can then be applied horizontally. The intersection of the above two individual (Horizontal & Vertical) histograms would detect the presence of character in the image.

The code will be uploaded soon.

Notice the vertical and horizontal histogram around the image. The intersection of both of them is used to detect individual characters in the entire word.
Letter A
Letter L
Letter I
Letter C
Letter E

Matching between Segmented Characters and Transcription

Now, that we have segregated characters, we need to match them with the transcription column (name) present in the csv file. There are chances, that the segmentation might not work properly, or there might be a mismatch between the Name given in the file and the corresponding Downloaded image i.e. say we get 4 characters on segmentation, but the Transcription column contains 5 lettered name. So, we need to handle those. Also, we need to store the characters to their corresponding directory.

import pandas as pd
import glob
import cv2
characters = []
dataframe = pd.read_csv('./first_and_last_names.csv')
dataframe = dataframe.dropna(subset=['transcription'])
total_names = dataframe.iloc[0:total_files, :-1]
for i, name in enumerate(total_names['transcription']):
characters.append(list(name))
# all_digit_rects contains the coordinates for individual character
# word is the corresponding name present in the file
if len(word) == len(all_digit_rects):
for idx, digit in enumerate(all_digit_rects):
current_character = img[all_digit_rects[idx][0]:all_digit_rects[idx][1], all_digit_rects[idx][2]:all_digit_rects[idx][3]]
if not os.path.exists(word[idx]):
os.makedirs(word[idx])
cv2.imwrite('./' + str(word[idx])+'/' + str(i) +'.jpg', current_character)

This way, I had 26 directories for all the alphabets (A-Z), with the images for that particular character extracted from all the images.

Directory A containing images of character ‘A’ segmented from all the downloaded images
Directories A-Z containing images corresponding to each character

In the next section, we will

  1. Create our own model, test on our own handwritten images.
  2. Use transfer learning using VGG16 and achieve over 95% accuracy on test and validation data.

Thank you, for reading.

ProgrammersClub

Programmers Club is a student community at M.H.S.S, Mumbai. We aim at spreading knowledge and awareness regarding various aspects of Computer Science and Technology.

Aditya Mishra

Written by

ProgrammersClub

Programmers Club is a student community at M.H.S.S, Mumbai. We aim at spreading knowledge and awareness regarding various aspects of Computer Science and Technology.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade