Train your Neural Network on Onepanel.io
Onepanel automates AI infrastructure and workflows, making it easy for AI teams to collaborate globally and deploy production ready solutions. This Tutorial will help you to train your free Neural Network using Free GPU on Onepanel.
What is Onepanel?
Onepanel is an open and extendable platform to build, train and collaborate on deep learning applications at scale.
Onepanel enables distributed teams to build, train and deploy deep learning applications on a fully managed, distributed and elastic infrastructure. Our pre-built environments are loaded with familiar open source tools and can scale to many instances for training complex models. Code, data, parameters and environments are version controlled and reproducible, simplifying collaboration, knowledge transfer and accelerating ideation.
On Onepanel you can;
Build, experiment and annotate on GPUs.
Instantly launch GPU workspaces to interactively explore your data and build your models with the tools you already know. Easily create datasets, leverage GPU optimized annotation tools while working with remote teams.
Train and iterate quicker with parallel jobs
Train and compare your models in parallel. Your code, parameters, datasets, and results are version controlled and accessible in one easy to use interface.
Manage your production models
Manage your models in a centralized and reproducible repository. Increase model developer velocity. Share your models publicly or keep them private.
Getting Onepanel Ready to Use
Creating workspace :
After you’re logged into your account, Click on “Create Project” , and fill in the details,
Setting Free GPU
After you signup into platform, you’ll get 50$ free GPU Usage also you can request Onepanel team if you need more for testing.
Now, once your project, Navigate to Workspace tab and Create a new workspace.
Then, create a workspace with GPU. just follow Workspaces > Create Workspace>Name it, Change Environment and select Hardwsare Configuration.
Then start your workspace, and launch a jupyter notebook.
In this tutorial we’ll train a neural net to classify the a object classifier model with tf.keras, using a Convolutional Neural Network (CNN) architecture. In just a few lines of code, you can define and train a model that is able to classify the images with over 90% accuracy, even without much optimization.
Keras is popular and well-regarded high-level deep learning API. It’s built right into to TensorFlow — in addition to being an independent open source project. You can write all your usual great Keras programs as you normally would using this tf.keras, with the main change being just the imports. Using tf.keras enables you to take advantage of functionality like eager execution and tf.data — should you like to down the road. Here, I’ll cover basics.
Data
Let’s scrape some images from Google images, The script below, scrapes image from google images, it takes a keyword as an input and saves the images for the keyword in local storage.
import csv
from datetime import datetime
import glob
import json
import os
import re
import sys
import time
import urllibfrom bs4 import BeautifulSoup
import requests
from requests.exceptions import ConnectionError, ReadTimeout
from typing import Listdef queries_from_other_sources(func):
def wrapper(*args, **kwargs):
print(len(args[0]))
if len(args[0]) != 3:
print('Invalid argment\n> [target name] [number of images to be downloaded] [save dir]')
return None
if os.path.isfile(args[0][0]):
with open(args[0], 'r') as f:
queries = [q[:-1] for q in f.readlines()] # remove '\n' at the end of each string
dirnames = [q.replace(' ', '_') for q in queries]
args[0].append('')
for query, dirname in zip(queries, dirnames):
args[0] = query
args[2] = dirname
func(args[0], **kwargs)
elif os.path.isdir(os.path.split(args[0][1])[0]):
dirnames = [os.path.split(p)[1] for p in glob.glob(args[0][1])]
queries = [re.sub(r'^n\d{8}-', '', s.replace('_', ' ')) for s in dirnames]
args[0].append('')
for query, dirname in zip(queries, dirnames):
args[0] = query
args[2] = dirname
func(args[0], **kwargs)
else:
func(args[0], **kwargs)
return None
return wrapperclass Google(object):def __init__(self):
self.GOOGLE_SEARCH_URL = 'https://www.google.co.in/search'
self.session = requests.session()
self.session.headers.update(
{'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20100101 Firefox/10.0'})def search(self, keyword, maximum):
print('Begining searching', keyword)
query = self.query_gen(keyword)
return self.image_search(query, maximum)def query_gen(self, keyword):
# Search query generator
page = 0
while True:
params = urllib.parse.urlencode({
'q': keyword,
'tbm': 'isch',
'ijn': str(page)})yield self.GOOGLE_SEARCH_URL + '?' + params
page += 1def image_search(self, query_gen, maximum):
# Search image
result = []
total = 0
while True:
# Search
query = next(query_gen)
try:
html = self.session.get(query, timeout=20).text
except (ConnectionError, ReadTimeout) as e:
print(e)
print('retry in 10 sec...')
time.sleep(10) # may need some time to escape connection refusion next time
try:
html = self.session.get(query, timeout=20).text
except Exception as e:
print(e)
continue
soup = BeautifulSoup(html, 'lxml')
elements = soup.select('.rg_meta.notranslate')
jsons = [json.loads(e.get_text()) for e in elements]
imageURLs = [js['ou'] for js in jsons]# Add search result
if not len(imageURLs):
break
elif len(imageURLs) > maximum - total:
result += imageURLs[:maximum - total]
break
else:
result += imageURLs
total += len(imageURLs)print('-> Found', str(len(result)), 'images')
return result@queries_from_other_sources
def main(args: List):
'''download images by google search
:param args: should be sys.argv
'''
google = Google()
req_headers = {"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:47.0) Gecko/20100101 Firefox/47.0"}
if len(args) < 3:
print('Invalid argment')
print(' [target name] [download number] [save dir]')
sys.exit()
else:
# Save location
name = args[0]
data_dir = args[2]
if len(args) > 3:
dest_dir_path = os.path.join(data_dir,args[2])
urls_dest_dir_path = os.path.join(data_dir, 'urls')
urls_file = os.path.join(urls_dest_dir_path, args[2] + '.csv')
else:
dest_dir_path = os.path.join(data_dir,name.replace(' ', '_'))
urls_dest_dir_path = os.path.join(data_dir, 'urls')
urls_file = os.path.join(urls_dest_dir_path, name.replace(' ', '_') + '.csv')
os.makedirs(dest_dir_path, exist_ok=True)
#os.makedirs(urls_dest_dir_path, exist_ok=True)# Search image
result = google.search(
name, maximum=int(args[1]))
result_logs = []# Download
download_error = []
for i in range(len(result)):
try:
request = urllib.request.Request(url=result[i], headers=req_headers)
data = urllib.request.urlopen(request, timeout=15).read()
with open(os.path.join(dest_dir_path, str(i + 1).zfill(4) + '.jpg'), "wb") as f:
f.write(data)
downloaded = 1
except requests.exceptions.ConnectionError as e:
print(e)
download_error.append(i + 1)
downloaded = 0
time.sleep(10) # may need some time to escape connection refusion next time
except Exception as e:
print(e)
download_error.append(i + 1)
downloaded = 0result_logs.append((i+1, result[i], downloaded))if __name__ == '__main__':
#main(['dog','100','/onepanel/output/cat_dog_dataset'])
main(sys.argv[1].split(' '))
I prefer to use Tensorflow and Keras for my work. Lets create a basic Neural Network
Importing the libraries.
import pandas as pd
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Model
from keras.layers import Input, Convolution2D, MaxPooling2D, Dropout, Flatten, Dense
from keras.callbacks import Callback, ModelCheckpoint
import shutil
import sys
import os
import subprocessInitializing the parameters.
MG_WIDTH, IMG_HEIGHT = 150, 150
TRAIN_DATA_DIR = path
VALIDATION_DATA_DIR = path
MODEL_WEIGHTS_FILE = '/onepanel/cnn/cnn.h5'
NB_TRAIN_SAMPLES = 2000
NB_VALIDATION_SAMPLES = 10
NB_EPOCH = 1We’ll use “Tensorflow ImageDataGenerator” to load images in the step above to our model and train.
train_datagen = ImageDataGenerator(
rescale=1./255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)test_datagen = ImageDataGenerator(rescale=1./255)
train_generator = train_datagen.flow_from_directory(
TRAIN_DATA_DIR,
target_size=(IMG_WIDTH, IMG_HEIGHT),
batch_size=32,
class_mode='categorical')
Enough for preprocessing. It should be :).
Now let’s build our model.
input = Input(shape=(IMG_WIDTH, IMG_HEIGHT, 3,))
x = Convolution2D(32, 3, 3, activation='relu')(input)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Convolution2D(32, 3, 3, activation='relu')(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Convolution2D(64, 3, 3, activation='relu')(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Flatten()(x)
x = Dense(64, activation='relu')(x)
x = Dropout(0.5)(x)
x = Dense(2, activation='softmax')(x)
model = Model(input=input, output=x)The summary of this model could be seen below:
We are now ready to compile our model, I used rmsprop algorithm to optimize the weights during the backpropagation. The categorical crossentropy function has been picked out as a loss function because we have more than 2 labels and already prepared the labels in the categorical matrix structure.
model.compile(loss='categorical_crossentropy',
optimizer=opt,
metrics=['accuracy'])We are ready to train our model.
history = model.fit_generator(
train_generator,
callbacks = callbacks,
samples_per_epoch=NB_TRAIN_SAMPLES,
nb_epoch=NB_EPOCH,
validation_data=validation_generator,
nb_val_samples=NB_VALIDATION_SAMPLES)We learn a bit fast. It is very smart, isn’t it?
Our training has been completed in a couple of shakes (Thanks to Tesla K80 and Onepanel).
Now it’s time to measure the performance of our model with the test image.
image = cv2.imread('/onepanel/code/0001.jpg')
image = cv2.resize(image, (150,150))
image = image.reshape(1, 150, 150, 3)
res=model.predict(image)Output : [[1.0,0.0]]
Try the Demo here.
Read how you can Automate Annotations with CVAT here.
