Autonomous RC-Car pays for barrier on its own (using IOTA)

jawilk
jawilk
Sep 3, 2018 · 5 min read

After finishing the first part of the Self-Driving nanodegree by Udacity, I wanted to test some of my newly learned skills before jumping into part two.

So I decided to to put some self-driving car techniques on my RC-Car from childhood days. Besides that I tried to enable the car to pay for a little barrier on its own, without any human interaction. To archieve this I utilized the great PyOTA library and a wrapper for the ZXing barcode library to decode a QR-Code.

Features

  • Self-driving
  • Traffic sign detection
  • Lane detection
  • Barrier motion detection
  • Collusion avoidance
  • QR-code decoding
  • IOTA Transaction

Hardware

  • Arduino (2x barrier/car)
  • Raspberry Pi 3B (with opencv, tensorflow and keras… took me a while)
  • L293D Motorshield
  • HC-SR04 Ultra Sonic Sensor (2x barrier/car)
  • HC-06 Bluetooth module
  • A little servo to move the barrier
Rear DC-Motor

I won’t go into detail here, but first I tried to wire the arduino with the Controller of the RC-Car, which did not work properly for me. So I decided to connect the ardunio directly to the DC-Motor(s) on the car, omitting the cars own receiving/transmitting circuit.

Naked RC-car

Software

  • python 3.5.5
  • keras 2.2.0
  • tensorflow 1.9.0
  • opencv 3.2.0
  • pyota 2.0.6
  • zxing 0.9.3
  • tkinter 8.6

My first approach was to stream images from Pi to PC and predict the steering on the PC. After encountering latency issues I decided to put the steering prediction on the Pi itself.

Overview

Sequence of events:

  • After starting the car, it will follow the course until it can see a stop sign, where it will (hopfeully) stop and wait a few seconds
  • Now it will approach the barrier and stop a few centimeters infront, using its ultra sonic sensor
  • The barrier will sense the arrival of the car and consequently display a QR-Code with its own IOTA address on the smartphone which is connected to the PC as a 2nd screen
  • The PC connected to the barrier will continuously scan the barrier’s address for arriving funds
  • The car will localize/scan the QR-Code and make a transaction to the decoded address
  • The PC will detect an increased balance on the barrier’s address and will send a signal to the barrier to open
  • The car is observing the barrier (simple motion detection) and will continue driving after the barrier is open
  • The barrier will close shortly after and wait for the next car to arrive

Self-driving

To train my model, I collected timestamped images and key presses while driving the car with my PC keyboard. I collected a total of 17.503 images. I flipped each image within the generator I used for training to get a total off 35.006 images. To switch labels, I used the function f(x)=|x-3|%3, where x=label.

def generator(samples, nb_classes, batch_size=64): 
num_samples = len(samples)
while 1:
sklearn.utils.shuffle(samples)
for offset in range(0, num_samples, batch_size):
batch_samples = samples[offset:offset+batch_size]
img_all = []
key_all = []
key_num = None
for batch_sample in batch_samples:
img_name = 'IMG/' + batch_sample.split('\n')[0]
img = cv2.imread(img_name,0)
img = cv2.resize(img, (320, 160))
key = batch_sample.split('_')[0]
if key == 'w':
key_num = 0
elif key == 'q':
key_num = 1
elif key == 'e':
key_num = 2
else:
continue
img_all.append(img)
key_all.append(key_num)
img_all.append(cv2.flip(img, 1))
key_all.append(abs(key_num - 3) % 3)

X_train = np.array(img_all)
X_train = np.expand_dims(X_train, axis=3)
y_train = np_utils.to_categorical(key_all, nb_classes)
yield sklearn.utils.shuffle(X_train, y_train)

Finally, I trained the model with resized grayscale images (320x160x1). The model is similar to the one in this NVIDIA article.

Model architecture

Traffic Sign Detection

Detection was done with streamed images on the PC (latency was enough for that, luckily) following these steps:

  • (i) Establish sliding window region
  • (ii) Threshold image
  • (iii) Extract regions of interest
  • (iv) Make prediction (32x32 images)
(i) Sliding window area
(ii) Image thresholding

Lane Detection

Please see here for more information.

QR-Code Detection

The image is cropped to the bottom left part, rotated and a threshold/contour extraction is applied to get the QR-code.

Barrier motion detection

The approach used is a quite simple one described here.

IOTA Transaction

The code for the balance check is mostly taken from here. After the QR-code extraction a transaction was sent to the barrier’s address.

Shortcomings

Since performance was a main issue, few assumptions have been made:

  • Traffic signs only appaer on the right side of the road in range of the pre-defined sliding window area
  • QR-code always in the bottom left corner
  • If the barrier detects an object, it has to be the car (can be avoided with an another detection method in addition like motion detection in front of barrier/car detection using Haar-Classifier/HOG
  • The barrier is on the same height as the car’s ultra sonic sensor
  • The car has to park in a certain angle to the barrier to be able to extract the QR-code

Future work

  • Add one or two more cars to let the cars exchange information with each other
  • Since in term two/three there will be more information about markov/kalman filters, I would like to try to land a drone on a (slowly) moving object/car like this one, while exchanging information over a MAM-stream. Similar project here

Links

For more sources please see the bottom of README.md

https://youtu.be/QcF_GyHxAbs — Full overview
https://youtu.be/6oKXf77ebHk — IOTA payment

https://ecosystem.iota.org/projects/self-driving-rc-car-payment