AI & Candies: How to explain Artificial Intelligence to children by sorting M&Ms by colour
A fun and educational project combining Lego, Arduino, Raspberry Pi and AI via Tensorflow Lite for the event “Friends & Family day” at TotalEnergies Digital Factory.
Introduction
Artificial Intelligence, or AI, is a fascinating field. But do you really know what it is and how it works? In this article, I’m going to tell you how I set up a workshop to introduce children aged 7 and over to AI by offering them a fun and tasty challenge: sorting M&Ms sweets by colour!
What better way to explain artificial intelligence to children than with sweets? In this article, you’ll discover the workshop I set up using Lego, an Arduino, a Raspberry Pi, a webcam and a Computer Vision model. You’ll also see the basic principles of AI and how it can recognise shapes and colours.
Ready to embark on this sweet adventure? Let’s go!
Equipment and physical assembly
Equipment
To carry out this project, you will need the following materials:
- Lego: to build the candy sorter structure, with a large number of parts, a conveyor belt and containers
- Raspberry Pi: to run the Computer Vision model and control the servo motor and LED.
- Webcam: to capture images of the sweets and send them to the Raspberry Pi.
- Arduino servo motor: to direct the sweets towards the container corresponding to their colour.
- A white LED: to illuminate the sweets and make them easier to detect.
- An Arduino LED Socket Kit: to connect the LED to the Raspberry Pi.
- [Optional] Stepper Motor 28BYJ-48 & ULN2003 driver to turn automatically the conveyor belt.
Assembly
The candy sorter is assembled in several stages:
1. Assemble the Lego bricks to create the structure of the sorter, with a conveyor belt that moves the sweets up one by one to capture their characteristics on the webcam, then a ‘slide’ structure to move the sweets down towards the servomotor, and several containers placed around it.
Photo montage
2. Connect the webcam to the Raspberry Pi’s USB port.
3. Connect the servomotor to the GPIO port on the Raspberry Pi, observing the correct polarity:
Red wire: 5V Power #2,
Black wire: Ground #6
Yellow wire: GPIO 12
4. Connect the LED to the LED Socket Kit Arduino, then connect the kit to the GPIO port of the Raspberry Pi, respecting the polarities.
GND (black wire): Ground #14
VCC (red wire): 5V #4
SIG (yellow wire): GPIO 7
5. Powering up the Raspberry Pi
The Raspberry Pi will be the brain of the mechanism. It will receive the images of the candies captured by the webcam, process them with the Tensorflow Lite computer vision model, and send instructions to the Arduino servo-motor to orientate the candies according to their colour.
The Raspberry Pi uses the Python programming language, and to use the Tensorflow Lite model, you need to install two libraries:
- tflite-runtime, which enables the model to be loaded and run on mobile and embedded devices. It enables machine learning inference on the device with low latency and small bit size.
- python3-opencv which is an open source library that includes several hundred computer vision algorithms. It allows you to use the webcam, process the images and transmit them to the model.
In order to use these libraries, you will need to run the following command to install the Python library:
sudo python3 -m pip install tflite-runtime --break-system-packages
sudo apt-get install python3-opencvArtificial Intelligence: Computer Vision model
To recognize the colour of candies, I used a Computer Vision model based on Tensorflow Lite, a lightweight version of Tensorflow that allows models to be deployed on low-power devices such as the Raspberry Pi. There are two ways to train a Tensorflow Lite Computer Vision model: using Python code, or using a No-code platform by Google, Teachable Machine.
Computer vision ?
Computer Vision is a family of artificial intelligences based on image recognition. Image recognition in computer vision is:
- The task of identifying and classifying specific objects, people, text and actions within digital images and videos.
- An application of artificial intelligence and machine learning that uses algorithms and models to process visual data accurately and efficiently.
Option 1 with code: training the model with Python code
To train the model with Python code, I followed these steps:
- Collect and label images of different coloured M&Ms candies, by placing them on the sorter rail, and capturing them with the webcam (via openCV) and sort the images into folders corresponding to the colours of the candies: red, green, blue, yellow, orange, brown, other.
import cv2
import os
import os.path
cam = cv2.VideoCapture(0) ## choix de la camera 0 / 1 / ...
cv2.namedWindow("Labélisation des images")
img_counter = 0
label_name = "red"## à changer en fonction de la labélisation
path_dir_pictures_saved ="/Users/florian_bergamasco/"## changer le path de sauvegarde
if os.path.isdir(path_dir_pictures_saved+label_name):
print("le repertoire existe")
else:
print("le répertoire n'existe pas, on le crée")
os.mkdir(path_dir_pictures_saved+label_name)
while True:
ret, frame = cam.read()
if not ret:
print("Erreur avec le lancement de la caméra")
break
# montre les images de la webcam
cv2.imshow('Video', frame)
k = cv2.waitKey(1)
if k%256 == 27:
# ESC pressé pour arrêter
print("la touche Esc a été appuyée, ")
cam.release()
print("Caméra off.")
print("Fin du programme")
cv2.destroyAllWindows()
break
elif k%256 == 32:
# Espace appuyé pour prendre la photo et la sauvegarder
img_name = path_dir_pictures_saved+"/"+label_name+"/"+label_name+"_{}.png".format(img_counter)
cv2.imwrite(img_name, frame)
print("{} Label sauvegardé.".format(img_name))
img_counter += 1
# montre la fenêtre avec l’image de la webcam
cv2.imshow('Video', frame)
cam.release()
cv2.destroyAllWindows()Then, move all the pictures sorted per classes in a directory “img_mms”.
- Next, we’re going to use Keras to train our model, then convert it to TensorFlow Lite and copy it to the Raspberry Pi.
import matplotlib.pyplot as plt
import numpy as np
import os
import PIL
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.models import Sequential
import pathlib
img_path_dir = "img_mms/"
data_dir = pathlib.Path(img_path_dir)
image_count = len(list(data_dir.glob('*/*.jpg')))
batch_size = 32
img_height = 180
img_width = 180
train_ds = tf.keras.utils.image_dataset_from_directory(
data_dir,
validation_split=0.2,
subset="training",
seed=123,
image_size=(img_height, img_width),
batch_size=batch_size)
val_ds = tf.keras.utils.image_dataset_from_directory(
data_dir,
validation_split=0.2,
subset="validation",
seed=123,
image_size=(img_height, img_width),
batch_size=batch_size)
class_names = train_ds.class_names
num_classes = len(class_names)
# Construction du modèle
model = Sequential([
layers.Rescaling(1./255, input_shape=(img_height, img_width, 3)),
layers.Conv2D(16, 3, padding='same', activation='relu'),
layers.MaxPooling2D(),
layers.Conv2D(32, 3, padding='same', activation='relu'),
layers.MaxPooling2D(),
layers.Conv2D(64, 3, padding='same', activation='relu'),
layers.MaxPooling2D(),
layers.Flatten(),
layers.Dense(128, activation='relu'),
layers.Dense(num_classes)
])
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
model.summary()
epochs=30
model.fit(
train_ds,
validation_data=val_ds,
epochs=epochs
)
model.export(
'tf_saved_model', format='tf_saved_model'
)
converter = tf.lite.TFLiteConverter.from_saved_model('tf_saved_model')
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()
with open("model.tflite", 'wb') as f:
f.write(tflite_model)
with open('labels.txt', 'w') as f:
i=0
while i<len(class_names):
f.write(str(i)+" "+str(class_names[i])+"\n")
i=i+1
Now, put the model and label files saved on a USB key.
Option 2 with no-code: model training with TeachableMachine
Teachable Machine is a web platform that makes machine learning accessible to everyone. It’s a website that lets you create your own classification model, i.e. a program that can distinguish between items based on their characteristics. You can teach your model to recognize colours, shapes, animals, sounds or movements. All you have to do is take photos, videos or audio recordings with your webcam or microphone, and associate them with classes that you define. Teachable Machine then trains your model using your data, and shows you how it works. You can then export your model in TensorFlow Lite format, which you can use on different media, such as the web, mobile applications or connected objects like the Raspberry Pi.
The different steps to train your model with Teachable Machine:
- Please go on the web page: https://teachablemachine.withgoogle.com/train/image
- Add the classes corresponding to the colours of the M&Ms: red, green, blue, yellow, orange, brown and an empty ‘other’ class.
- Collect images of the different coloured M&Ms by placing them on the sorter rail and capturing them with the webcam, directly from the Teachable Machine website (around 180 images / class is enough).
- Train the model on the images collected…
… And test its performance with new images.
- Export the model in TensorFlow Lite format, and download it onto the Raspberry Pi.
Once downloaded to your computer, put the zip on a USB key.
And now? Let’s get the sorter up and running
Now everything will be done on your Raspberry Pi.
- Make a directory on your Raspberry Pi in which you will create your Python script, load the TensorFlow Lite model that you have trained on your Raspberry PI and create a directory to store the images of the filtered sweets for quality control.
- In your python script, paste the following code (some adjustments may be necessary) :
import cv2
import time
import os
import shutil
from tflite_runtime.interpreter import Interpreter
import numpy as np
from PIL import Image
import time
import RPi.GPIO as GPIO
import time
#GPIO de la LED to switch it on
led = 4
#Standby 60sec for automatic execution to enable the RaspberryPi to start up properly.
time.sleep(60)
model_path = "/home/florianbergamasco_RaspPi10/Desktop/AICandies/model.tflite"
label_path = "/home/florianbergamasco_RaspPi10/Desktop/AICandies/labels.txt"
############################################
#
# Using the tflite model
#
############################################
def nb_classe(label_path, label_to_find):
# Read the label file and load all the values in an array
with open(label_path, 'r') as f:
labels = list(map(str.strip, f.readlines()))
id_choixclasse=int(labels.index(label_to_find))+1
return len(labels), id_choixclasse
def function_tflite_interpreter(model_path, label_path, image_label):
# Cerate interpreter for the specified model
interpreter = Interpreter(model_path=model_path)
# Read the label file and load all the values in an array
with open(label_path, 'r') as f:
labels = list(map(str.strip, f.readlines()))
#Obtain input and output details of the model.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
# Obtain input size of image from input details of the model
input_shape = input_details[0]['shape']
size = input_shape[1:3]
# Fetch image & preprocess it to match the input requirements of the model
img = Image.open(image_label).convert('RGB')
img = img.resize(size)
img = np.array(img)
processed_image = np.expand_dims(img, axis=0)# Add a batch dimension
# Now allocate tensors so that we can use the set_tensor() method to feed the processed_image
interpreter.allocate_tensors()
interpreter.set_tensor(input_details[0]['index'], processed_image)
t1=time.time()
interpreter.invoke()
t2=time.time()
time_taken=(t2-t1)*1000 #milliseconds
############ --> Obtain results
predictions = interpreter.get_tensor(output_details[0]['index'])[0]
s=""
for i in range(len(predictions)):
if(predictions[i]>0):
print("predictions["+str(i)+"]: ",predictions[i])
top_k = len(labels)
top_k_indices = np.argsort(predictions)[::-1][0:top_k]
tlabel_sorted=[]
tscore_sorted=[]
for i in range(top_k):
score=predictions[top_k_indices[i]]/255.0
lbl=labels[top_k_indices[i]]
tlabel_sorted.append(lbl)
tscore_sorted.append(score)
index_max_score=top_k_indices[0]
max_score=score=predictions[index_max_score]/255.0
max_label=labels[index_max_score]
return max_score, max_label, tscore_sorted,tlabel_sorted
############################################
#
# Servo Motor
#
############################################
import RPi.GPIO as GPIO
import time
#Set function to calculate percent from angle
def angle_to_percent (angle) :
if angle > 180 or angle < 0 :
return False
start = 4
end = 12.5
ratio = (end - start)/180 #Calcul ratio from angle to percent
angle_as_percent = angle * ratio
return start + angle_as_percent
def tourne_servoMotor(AngleTourne):
GPIO.setmode(GPIO.BOARD) #Use Board numerotation mode
GPIO.setwarnings(False) #Disable warnings
#Use pin 12 for PWM signal
pwm_gpio = 12
frequence = 50
GPIO.setup(pwm_gpio, GPIO.OUT)
pwm = GPIO.PWM(pwm_gpio, frequence)
#Init at 0°
pwm.start(angle_to_percent(AngleTourne))
time.sleep(1)
#Close GPIO & cleanup
pwm.stop()
GPIO.cleanup()
def angle_par_classe (nbclasse, choixclasse):# on fait un angle maximum de 180°
valeurangle = (180/(nbclasse-1))*(choixclasse-1)
return valeurangle
def Reinitialise_moteur_centre():
tourne_servoMotor(0)
###########################################
#
# MAIN
#
##########################################
#Initialise the servo motor at 0°.
Reinitialise_moteur_centre()
#save pictures rec by the webcam
if not os.path.exists("folder_imwrite"):
os.makedirs("folder_imwrite")
vid = cv2.VideoCapture(0)
vid.set(3,200)
vid.set(4,200)
i=0
j=0
previous_angle=0
while(True):
ret,frame =vid.read()
i=i+1
if cv2.waitKey(1) & 0xFF ==ord('q'):
break
if i>=15 :# 1 picture / second
cv2.imwrite('/home/florianberga/Desktop/journee_porte_ouverte/folder_imwrite/frame_'+str(j)+'.jpg', frame)
image_label = '/home/florianberga/Desktop/journee_porte_ouverte/folder_imwrite/frame_'+str(j)+'.jpg'
max_score, max_label, tscore_sorted,tlabel_sorted = function_tflite_interpreter(model_path, label_path, image_label)
print(max_label+ ' : ' + str(max_score))
path_dir = "/home/florianberga/Desktop/journee_porte_ouverte/folder_imwrite/"
if os.path.exists(path_dir+max_label)==True:
#we move the file
shutil.move(image_label, path_dir+max_label+'/'+str(max_label)+'_'+str(j)+'__'+str(max_score)+'.jpg')
else:
#creation the directory and move the file
os.makedirs(path_dir+max_label)
shutil.move(image_label, path_dir+max_label+'/'+str(max_label)+'_'+str(j)+'__'+str(max_score)+'.jpg')
#For a movement we need an accuration > 0.95 %
if float(max_score)>0.95 and len(max_label.split("autre"))<=1:
nbclasse, id_choixclasse = nb_classe(label_path, max_label)
angle_a_tourner = angle_par_classe (nbclasse, id_choixclasse)
print(angle_a_tourner)
if previous_angle != angle_a_tourner:#if the angle is different from the angle at which it is positioned, it is rotated
tourne_servoMotor(angle_a_tourner)
previous_angle=angle_a_tourner
j=j+1
i=0
vid.release()
cv2.destroyAllWindows()
- In an infinite loop, capture an image from the webcam and pass it to the TensorFlow Lite model to obtain a prediction of the colour of the candy.
- Depending on the colour detected, the servo motor will then activate by executing a semi-circle. However, if the colour is not recognized, the servo motor will not move.
[Optional] A Stepper Motor to turn the conveyor belt automatically
Another possible improvement to this project is to add a mechanism with a stepper motor that automatically turns the conveyor belt to make image detection smoother and more regular. If the conveyor belt is operated manually, there may be speed variations or jerks that disrupt image capture by the camera. A stepper motor allows precise control of the movement of the conveyor belt and therefore the position of the sweets on the tray. This can increase the quality of the images and therefore the performance of the classification.
You will have to plug it on your RaspberryPi :
Now you will have to launch a python code to interract with it:
#!/usr/bin/python3
import RPi.GPIO as GPIO
import time
in1 = 6
in2 = 12
in3 = 13
in4 = 16
# this is the the motor limitation of how quick your motor can move
step_sleep = 0.002
step_counter = 4096 # 4096 steps is 360°
step_counter = step_counter * 1000000 # number of turn (+/- infinite)
direction = False # True for clockwise, False for counter-clockwise
# stepper motor sequence (cf found in documentation http://www.4tronix.co.uk/arduino/Stepper-Motors.php)
step_sequence = [[1,0,0,1],
[1,0,0,0],
[1,1,0,0],
[0,1,0,0],
[0,1,1,0],
[0,0,1,0],
[0,0,1,1],
[0,0,0,1]]
# setting up
GPIO.setmode( GPIO.BCM )
GPIO.setup( in1, GPIO.OUT )
GPIO.setup( in2, GPIO.OUT )
GPIO.setup( in3, GPIO.OUT )
GPIO.setup( in4, GPIO.OUT )
# init
GPIO.output( in1, GPIO.LOW )
GPIO.output( in2, GPIO.LOW )
GPIO.output( in3, GPIO.LOW )
GPIO.output( in4, GPIO.LOW )
motor_pins = [in1,in2,in3,in4]
motor_step_counter = 0 ;
def cleanup_ini():
GPIO.output( in1, GPIO.LOW )
GPIO.output( in2, GPIO.LOW )
GPIO.output( in3, GPIO.LOW )
GPIO.output( in4, GPIO.LOW )
GPIO.cleanup()
# the meat
try:
i = 0
for i in range(step_counter):
for pin in range(0, len(motor_pins)):
GPIO.output( motor_pins[pin], step_sequence[motor_step_counter][pin] )
if direction==True:
motor_step_counter = (motor_step_counter - 1) % 8
elif direction==False:
motor_step_counter = (motor_step_counter + 1) % 8
else:
cleanup_ini()
exit( 1 )
time.sleep( step_sleep )
except KeyboardInterrupt:
cleanup_ini()
exit( 1 )
cleanup_ini()
exit( 0 )Save in the same directory as the models and the previous python script with the name ‘turn_motor_step.py’.
Launch the mechanism when the Raspberry Pi is switched on
If you want to run the code when you switch on the Raspberry Pi, you just have to modify the setting “ crontab -e ”:
You can use a console mode editor like nano. Simply add the following line with the @reboot option:
@reboot python3 /home/florianbergamasco_RaspPi10/Desktop/AICandies/main.py &[optional if stepper motor installed] : add this line below the previous one
@reboot python3 /home/florianbergamasco_RaspPi10/Desktop/AICandies/turn_motor_step.py &It is important to note 2 points:
1. Use absolute paths
2. Add the & at the end of the line to launch your command in a separate process. This avoids blocking the Raspberry Pi’s start-up if your software doesn’t give up quickly.
Conclusion
I hope this has been helpful and inspiring to use Computer Vision in your own projects. Of course, this is just a demonstrator, so I’ve also used a teaching support adapted to my audience, in order to explain and acculturate younger people.
Please don’t hesitate to contact me with any comments, questions or suggestions for improvement. Enjoy sorting your candies!
