Developing a slot car timekeeper app, using a cardboard, mobile phone and web technologies only
Setting
Santa Klaus brought my kids a nice long-wished for toy: A slot car Hot Wheels Carrera GO!!! track.
The kids had big time fun building different tracks. Took some time that they got the handle on racing, so the cars do not always fly out of track, but quickly they managed it and gave each other hot duels!
Eventually they came up with manual timekeeping.
The timekeeping worked fine, but there were some flaws:
- Only one kid at a time could race — the other one needed to take the time, and even if I was taking the time, I could keep track of one car only
- Manual timekeeping is never exact — especially when you don’t want your brother to beat your best lap time 😉
Time to start working backwards…
An idea was born
Technical challenges?! There is an engineer in the family for that!
Working as an engineering manager now, I have very little opportunity touching code — and if I do anyways, my engineers start giving me eyes 🙄
But hey, it’s xmas holidays, I have to match my quality goals only, and my main goal is having fun, programming something that gives my kids a good “time”.
Got my mobile phone and programming skills ready…
So, let’s get started!
Briefing
Even though my daily business is native mobile Apps with iOS and Android, I wanted to take the most bootstrapped approach with the lowest possible usage barrier, so I limited myself to:
- a mobile phone with rear camera
- cardboard/shoe box (really love the concept of Google Cardboard) used as a bridge over the track to hold the camera
- the app should run in the mobile browser only, and should need no installation
Benchmarking
Before totally wasting time, I was checking the market and found two types of solutions:
First — of course — the official hardware round counter (74,99€) solution by Carrera. I didn’t want to buy and waste more plastic, so I turned this option down.
Second a software called SmartRace by Trademarc for iOS and Android for 12,99€ in the app stores.
I gave SmartRace a shot — and was instantly disappointed figuring out that I not only had to pay the 12,99€ for the app, but additionally 2,99€ for a server add-on, that is mandatory to work with Carrera GO!!! of my kids.
On top of that, you need two hardware devices:
- One to run the main application and server
- Second one to run the screen
RLY?
Having an iPad and iPhone at hands I gave a shot anyways — but even me couldn’t make it run!
Well, some people were able starting it and 12 people gave an average of 4,7 stars rating, so maybe it’s just me…
Anyways I asked Apple for reimbursement and started to come up with something more bootstrapped.
Plan & Tech
I wanted to grab the video frames of the camera stream and analyze color changes in detection areas on each of the cars tracks. When the color or pattern changes, this means a car just drove by and time needs to be taken.
Sounds easy. Isn’t it?
Camera access
First things first: How do I get rear camera access with HTML5?
There is this JavaScript API to access the MediaDevices.getUserMedia(), which is compatible with most modern browsers:
(async () => {
let videoStream = await navigator.mediaDevices.getUserMedia({
audio: false,
video: {
facingMode: 'environment',
width: VIDEO_SIZE.width,
height: VIDEO_SIZE.height
},
frameRate: {
ideal: 60,
min: 30
}
});
})();One could even check about the different devices available (e.g. ‘environment’ and ‘front’ camera) via MediaDevices.enumerateDevices(), but to keep it lean, I didn’t went there.
So got the video stream, now what?!
Video Stream processing
There is the MediaStream Image Capture API, which would allow to grab image data from a stream, but it’s in experimental state…
Also I didn’t want to dive into video stream processing via MediaStreamTrack. Found a solution there as well, but this seems to be so unreliable, that after capturing three frames, it stops working.
So GIYF, the most common solution seems to be, to pass the video stream to an HTML video element, then draw the video to a HTML canvas element, and then grab the image data from the canvas element.
RLY?
Sounds like a heck of a workaround, but I decided to go for HTML5, so let’s get the show started.
Let’s talk about a big pitfall, though:
Meanwhile the desktop browser does not care, the mobile Safari requires the page that requests camera access MUST be delivered via SSL/https. Otherwise the video stream will not be available — and no warning will be given why the access was denied, or even that the access was denied.
That gave me some headaches, woMan! Hope you don’t get stuck at the same point!
Color/pattern analysis
For analyzing the pattern, I grabbed the image data of the detection rectangles and calculated block-wise the average color to have a default.
Fun fact: My buddy Emanuel, an experienced developer himself, asked me whether I used a neural net for car detection… 🤣
Thanks for that buddy! Well, not rly…
function getAverageRGBwithBlock(imageData, blockSize) {
const RGBA_ARRAY_LENGTH = 4;
let length = imageData.data.length;
let data = imageData.data;
var count = 0;
var rgb = [0, 0, 0];
for (var i = 0; i < length; i += blockSize * RGBA_ARRAY_LENGTH) {
count++;
rgb[0] += data[i];
rgb[1] += data[i + 1];
rgb[2] += data[i + 2];
}
rgb[0] = ~~(rgb[0] / count);
rgb[1] = ~~(rgb[1] / count);
rgb[2] = ~~(rgb[2] / count);
return rgb;
}The video is rendered with 30 fps (fun fact: before specifying a frameRate the phone rendered with 120 fps!) on the canvas, so it should catch a car passing by. If the car was visible, the average color should change, so constantly comparing to the default color should do the job.
function calculateColorDifference(colorDefault, colorCurrent) {
var diff = 0;
for (var i = 0; i < colorDefault.length; i++) {
diff += Math.abs(colorDefault[i] - colorCurrent[i]);
}
return diff;
}Timekeeper App and Screens
To satisfy the requirement that the page will be delivered via SSL, I made the project open source, and delivered the page via GitHub Pages.
The app itself is designed as single page application with three screens:
- start screen
- settings wizard
- timekeeper screen
Ok, Heiko, for a true SPA the client-side routing was missing, but I felt being generous to myself concerning this point 😉
Start screen
This screen offers a quick start option, or access to the settings wizard.
Quick start will start the camera, wait a bit, detecting the default pattern and then start the race.
Settings Wizard
In the settings wizard you can mainly configure the size and location of the detection rectangles and the sensibility of the pattern change detection.
Pressing Capture default will set the pattern currently visible in the detection rectangle as default pattern.
Timekeeper
Now it’s time to start the race.
You can see the total laps, lap times, best laps, lap average and so on…
Demonstration
After putting everything together, it was time for a real life test!
Fun fact: First time in my programmers’ life that the time I put it to production, it worked flawlessly!
My kids went beserk with an initial race with timekeeping of 298 rounds… so mission accomplished, I guess!
Please find some impressions of the final app here:
Summary
I had big time fun touching code again — and double big time, because from idea, to plan, to implementation, to first real world test everything worked out like a charm!
So happy that everything worked out, but the required HTML5 workarounds still dazzle me.
Of course there where some caveats and pitfalls to overcome (see appendix), and everything took a bit longer than expected (I would have totally sucked at planning poker…), but it was really rewarding, that everything worked out at the end AND my kids had fun with it!
App and Code
If you want to give a shot for yourself:
- Slotcar Timekeeper App on GitHub Pages to be directly used on your phone
- Slotcar Timekeeper Source Code on GitHub
Have fun, and let me know what you think about it!
Happy new year 2023!
Appendix: Hints and caveats
Before beeing able to come up with a solution, I stumbled upon many field stones, that I wanted to let you know of.
On mobile Safari you will get NO camera access at all (and no warning either), if you don’t deliver your page via SSL/https!
To make the video work on iPhone Safari, I had to add following magic boolean attributes:
<video id="video" autoplay playsinline>Via videoElement.requestVideoFrameCallback(function) you may specify a function that is called any time a video frame was ready.
Trying to hide the video via display=’none’ or visibility=’hidden’ on mobile Safari only (it will work on desktop though…) will result in the video frame buffer will not be rendered at all — for optimization reasons — which will result in a “stuck” video frame painted on the canvas.
I ended up hiding the video element behind the canvas element with position:absolute and z-index: -1.
The UserMedia.getUserMedia() lets you specify arbitrary size values.
The browser will try to honor this, but may return other resolutions if an exact match is not available, or the user overrides it.
You might even be able to enable the “torch” of the device via the MediaStreamTrack.applyConstraints(), but it didn’t work for me.
videoStream.getMediaTracks()[0].applyConstraints({
advanced: [{torch: true}]
})The video element must use absolute pixel values for size, and does not accept percentages.
Google Chrome gave me the advice to set the contextAttribute willReadFrequently, when getting the 2d context... and reading frequently… and I obeyed.
A boolean value that indicates whether or not a lot of read-back operations are planned. This will force the use of a software (instead of hardware accelerated) 2D canvas and can save memory when calling getImageData() frequently.
canvasElement.getContext('2d', { 'willReadFrequently': true });The pure non-video-element solution taken from webrtchacks.com, that you might want to give a try:
stream = await navigator.mediaDevices.getUserMedia({video: true});
const [track] = stream.getVideoTracks();
const processor = new MediaStreamTrackProcessor(track);
const reader = await processor.readable.getReader();
async function readFrame() {
// Reading more than 3 frames per reader.read() freezes.
const {value: frame, done} = await reader.read();
// value is the frame object
if (frame) {
const bitmap = await createImageBitmap(frame);
console.log(bitmap);
storage.push(bitmap);
imageCountSpan.innerText++;
frame.close();
}
if (done)
clearInterval(captureInterval);
}
