Building your own interactive touchscreen table with object recognition

Published in

The Startup

7 min readJun 11, 2019

Recently there’s been a kind of touchscreen table application getting more and more popular at events and showrooms. People interact with an application by putting objects directly onto the screen. The touchscreen will then immediately recognize the objects and react by showing information related to the objects.

Example of an interactive touchscreen table application with object recognition

The objects can also be moved around or act as dials to interact with the touchscreen user interface. This allows for some interesting and creative UI concepts which integrate real-world objects as part of the user interaction.

It’s pretty cool stuff — So let’s have a look at how this is actually done (and build one ourselves!)

Touchscreen table object recognition methods — A comparison

There are multiple ways to achieve this depending on your budget and requirements:

Cameras combined with real-time image recognition software will give you great flexibility in your choice of objects and eliminate the need for mounting the objects on a base disc but it is also (much) more expensive to setup and maintain than touch-only based solutions. Use this solution if you need large numbers of objects to be distinguished and you have the 💰💰💰 to spare.

Capacitive multi-touch screen panel with base discs are a great (and budget-friendly) solution if you are looking for only having a limited number of objects on the table at the same time and do not require many different objects to be distinguished. In this case you could go for a touch-only solution as shown in the video below:

Touch-only solution with objects mounted on a base disc

The touch-only solution you can build by yourself completely from off-the shelf parts combined with a bit of math and Maker-fu.

There are some limitations though:

Your objects need to be mounted on a base disc (see the video above)
Depending on the multi-touch capabilities of your screen and operating system you cannot have more than three objects on-screen simultaneously
You can only distinguish about 10 different objects
There will be a lag of about ½ second when moving and turning objects

Most touchscreen table UI concepts will work perfectly fine within the limitations above, still you need to make sure that none of the above is a showstopper for you. Simply because there aren’t really any workarounds available.

“A bit of math and Maker-fu”

As mentioned above your objects will need to be mounted on a base disc in order for the screen to recognize and distinguish the object. How does this actually work? The answer is not 42 but “Triangles”.

The base disc is merely emulating three finger presses on your touch screen making up a triangle. We won’t use any triangle but we will use isosceles triangles with distinguishable vertex angles. These angles can then be mapped to different objects. So you can later say “A 20 degree vertex angle maps to object A while a 36 degree vertex angle maps to object B”.

This approach also explains the limitations described above. Since most multi-touch screens only support up to 10 fingers at the same time we can only use three objects simultaneously as they will already take up 3x3 of the available touch points. Also as the theoretically maximum vertex angle must be less than 180 degrees, there are only about 10–12 distinguishable angles you can use.

Note: You might be able to squeeze a bit more angles out of this setup but from our experience results are increasingly unstable if you go beyond 12 different angles. If you really need more than 12 objects try adding the triangle height as an additional dimension for your object recognition but YMMV.

In order to make this work we need to create a small piece of software with the following core functions:

Accept a set of coordinates (our touch points)
Identify all isosceles triangles within the set of coordinates and calculate their vertex angles, rotation and height
Limit the calculation to only group coordinates within reasonable distances as triangles to avoid overlapping or bogus shapes

To make your life easier I have already implemented a JavaScript based library which you can simply drop into your own project and get started without dealing with any of the math. You can download the library from my GitHub repository. The repository also includes an HTML5 demo application which will let you test your discs.

Let’s start making!

Now with the math stuff out of the way we can directly jump to the fun part — Building your own object-recognition touch table!

What you will need:

Hardware:

A capacitive multi-touch capable LED screen (42’’ or above recommended)
A halfway decent PC running Windows 10 and Chrome (A Raspberry Pi won’t be able to handle this)

For the DIY discs:

Casting the discs

Use a measure cup to mix both resin components (epoxy + hardener) according to the required proportions (usually 2:1 or 3:1 — check the instructions)
Pour the mixture into your prepared silicone mold and wait for the resin to set. Depending on the kind resin this will take between 12 and 36 hours.
Be sure not to shake or move the mold during this time as you want the surface of your discs to become perfectly flat.
Wear rubber gloves while you are working. Resin based epoxies are REALLY hard to get off your skin.

Once the resin has set take out your discs and ensure the lower surface has come out flat. If it’s flat you can move on to the next part.

Note: In my pictures you can see both “solid” and “donut” style discs. Both will work fine but after a couple of projects I prefer the solid ones. The donut shaped discs (while lighter) tend to bend or shatter if they are dropped on the floor.

Adding the conductive spots

The next step is to make your disc electrically conductive so it can actually emulate three finger presses. We will use the copper tape for this:

Wrap one strip of copper tape all around the outside of the disc, but DO NOT CLOSE the “loop”. Means you should leave a little gap of about ½ inch between both ends (see picture).
Starting from this long strip of copper tape stick three more short strips of copper tape leading to the bottom of your disc at the desired angle. I usually cut the copper strips lengthwise in half to make the touch points smaller (and therefore more precise).
Finally prepare a round piece of self-adhesive paper with three holes punched into it at the desired angle (I used a stapler to punch the holes). Stick the paper to the bottom of your disc leaving only three dots of copper tape exposed. The paper will additionally serve as a protection layer, preventing your disc from scratching the screen. Other materials (like felt) will work great, too.

Adding the conductive spots to your disc. Example pictures show 30 and 54 degree angles

Note: There are other ways to create the conductive touch points. If you require a bit more precision you could for example use conductive small foam pads on top of the copper tape.

I suggest using angles of 18, 36, 54, 72, 90, 108 and 126 degrees for your first test. Once you got the hang of it you can reduce the distance between angles to 12 degrees. Just don’t use 60 degrees as this will create an equilateral triangle and you can’t track its rotation.

Testing your discs

Once you have finished your discs you can test them using the example HTML5 page from my GitHub repository.

Happy making and I’d love to see your creations!