Adapted from Pexels

Bionic Data Science: Machine Learning and Human Learning

As Robert Greene writes in Mastery: to reach true mastery you should resist your own impatience.

If you are reading about our project for the first time; we are on an adventure in Bionic Hands, Machine Learning and Social Entrepreneurship! Have a look to see how we got here:

• Start of the project
• Project after 1 month
• Project after 2 months
• We’re now in the 3rd month!

Now back to our impatience. So far, we’ve been heavily involved in finding partners, learning from experts and getting feedback on our own ideas for the project direction. Despite great results so far, we thought it was time to pause this stakeholder/customer interaction and focus on the technology of biometric control algorithms. This way we’re giving in to our impatience a bit, since as engineers we love to focus on the technology rather than on the process (too much blah blah blah).

As we do not have a fully functioning biometric device yet, we requested raw data samples from a startup that builds myoelectric sensor armbands, OYMotion, to get started with our data analysis and convince ourselves and partners this is good technology to begin iterating with.

Instead of rushing to a final result with half-baked scripts (my preferred strategy), I’ve spent the last month slowly writing a Python package with our data processing, visualizations and learnings so far, while teaching fellow project members Python and machine learning concepts.

My first workflow is a bit as follows:

• I start with a Jupyter notebook, to get a feeling for the data and required code.
• Then I move to Pycharm and create more decent object oriented code. (Hmm… should write tests first, TDD)
• Commit that to github and add a new distribution to PyPi.
• Inform the team, and get them and myself using the packaged Python code in a notebook again.
• Iterate

Hopefully other members can move into a similar cycle and now contribute to the code base. I am not always very good at delegating… more to learn here.

Developments so far

Created a Python package. If others want to use and contribute to our source code, it should be easy. So obviously we had to create a package. The source code is stored on github. You can install it via pip and then just import it into Python. Maybe it’s not impressive yet, but I’m very proud of this, since it’s my first Python package ever:

import symbionic

Loading raw sensor data. OYMotion stores their raw signals in binary format. I converted these into Python Pandas dataframes for convenience of data visualization and analysis. The training data contains roughly 30 seconds of EMG signals per sensor, for each loaded gesture. For now you can only load 1 file at once. If you’re eager to do this, please wait a bit till we collect our own data that we’re allowed to share:

emg_data = symbionic.EmgData()
emg_data.load(’filename.hex’, gesture=’g1’)
emg_data.load(’filename2.hex’, gesture=’g2’)

Visualizing the data. Before doing any machine learning it’s important to understand your data. The best way is to use your very well performing human visual pattern recognition system. Billions of years of development went into this (using evolutionary algorithms):

emg_data.plot()

Raw data samples for all the gestures (shown with images above), for the 8 channels/sensors of the OYMotion armband. The different gestures show clearly distinct behavior that could be used to classify them (already by your human brain).

Labeling the data. We’ll be opting for supervised machine learning, in other words algorithms that learn from data, in which they are told what to look for. Therefore, the above data needs to be labeled, so we know where the gestures are, and where there is only noise. Below you see an attempt to do this automatically on a gesture, by calculating the overarching pattern (envelope function) and taking a threshold on the sum of these patterns across channels. Probably someone can come up with a better approach, which would certainly help if we move to amputees with weak muscle signals.

emg_data.label_patterns()
emg_data.plot(show_envelope=True)

Labeling the gestures in the raw data, using the envelope functions (blue lines). The labeled data is indicated in the light blue shaded area. I do not guarantee that it’s done perfectly.

Machine learning

I am currently busy with writing machine learning code in a Jupyter notebook, together with insightful visualizations for the results. First results are promising, now we need to consider how to add this to the code base. I won’t go into all the details of the algoritms yet, but I can share some first results.

To start, it’s important to understand what an algorithm is doing, so you can improve it further. A great thing to use for analysing the predictions is a so called confusion matrix. A wonderful name for something that tells you how confused the algorithm is. Ideally, each gesture is classified correctly (no confusion), but often the algorithm still makes mistakes. In our case, the worst part is when it thinks there are signals when there was actually noise (hence the bionic hand would open or close unexpectedly). However, I should also still check if it is not actually correct, as perhaps I labeled the data wrong.

First classification results for a Random Forest algorithm on EMG data samples. The total accuracy was 68%. You can see that gestures are often confused with noise, but not often confused as other gestures.

You can see in the confusion matrix that the current algorithm often classifies gestures as noise. But this is also because you need enough signal before you can make a proper prediction. My hypothesis was that the classification should become more accurate as the algorithm sees more signal from a gesture. For that I made the graph below, which confirmed my hypothesis. From experts in the field we heard that ideally you should classify a gesture after 200 milliseconds (0.2 seconds), so the user doesn’t experience a huge time delay. This seems feasible to me from the result.

Classification results in time. At time = 0, the gesture really starts, but there is not much signal available yet, so it takes a while before any algorithm can detect it.

We can now start to think about how to add such models and insights to our Python package.

As soon as we’ve collected real sensor data, we can open source that and external data scientists may start helping out if they’re up for it (please?).

Ongoing human learning

We have learned and are in the process of learning so much already. I am just going to summarize the main things here:

• Learn what bionic hand users really need
• Learn what EMG signals look like
• Learn how to process and visualize them in Python
• Learn how to create an open source Python package
• Learn how to do multivariate, multinomial time series classification (predicting multiple gestures from EMG signals from many sensors)
• Learn how to convince our managers to fund our project further
• Learn how to inspire others to join our cause
• Learn how to keep a team going
• Learn how to teach coding and machine learning
• Learn how to visualize real time data streams in Python
• Learn how to use an Arduino
• Learn how to collaborate globally with other projects
• Learning about online collaboration tools (Trello, Slack, Google Docs, etc.)

The plan now

We have a go from our managers to buy a myoelectric armband. We did get the feedback to write down our desired technical architecture and a more detailed project time line. Personally, I do not believe in detailed planning, we’re going about this pretty Agile, but making our vision and architecture explicit will certainly not hurt.

After acquiring an armband, the first step is collecting data with it, so we can learn from the signals in real time and allow outsiders to contribute to our software and algorithms. I’d like to start as follows:

• Collect data from ourselves
• Collect data from some volunteering amputees
• Collect data in ever harsher circumstances (such as sweating)

With such data available to everyone, we can start creating a simple open source human and machine learning environment. Ideally, this should be easy to use by non-experts.

After that we should focus on gesture robustness, not just accurate and fast classification, but also add some hysteresis/penalty to switching between gestures. So that when we’re certain about a gesture we make it harder to switch to another, thereby not accidently dropping grabbed objects for example. It’s worse to perform a gesture when it’s not desired than not performing a gesture when desired. In the first case you can drop and break things, in the latter you just have to try again. This may also require a different metric for the optimization.

At the same time we’ll have to move to running the algoritms on streaming data from the armband somehow, so we can actually classify gestures in real time. I could really use some senior (Python) software developers right about now.

Ideally, we also want the software to be able to interface with any device. Be it an OYMotion or Myo or even an EEG sensor for reading in signals. And at the same time be able to actuate any motor system in the bionic arm. This would require a complex modular architecture, the exact outline of which I cannot yet envision right now.

In the mean time I still want to find time to document and share our knowledge. I am writing these blog posts during precious time in my evening hours, where I could be writing code. This is a huge part of trying to be patient. Next to that, we also did a first presentation of our ideas and results at our company event. So far the feedback is very positive!

In the end, to stay patient, we should remember all of the above are just tools. Machine learning, software or management skills, those are just tools to serve a purpose. Perhaps the goal is to create a bionic hand, but even that is just a tool. For me it’s the continuous proces of learning, creating and sharing that counts, and enabling others to do the same. Bionic hands can help others in doing just that.

If you want to follow our progress, feel free to follow me here on Medium, on Github or on Linkedin. If you want to help out, need help yourself, or have any good ideas to share; please reach out to me!