How to train an invisible limb?
A retrospective on the Symbionic Project
We’ve been going for almost 10 months now, working part time on this awesome bionic arm project. However, by the end of the year we will run out of our current budget. We are re-orienting on how to continue, and in the process it seems like a good idea to document and reflect upon everything we accomplished.
The journey so far
We’ve been continuously on the lookout for finding valuable insights and adjusting our our course accordingly, in order to end up with a result that will really help our goal.
The general vision is to help amputees with bionic technology. With the vision in mind the strategy and product is allowed to change (when needed) throughout any startup.
With our last strategy, amputees can get ready for bionic limbs by keeping their muscles in shape, even if they are not yet available to them yet. Next to that training software could help reduce phantom pain according to our amputee contact Arie Rommers. In general, the software should be fun to use, so people continue to use it.
The benefit with this lasest idea is that our single product could be valuable and sustainable all on it’s own, instead of depending on the existence of a modular ecosystem for bionic limbs.
Another great idea on this part comes from the Limbitless project where they enable disabled children to play video games with myo-electric sensors. This a very fun way to train their muscles!
The future is not that clear from here on out. Our project runs out of budget and we’re not great software developers who build this in a whiffy.
System architecture
Overall, I really discovered the power of simple and clear figures in our internal and external communication, both within the Symbionic project and in my general work. These figures help explain the vision of the project, as well as the ‘system architecture’ and related work breakdown from that.
The most used figure in the project in the last months was my functional flow through the software ecosystem. Though it is already a bit outdated by now, it helps to explain what we were working on:
I subdivided into two streams: the training phase (upper stream) on historical data and the prediction phase (lower stream) on real-time data.
The upper stream uses the data received from the myoelectric armband and trains a model on it. Many steps were needed, including a lot of visualization for our learning.
In the lower stream we struggled for quite a while, especially building a Real-Time Interface from the OYMotion data on the USB-port and loading this directly into our Python environment. We finally solved it by admitting defeat and finding a volunteer C/C++ developer to help us out build a ‘socket’ around the OYMotion code.
Algorithm results
We have an intern, Paul, working full-time on algorithm development right now. Here’s a short summary of his results. Note, that we’re looking for the right balance between:
- Ease of implementation (in our the final software)
- Prediction time (fast enough to not cause noticable delay)
- Prediction accuracy (of the 6 classified gestures)
For finding the best algorithm for gesture prediction, it is necessary to test out different algorithms. This is due to the ‘No Free Lunch’ concept in machine-learning.
The following algorithms are used for testing: Xgboost(Xgboost library), Logistic Regression(LR), Linear Discriminant Analysis (LDA), KNeighbors Classifier (CART), GaussianNB(NB), Support Vector Machine (SVM), Gradient Boosting Classifier (GXboost from sklearn), Random Forest Classifier(RF), Extra Trees Classifier (Extra Trees).
The left figure displays the prediction accuracy of each algorithm on the raw data from OYmotion armband. As can be noticed the Xgboost algorithm has the highest accuracy but the time duration of the training is over 240 seconds on Paul’s laptop.
The right figure displays the accuracy of each algorithm on preprocessed data. Note that other algorithms such as ExtraTrees now have the same or higher accuracy as Xgboost but the time duration of the preprocessing+training is less than a second.
OYMotion review
Halfway through the project we decided to buy the OYMotion armband with 8 integrated myoe-electric sensors. With the support of OYMotion and their software, the armband was relatively easy to set up. The OYMotion OTrain software stores the data during the training session as binary files in a folder, so acquiring data and running our own algorthms was relatively straightforward as well.
The difficult part was connecting the real-time raw data to our own Python software, though it doesn’t help that we have little software development skills in our project. We finally managed with the help of an experienced embedded software developer. Now we can visualize the armband data in our own environment, and run algorithm predictions on real time data.
We considered the Myo armband from Thalmic labs as alternative, but we received a lot of feedback that the bluetooth connection for raw data acquisition was not stable. And the OYMotion armband had better battery life, higher bandwidth, their own training software, and much more. Next to that, Thalmic Labs announced they stop the sales and support for the Myo! Good news for OYMotion I guess, since they are now alone in the affordable myo-electric armband market. Or it means this kind of business is just not viable.
Lessons Learned
What did we learn and what advice can we provide any other team out there?
The lessons learned and the advice we can give can be divided into these main categories:
Technical skills
We have learned a lot regarding technical skills; topics such as programming in python/c++, EMG sensors, signal processing, data analysis, machine learning and ofcourse bionics in general. All of our team-members where in some way involved in the technical work; ranging from actual programming new algorithms to understanding what others have made.
Soft skills
Even more so, we have learned that starting up an innovative project is not only about having a good idea and working on it. We have learned that the human-factor in technology cannot be neglected. In bionics; we have realized ourselves that the bionic hand should be designed to be accepted by the user rather than having the best performance on paper. From an internal point of view this project has taught us to work together in a multi-disciplinary team. It is very important to understand what everybody is doing and how we should interact with eachother.
Motivation
Just having an interesting topic and a project-team is not enough to keep people motivated. To ensure motivation, it is important to base the roles of team-members on their personal preference. Some people really excel when they are thrown into the deep on a topic that they’re not familiar with, while others prefer small incremental steps of learning new skills. There should be a good personal balance between doing what you’re used to do and doing what you’ve never done before.
Organisation
A project cannot be successful without a proper organization. Regarding worksessions and meetings, we have found that working together in a team should be done on a regular basis to keep momentum within the team. When working isolated, it is very easy to either lose motivation or drift away from what’s important within the project. Besides alignment, regular meetings are also used to define small and clear tasks with attached deliverables. This gives focus and efficiency within the team. Finally, we have also found that a project is unthinkable without a “champion”; someone that brings motivation and a vision into the project.
Communication and network
Building up a network is one of the first things that should be done within the project. When talking to potential end-users or people with experience with the topic, it becomes clear what is required in the project. After building up a network and communicate with them, an end-goal can be defined. This end-goal is the starting point of the project. The planning and activities should be defined with the end-goal in mind. It is also very important to review this end-goal regularly. We discovered that adjusting the end-goal when it doesn’t seem to be feasible anymore is key to keep motivation up and be able to keep up the deliverables. During this process, it is very important to keep in touch with the network; we found ourselves in the situation every now and then where we would fill in the questions for the customers instead of actually talking to them. Besides the communication with the customer, we’ve also learned that communication to people who are interested in the topic is also very important. This way, we have gained (unexpected) help for our problems.
Outlook
Looking to the future, it is not yet clear how we will continue with this project. At first we have to redefine the end-goal of our project. Recent meetings gave us some new insights how we could use our developments of the last year. We came up with four possible use-cases:
- Control a bionic hand:
This is where it all started with, develop an algorithm to give back amputees more functionality. We should think of the priority of functionality with respect to esthetics - Gamify the learning procedure:
It is generally known, someone learns quicker when it is in some sort of game. Our algorithm can also be applied in a gamified learning experience - Train Muscles
When an amputee doesn’t use their muscles often, they will lose strength. However, stronger muscles give better results on EMG devices, therefore amputees will benefit from a manner to keep muscles into condition. This an easily be achieved by using simple procedures together with the already developed algorithm/GUI. - Reduce phantom pain
Phantom pain is a major problem after a limb amputation, using a realistic model which interacts on muscle movements will most probably reduce the phantom pain. The mirrorbox therapy, a more simpler form where the non-amputated limb is in front of a mirror to trick the brains, is already proven to decrease phantom pain.
To get the most out of the continuation of the project we should first focus on finding a real end-customer. Someone wo can give us feedback on our progress and can keep us motivated and sharp. Together we can define sub-deliverables and a rough planning. We’re curious and looking forward to the next year.
