The SoundDetective
A novel approach to predictive maintenance based on sound analysis
Stefan Muderack, Chief of the SoundDetective explain, how they developed a device that could save companies tremendous amounts of money caused by production machine damages even before they occur — on the example of a simple coffee machine.
All mechanical devices are subject to wear and tear and then: failure. Outages and defects of such devices can cause significant delays and therefore costs if they happen during production in a Porsche factory. The promise of predictive maintenance is that the control room of the production plant is able to detect and react to defects before they occur. Predictive maintenance has become a cornerstone in the smart production of most manufacturing companies. Production machines, engines, robots and other utilities like baskets or mechanical lifts are equipped with sensors by their original equipment manufacturer to enable the collection, storage and analysis of data to foresee defects and to notify the customer of the machine, e.g., Porsche of a pending outage.
Predictive maintenance relies on large amounts of data which is created in a vendor specific format within those machines and relies on the manufacturer of the machine to supply software or algorithms to interpret it — often you’ll get a notification if sensor information exceeds a certain threshold. Predictive maintenance is also limited by technical and legal restrictions.
Technical restrictions are for instance that the networks in production environments are usually resource limited and optimized for high Input/Output — that means you cannot transfer large amounts of data over the production network.
Legal restrictions are for instance that the manufacturer does not grant full access to the generated data and also prohibits that the machine can be opened to access more sensor data — or equip it with more sensors.
Everything that moves creates a certain sound
Every machine with moving parts has an acoustic fingerprint which is as unique to other machines as the human fingerprint.
One of our projects in the Porsche Digital Lab delivered a viable alternative to the current state of the art in predictive maintenance, assistance of the production workers and even autonomous driving.
Every system which contains moving parts creates a certain sound and enables a differentiation and identification of production states (moving, stationary or standby). In the automotive production domain those moving systems comprise — but are not limited to robots, hydraulic lifting platforms and also smaller systems like engines and cordless screwdrivers. Every machine features its own unique sound profile which changes when it’s running, idling, in standby or in case that it needs service. Even identical machines sound slightly different — you can distinguish Machine A and Machine B just by listening to the machines. Every machine with moving parts has an acoustic fingerprint which is as unique to other machines as the human fingerprint.
Skilled technicians recognize just by listening to the sound of the machine if it works smoothly or if it’ll need service soon — sometimes they can single out screws which are loosening themselves. Think about the last time you drove with your car and it created odd sounds when you were turning the wheel — you instantly knew something is off and could probably also distinguish where the sound was coming from. Maybe you even singled out the fan belt as the creator of those noises.
A combination of methods from the Internet of Things and artificial intelligence enables computers to do just that. We took a Raspberry PI and outfitted it with a standard USB microphone which we bought online. This hardware stack was complemented with our software stack based on Python, Tensorflow, Keras, MQTT and several other open source libraries. This hardware- and software-stack enabled us to develop an artificial intelligence which was able to listen to the production machine which was available and accessible directly in the Porsche Digital Lab — our coffee machine. Our SoundDetective was born and eager to investigate the sounds of production machines. You could argue that a coffee machine is a hardly a production machine but think about it: it consumes input (coffee beans, water, milk), does processing steps on those inputs (grinding, brewing) and finally creates an output based on your choice and inputs.
Sound-based analysis of every system with moving parts
We have attached the microphone to our coffee machine in a non-intrusive way and thus kept the structural integrity of the coffee machine, i.e., the machine was not altered or damaged in any way. One challenge was that the microphone recorded so many ambient sounds like the dishwasher, the cooling cycles of the fridge and of course chatting colleague in close proximity of the coffee machine. Finally we found a good position inside the coffee machine which shields the microphone from external influences and internal effects of the coffee machine like high humidity and temperature changes during the brewing of the coffee.
We are able to correctly distinguish between various hot beverages just by the sound of the coffee machine.
The Raspberry PI records the audio stream from the microphone and splits it into one second slices. These samples are fragmented into more than hundred features like a spectrogram, tempogram or Tonnetz. These features are then in turn fed into our deep neural network which decides if the sound is a coffee, the cleaning program of the coffee machine (which sounds surprisingly similar to tea) or maybe even a real tea. We are able to correctly distinguish between various hot beverages just by the sound of the coffee machine. The analysis and execution of the neural network is done exclusively on the Raspberry PI — only one information is transferred via the available network connection: coffee +1. This edge computing feat is enabled through the huge advancements in the hardware of the Internet of Things — a Raspberry PI and smartphones for that matter became CPU powerhouses and the powerful open source libraries like Tensorflow.
Benefits
Our coffee machine case is not a classic predictive maintenance case but rather a product classification. The results of this experiment however can be ported and applied to a wide variety of use cases. We have used the sound data of known production sounds (hot beverages) to do an on-device realtime-differentiation between those. One possible alternative would’ve been to identify error-cases such as low water status, low bean status or other possible errors to realize a predictive maintenance case but we’ve opted to record the amount of drunken coffees in the Porsche Lab first and correlate them for instance with events.
The link between a coffee machine and the production
One use case which we’re working on together with the StartUp iNDTact is the usage of a vibration sensor embedded into a glove or on a thimble to detect if connections are mode successfully during the montage of a Porsche. Vibrations are sounds which we can’t hear but the SoundDetective can observe and analyze those sounds just like the coffee machine. According to the poka yoke principles and the underlining kaizen principles, every step in the production should be failsafe and only done once. This is why the connectors at Porsche make a loud and clear “click” sound to notify the worker of successful connections. However, once in a while, a connection has to be made which the worker can’t see because it’s for example behind the glovebox or in the drivetrain and he can’t hear the connection because a robot or a mechanical press is working close to him. In this case we can assist the worker with the SoundDetective. We’re measuring the connection process through the vibration sensor and then we’re providing the worker with a fast, easy and failsafe feedback through a device which we’re going to define together with the workers during a Design Thinking sprint.
We’re living in exciting times with great opportunities and everyday we’re trying to push the boundaries in the Porsche Digital Lab a bit farther.
Team: Stefan Muderack, Matthias Auf der Mauer, Karthick Perumal, Christopher Hansen, Dr. Tristan Behrens, Dr. Mahdi Derakhshanmanesh, Claudio Weck, Judith Gabbert, Alice Chan, Roman Siejek
Acknowledgements: The case is a result of research conducted by the authors in the scope of the Porsche Digital Lab Berlin. This lab is operated by MHP — A Porsche Company.
