How Your Ambient Noise Solves Proximity Pairing
Discovering the Perfect Solution
Discovery
It began with a problem that we stumbled upon while developing a social networking application.
The idea was to enable nearby users to quickly and seamlessly create group chats, or send a friend request to someone they just met. We tried solving this issue with the help of existing solutions such as: geolocation, Bluetooth, Wi-Fi, and ultrasound technologies, but each of these methods exhibited critical deficiencies that hindered our vision and the user experience.
Eventually, we devised and patented a brand-new solution based on matching the samples of ambient noise. In simple terms, we discovered that if two or more devices are hearing the same thing in real time, they are in the same place. We called this method Resonance Nearby. Resonance became the reason for our major pivot as a company.
Interaction between nearby devices
People do a vast amount of things in close proximity. They exchange files, use social networks to befriend a new acquaintance, join online games, transfer money, etc. Such interactions would be much more efficient and prevalent if it was technologically possible to seamlessly connect devices without worrying about overhead.
For example, John just got acquainted with Mary, and they are trying to ‘Friend’ each other on Facebook. After failing to find each other in the long list of namesakes, they will most likely just exchange phone numbers. This is a huge waste of time. No solution exists that can expedite this experience as of now and work universally.
A potential solution must:
- Work wherever and whenever. On the street, while in transit, in the office, etc.
- Be cross-platform. It should support iOS and Android at least, and ideally function in the browser Web App too.
- Accurately find devices that are actually nearby.
- Be fast. Detection time should be less than 10 seconds.
- Be simple, non-invasive and intuitive.
The Resonance Nearby Technology
We live surrounded by ambient noise. Voices, music, engines, footsteps, etc.
A short sample of natural ambient noise — along with the exact time of its recording — is completely unique.
There is not a single place in the world that has the same ambient noise that you have right now, reading this article.
Using the parity of ambient noise and time we can conclude pretty surely that the devices are nearby. This caveat became the basis of our technological solution.
Each device captures sound from the microphone in real time, and converts it into a special fingerprint using a perceptual hash function. The peculiarity of perceptual hash functions is that tiny differences in the source data are expressed by small differences in the resulting hash.
An audio fingerprint with an accurate timestamp is then sent to the server, where it is matched with the fingerprints from other devices captured at the same point in time and used to determine the similarity of the original sounds. If the similarity indicator is above a certain threshold, the devices exchange identifiers for further interaction.
It is important to note that the actual audio sample never leaves the device, thereby negating any potential risks of privacy infringement.
It was crucial for us to make sure that this concept worked, and the technology was able to find parity in the audio samples recorded by devices located several feet apart. For testing purposes, we manually collected hundreds of hours of noise recorded simultaneously on several devices, in a variety of places including: subways, office buildings, coffee shops, concerts, sports stadiums, libraries, freeways etc.
Using this data, we looked through various options of acoustic fingerprint generation and comparison algorithms to achieve the best possible outcome. Ultimately, we achieved 96% accuracy through a 6-second fingerprint, detecting all devices within a 15 foot range. The technology yielded a false positive result in only 0.0039% of all test cases.
The obvious drawback of Resonance Nearby is that it didn’t not work in absolute silence. The algorithm does not distinguish silence, so it deliberately ignores it to avoid false positives. However, absolute silence is hard to come by in real life. A mere keystroke or a subtle tap, or clearing of a throat can be enough for devices to discover each other.
This quirk can make the device discovery process rather entertaining. After launching a service that uses Resonance Nearby consciously, users tend to await device detection in complete silence. After a few seconds, someone may lose patience and exclaim how Resonance Nearby “doesn’t work!” The phrase works like a charm. The devices will immediately discover each other.
One of the major advantages of Resonance Nearby is its cross-platform capability. We recently developed a JS-version of the library, which works in Chrome, Safari, Firefox, and Edge, including mobile versions of these browsers.
We have developed libraries for iOS and Android that allow the implementation of Resonance Nearby through a simple API, and also built them into our own mobile apps.
Other solutions….
We tried various existing technologies for device detection before developing the Resonance Nearby solution, but encountered some critical limitations.
Let’s examine some other solutions in detail.
Geolocation
This is the most obvious way to meet the challenge of detecting nearby devices. The moment a user activates the GPS-supported “Nearby” function, we can obtain his current location and use server data to search for the nearest devices.
Let’s depict the location as the center of a circle, while its radius will represent the accuracy measurement, presented by the image below:
If the distance between the devices (d) is less than the sum of the accuracies (r1 + r2), then there is a probability (P) that users are in close proximity.
The distance of a user search (radius) cannot be less than the inaccuracy of the coordinates, otherwise we will definitely fail to locate someone. However, the actual location coordinates can be outside of this radius — i.e. Android allows 32% probability for errors. Hence, even for users staying in proximity, there is a high chance of failure.
The coordinates obtained with GPS can be precise, but the technology may work poorly indoors. It may take up to a minute to search for the satellite signal, for instance. Geo-location alone fails the ubiquity requirement as not all devices may be equipped with a GPS module (cheers! to iPad WiFi), or it may be disabled on a system level (cheers! to Android).
In fact, GPS may be erroneous, even outdoors, especially in densely built-up areas, because of the reflection of the signal from buildings. It may provide >300 feet accuracy:
This is why, in most cases, the coordinates are derived from surrounding Wi-Fi networks and cellular towers to triangulate the signal. This method is fast and energy efficient, but the accuracy is lower by an order of magnitude of between 100–1000 meters. In practice, built-in receivers often determine the wrong location, and can sometimes “teleport” the user to another city!
When we tested the suitability of geolocation for our use cases, the devices could not find each other in about 15% of the tests because of incorrect coordinates. Often, the errors occurred inside high-rise downtown areas and in the subway network. Also, we faced irrelevant detection of user devices that were not actually nearby, due to the inaccuracy of this method.
Pros and cons / outcomes:
+ easy to implement
– low accuracy
– poor performance for detecting objects in motion
Bump
An original approach to enhancing the precision of geolocation was proposed by a Californian startup, Bump Technologies, some eight years ago. Users were required to physically bump their smartphones together so the accelerometer captured the exact time of contact and sent it to the server, along with the location data. The algorithm would only pair devices with identical time of contact. This simple idea significantly reduced the probability of false positive results, and helped to significantly increase the search radius. Details of Bump’s functionality are described in this patent
Bump was acquired by Google in 2013, and in 2014 the project was shut down, despite the fact that the Bump SDK had already been built into many third-party applications, and that Bump’s own mobile app for file sharing had received hundreds of millions of downloads. The fate of the technology is unknown.
The main constraint of the Bump technology was that only a pair of devices could be coupled through one bump, whereas in order to connect a group of users, multiple touches were necessary.
+ high accuracy
– one-to-one device detection
– need to collide devices with each other
– the project is closed
Bluetooth and Wi-Fi
Due to restrictions imposed by Apple iOS and Android, devices cannot natively connect via Bluetooth. Only a certified Made For iPhone (MFi) Bluetooth module can be utilized by the apps. Exchanging data between these platforms is a challenging exercise that requires third-party solutions and work-arounds.
In order for devices to detect each other, the following method is used:
· iOS emulates any Bluetooth Low Energy peripherals by setting its token as the BLE device name.
· Android temporarily changes the Bluetooth name of the smartphone to its token and turns on the detection mode.
· In order to browse for nearby peripherals, Android initiates Bluetooth service discovery for Android device detection and BLE for iOS device detection. iOS scans only BLE for iOS device detection, because Bluetooth devices cannot be identified using a public API. In order to discover the Android devices, iOS needs to receive the identifiers of the surrounding Android devices that detected the iOS BLE-token from server.
In some cases, surrounding Wi-Fi networks can help to detect the devices in proximity. iOS apps can receive the BSSID of currently connected Wi-Fi hot spots, while Android can discover the BSSID of all visible networks. Whenever a match is found, users are presumed to be located nearby.
This method is not easy to implement, because of BLE stack implementation issues under different versions of Android and iOS. SDKs that bundle the proximity and device detection processes under the hood of your apps are also available on the market.
In our project, we tried the Google Nearby cross-platform SDK. Device detection took 15–20 seconds on average, and in some cases, it can take up to 40 seconds.
You also need to take into account the fact that Bluetooth is usually turned off on most smartphones, and that calling the function from the app would require the user to respond to system messages every time to enable it.
Also, it is important to note that Bluetooth (on Android) severely drains a devices battery every time. Google warns that the use of Google Nearby increases battery consumption by 2.5–3.5 times the normal rate.
+ Proof of proximity (guarantees that the devices are nearby)
– slow detection (Google Nearby)
– high power consumption
Audio Generation
Every mobile phone is equipped with a microphone and a speaker. You can encode an ID into a sound and play it on one device, while capturing and decoding it on another device within hearing range, and thus link the devices into a group.
In the audible range, the signal is mixed with voice, music, and ambient noise. To increase the probability of correct decoding, the sound needs to be played at maximum volume. FSK and PSK modulation are most frequently used to generate a sound similar to whistling or noise, depending on the density of data. The resulting audio samples are extremely irritating to everyone around you (example) and perform poorly in noisy environments. This method is implemented in the project Chirp.io.
– poorly performs in noisy areas
– annoys others
You can use a range of 18–20 kHz, which it’s usually not polluted. Most adults are unable to hear this frequency. Unfortunately, some smartphones demonstrate a poor reception of these sounds. Due to audio wave reflection and interference, the communication range is reduced to about 1 to 3 meters. This method is implemented in the Google Nearby and Chirp.io, but needs to be enabled separately.
– short-distance detection limitation
Conclusion
There is no silver bullet in proximity device pairing, but using comparative analysis of ambient noise seems to be the safest bet.
When choosing the solution, It is important to consider the following aspects:
· Where and how often device pairing is needed in a use case.
· The number of devices a use case needs to be able to connect simultaneously.
· What OS/platforms should be supported.
· What is the anticipated user base and usage frequency.
We have tried to develop a universal tool for proximity pairing that would make our applications user-friendly and efficient. We hope that more and more applications will be able to improve their UX with Resonance Nearby in the near future. So far we’ve been piloting Resonance Nearby in some closed Betas, and publicly launched our technology in a seamless contact sharing app called Drop.
Resonance Nearby is available as iOS and Android libraries. For more information visit getresonance.net