We Tested Mobile GPS/GNSS Accuracy and Found Some Surprising Results

Are modern mobile phones as accurate as they say?

It is no secret that all mobile phones use GNSS for location. This has been the case for some time, and by now, most would assume that GNSS is getting more and more accurate every year. However, we recently conducted a test to see just how accurate different mobile phone models really are in their GNSS location. The results were genuinely surprising.

NOTE: GNSS refers to a variety of satellite systems used across the globe. These systems include the following: GPS (US), GLONASS (Russia), Galileo (EU), BeiDou (China), QZSS (Japan), and IRNSS (India).

This article will discuss our goals with the test, the design and methodologies we used, and the results that we found. If you’d like to skip ahead, scroll to the results section of this article. Lastly, reach out to info@important.com if you are a phone or vehicle manufacturer and would like to be part of our next test in September 2020.

Our Goal

First off, why did we do this? Our company is soon deploying a free mobile app that aims to predict collision between Vulnerable Road Users (VRU’s — aka pedestrians, cyclists, and motorcyclists) and connected vehicles. We use location data in combination with machine learning to predict if a collision is about to happen, alerting users, and even triggering the vehicle's brakes automatically if necessary.

While our technology does use software to improve accuracy, we still rely heavily on the quality and performance of chips embedded in smartphones. In particular, GNSS (Global Navigation Satellite System) performance is absolutely critical to our success.

Watch this 30-second video demonstrating our pedestrian-to-vehicle communication.

Ultimately, we will need to achieve sub-meter precision of mobile GNSS location for our service to save lives. Thus, we devised a test to calculate location on the most accurate level possible.

Test Design

Overview

Our test aims to compare precise GNSS performance based on smartphone type, location of the phone, and the path taken by the phone. We do this by using the Eos Positioning Systems Arrow Gold.

Test Conditions & Hypotheses

Location: Montreal, Canada

Date: May 19th, 2020

Conditions: Sunny, clear skies, no skyscrapers

Hypotheses

• High-end mobile devices, in conjunction with our software, may already offer a sub-meter accuracy.
• A variety of conditions such as the location of the phone and distance of the path (jacket pocket, pants pocket, etc.) may affect the accuracy of GNSS.

Methodology

To be proven “safety-critical” with a sub-meter capacity, the coordinates outputted by each of the 5 smartphones (iPhone XR, iPhone 6 SE, Samsung Galaxy 10 Note+, Pixel 4 and Samsung Core A2) have been compared with the coordinates ones of an ultra-precise system offering a half-inch level GNSS accuracy. This device is the Arrow Gold made by Eos Positioning Systems.

Ground-Based Augmentation Reference Beacons or Real-Time Kinematic (RTK) were used to regulate the position.

Parameters

There were two primary testing parameters.

1. The path taken by the phone (straight vs. curved)
2. Use case of the phone (in hand, in jacket, in pants)

Figure 1: Setting up the experiment in Montreal parking lot

To accurately assess the accuracy, we performed one test along a straight line and another along a more complex line with multiple curves and changes in direction. To identify confounds in our accuracy, we ran the test following multiple real-life scenarios, including phone in the side pocket, phone in the rear pocket, phone in the hand, phone by a swinging arm.

We finally compared the raw device data to determine accuracy.

Path Design

Figure 2: Left = Complex fixed coordinates with RTK only | Right = Simple fixed coordinates with RTK only

Each red ‘x’ represents a position that we marked with very high accuracy using the RTK device. These points are referred to as “reference points” in this document.

Data collection & calculations

Once the data was collected both by our test devices and by the high accuracy equipment, we deducted the accuracy mean using confidence intervals. The following methodology was used:

1. Install EOS Tools Pro to pair a smartphone with a high precision positioning system.
2. Collect the exact positioning data (reference data) of the two test paths with the Arrow Gold RTK device via the smartphone. The test paths are described below.
3. Extract the data in CSV and KML formats.
4. Export it on a GIS system.
5. Collect the data following the test path with the four aforementioned test devices, 3 times for each of the 4 scenarios.
6. Export it on the same GIS system, with the option to show or hide each sample associated with its metadata.
7. Regroup the test data by scenario and device pair, average each group and compare them with each other or with the reference data, to extract the metrics as described below.

Each result was averaged over multiple recordings in the exact same scenario for a greater level of confidence.

Results

As can be seen in Figure 2 below, the 6 points on the red line indicate various precision measurements taken with the Arrow Gold throughout the study. Thus, the red line represents the true path. We then subtracted the recorded mobile path from the true path to determine each phone’s precision at each point. These results are listed in rows 2 through 7 in figure 3.

Figure 3: Map comparing True GNSS data (red) with the phone’s GNSS data (others).

Figure 4: Results from the experiment. Columns ordered left to right from most precise to lease precise APP.

1. Some Smartphones have great GNSS accuracy

As shown in figures 3 & 4, there is quite a big difference in GNSS accuracy between smartphone models. The Samsung Galaxy Note 10 clocks in an impressive 1.898 meters from the true location while the iPhone XR is ~22 meters away.

Accuracy is shown to differ based on the manufacturer, the device’s age, and the device’s components. These results are based on the phone coordinates as they follow a jagged path, about 18 meters in total length.

Figure 5: Same as figure 3 with accuracy estimation highlighted.

2. Phones’ Estimation of Accuracy is Not Accurate

The highlighted row in figure 5 is the distance up to which each phone claims it can be accurate. Below that figure is the percentage difference from what the phone claims versus what our findings indicate.

As can be seen, the estimated GNSS accuracy by the phones can greatly vary from the true location. The Samsung Galaxy Note 10+ actually undervalues its accuracy by 67.3% while all other phones tested overvalue their accuracy by at least 130%. The iPhone XR overvalued its accuracy by 450%.

3. Dual Frequency with Galileo is a Big Deal

The Samsung Galaxy Note 10+ seems to have greater GNSS capabilities than all other phones tested. This could be due to the dual-frequency antennas available on this model. Further tests are needed but it appears to make a clear difference, especially when dual-frequency is combined to the Galileo satellite system. Although, Galileo by itself doesn’t seem to offer greater accuracy on other phone models.

4. Moving Precision is Better than Still Precision

On average, the devices offer nearly 20% higher precision when moving compared to when still. During movement, the number of connected satellites is higher, leading to better precision. Conversely, when still, the GPS drift effect disrupts the signal because of environmental factors like buildings, trees, and weather conditions.

These hypotheses can be validated in future testing.

5. Longer Paths Offer Greater Accuracy than Acute Ones

Figure 6: A longer looped path comparing Galaxy Note 10+ accuracy with the true path (Arrow Gold)

Here the accuracy of smartphones was compared in a large path with a turn. We found that accuracy improved by 67% when a larger path was taken. For the Galaxy Note 10+, precision increased from 3-meter accuracy in the short path to 1-meter accuracy on the longer path.

Longer paths seem to offer greater reliability and precision of GNSS signal, an observation also evident in the prediction of normal pedestrian behavior. The reason seems to be that the large path has less sharp changes in direction, but also that movements can help the signal stabilizing, as stated in previous results.

We noticed that there is a significant variation in the GNSS accuracy depending on how a user carries their smartphone. We tested this with the Samsung Galaxy Note 10+.

Figure 7: Looped path visualizing accuracy variation in 5 different carrying methods

Figure 8: Table displaying accuracy and relative precision by 5 different carrying methods

As shown above, when the phone moves out of sync with the movement of the carrier, accuracy is heavily affected. Note that in the “swinging hand” case, we performed large swings during the first half of the recordings and then fixed the phone to the body during the second half. We can observe that the swing of the hand induces a “staircase effect” with very poor precision, while the precision is great after the swing stops. This is probably due to internal sensors (gyroscope and accelerometer generating an inertia effect) used to correct the GNSS signal if disrupted by the movements. This assumption is to be validated with future testing.

After stabilizing the phone halfway, the precision improves from a 9x reduction in overall accuracy to a 3x reduction. These findings lead us to recommend carrying phones in coat pockets or in hand with limited movements. Abiding by this method of carrying while using the !important app will optimize the GNSS signal and the likelihood that one is saved from a collision.

Conclusions

Our goal was to find when the use of !important’s mobile application would prove to be safety-critical. We are happy to report that our technology has immediate functionality in two important areas: alerts & slow-downs.

1. !Important App is Safety-Critical for Alerts

All current phones below 10m accuracy are safety-critical for alerting users of possible collisions. This means that every single model, excluding the iPhone XR, can immediately deploy the !important application and benefit from its safety alert feature.

2. Safety-Critical for Slowing Down the Vehicle

In all phone models that achieve sub-meter accuracy, !important technology is safety-critical for slowing down vehicles. This is an incredible development! Our results indicate that phones equipped with a GNSS chip and dual-frequency antennas paired with Galileo offer an accuracy below 1m when in movement and 2m while stationary. While this is currently only seen in the Samsung Galaxy Note 10+, we can expect newer phone models to follow suit and achieve a sub-meter level of GPS precision in the coming years. When this occurs, our slow-down functionality will be applicable to almost every phone on the market.

As can be seen in the chart below, speed is a massive risk in auto collisions. Thus, the feature of slowing down vehicles accurately with mobile phone location data is absolutely crucial to our goal of saving lives.

Figure 9: Speed Risk Table | © Important 2020 from Pitt et al. (Australia & US, 1990)

Over time, further tests will reveal which phone models are most relevant and sufficient for mass adoption of !important technology. It is fair to say that, based on historical data, the smartphone industry will recycle itself with newer models and increasingly greater GNSS accuracy. Today — in June 2020 — we may be about 1 to 3 years out from a critical mass of accurate smartphones.

3. Not Yet Safety-Critical to Fully Apply Brakes

While the current GNSS precision of smartphones available in the market today (June 2020) is very close, we want to wait until all phones catch up in precision before we deploy our full braking functionality. Therefore, we can’t yet conclude that !important is safety-critical in applying brakes to vehicles based on current smartphone models.

Of course, this situation is subject to improvement as time goes on. !Important will be diligent in monitoring the changes in GNSS accuracy of smartphones over the coming months and years. We know that soon, mobile phones will be transmitting their position with enough accuracy to be safety-critical to trigger brakes on vehicles. Our goal is to remain at the forefront of this technology and on the cutting edge of collision detection.

Next Tests Coming on September 1, 2020

Thank you for reading! We hope that you found this article useful and interesting. New tests will be coming on September 1, 2020, where we plan to evaluate a variety of new features and conditions.

We invite phone manufacturers (like Apple, Android, Samsung, Nokia, LG, Huawei, HTC, BlackBerry, Motorola, Sony, OnePlus, Xiaomi, Oppo, Mobicel, Lenovo, and others) to communicate with !important and have their new or old phone models tested and compared with accurate GNSS devices like the Eos Positioning Systems Arrow Gold.

Additionally, we encourage car manufacturers (like Waymo, Aurora, Uber, Lyft, Toyota, GM, Ford, Tesla, Honda, Nissan, Mitsubishi, Audi, Volkswagen, Mercedes-Benz, Jaguar, Chrysler, Fiat, Renault, Volvo, Hyundai, Jeep, BMW, Citroën, KIA, Land Rover, Dodge, Opel, Peugeot, Tata, Suzuki, Porsche, Isuzu, and other OEMs) to contact !important to get their vehicles tested.

To join our next phase of mobile device and vehicle testing, please write to info@important.com before August 15, 2020.

Together we can help improve public safety. Because we are all important.