Mapping sun glare (dazzling sun while driving)using Google Street View

Just to think about that every time you drive from West to East at the sunrise time or drive from East to West during sunset time, do you feel the dazzling sun blind your eyes and make driving super difficult? I guess many people want to know when and where the dazzling sun occurs. I got the idea to predict the occurrence of sun glare when I was in a meeting with Liberty Mutual in Boston. I then think about how to predict it. I have been using Google Street View images in urban applications for many years, my previous project is Treepedia is a good one. I then to realize Google Street View may be a perfect dataset for predicting and mapping the sun glare occurrence because the Google Street View images have similar view angles with drivers and those images are actually collected by cars. Here is the tutorial to show you how to predict and map the sun glare using Google Street View and deep learning.

Step 1. Data preparation

In order to collect Google Street View images, we need to generate sample points along the streets first. The street map can be found from the city government or TIGER (https://www.census.gov/geographies/mapping-files/time-series/geo/tiger-line-file.html) website. We can split each street segment by a distance of 40 m in order to create sample sites. I have developed a code for Treepedia project to create sample sites before. We can reuse this code to create sample sites along the streets. Fig. 1(a) shows the generated sample sites along the streets of Cambridge, Massachusetts. Based on the coordinates of those created sample sites, we can further collect the metadata of GSV panoramas (Fig. 1(b)) using Google Street View metadata application programming interface (API).

Fig. 1. The workflow for GSV panorama collection in Cambridge, (a) the created sample sites (b) the metadata of GSV panoramas, (c) a collected GSV panorama of one sample site using Google Map API.

Step 2. Image segmentation using PSPNet

Different objects in the street-level images would have different impacts on the dazzling sunlight. For example, the sky would not block sunlight while buildings and trees would obstruct the sunlight. Therefore, we need to recognize different objects from the street-level images in order to predict the occurrence of sun glare. Recent progresses in artificial intelligence make it possible to recognize obstructions along streets from GSV panoramas accurately. In this study, we used the Pyramid Scene Parsing Network (PSPNet), which is a deep convolutional neural network trained on the ADE20K dataset, to segment GSV panoramas. There are many implemented PSPNet repositories on Github. I suggest using the code maintained by MIT CSAIL (link).

Fig. 2. Image segmentation results on GSV panoramas using pyramid scene parsing convolutional neural network (PSPNet).

Step 3. Sun glare prediction

We get everything ready. We can then predict the sun glare by overlaying the sun paths on the segmented Google Street View images. We can use the open-source python module - Pysolar to calculate the sun position for any given time and location. The following figure is the geometrical model of the sun position and the driver.

Fig. 3. The geometrical model of the sun position and the location of vehicle on driveway

The geometrical model of the sun position and the vehicle can be modeled using street-level images. The following figure shows the overlay of sun positions on Google Street View images. If the sun position is on non-sky pixels, then we can say that the sun is obstructed.

Fig. 4. The sun paths on July 15th, August 15th, September 15th, and October 15th on cylindrical Google Street View panoramas (a) and the segmented cylindrical panoramas (b) at three sites of the study area.

Usually, the sun glare happens when the sun is in the field of view of drivers and the sun is not blocked at the same time. If we assume a driver’s horizontal and vertical view angles are 50 degrees. Therefore, three rules must be met if sun glare occurs, (1). the relative horizontal angle is smaller than 25 degree, (2) the relative vertical angle between the sun and driver is smaller than 25 degree, (3) the sunlight is not blocked by obstructions, which means the sun position is projected on sky pixels.

Fig. 5. The geometrical model of projecting sun position at a certain time on a cylindrical GSV panorama.

We can easily use these three rules based on the generated Google Street View images segmentation results and sun position model to predict the sun glare map. I have generated the sun glare occurrence map for whole Florida using more than 300,000 Google Street View images. Here is a video to show the sun glare map on highways of Florida.

Sun glare map in Florida

I used my iPhone to take pictures around the Microsoft Building at Cambridge, Massachusetts during sunset and validated the results. Here you can see the comparison of predicted sun position on Google Street View images and the sunset pictures I took.

Fig. 6. The comparison of the predicted sun path on a GSV panorama and three photos taken at the same location at three different time points of July 5th, 2018, (a) the sun path of July 5th on a GSV panorama, (b) three photos taken at the same location at different times, © the overlay of sun path of July 5th on the segmented GSV panorama, black part represents obstruction pixels and white part represent the sky pixels.

The validation results show that the proposed method can estimate the exact time point of sun glare been blocked with an error of less than 3 min, which further proves that the proposed method based on GSV panoramas can estimate the obstruction of sunlight with high accuracy.

Reference

Li, X., Cai, B. Y., Qiu, W., Zhao, J., & Ratti, C. (2019). A novel method for predicting and mapping the occurrence of sun glare using Google Street View. Transportation Research Part C: Emerging Technologies, 106, 132–144.