Towards Large-Scale Tree Mortality Studies in Cities with Deep Learning & Street View Images
Many inventories of street trees are recorded manually by an expert walking through cities and recording each tree with different attributes. This process is obviously expensive and doesn’t scale very well.
To address this problem, Wegner et al. (2016) proposed a way of automating the creation of urban tree inventories by using a deep learning model that detects and assigns geo-coordinates to trees from Street View images. However, in order to perform long-term tree health studies, with time-series starting before street-level imagery was available, we still rely on these traditional inventories.
Most of the existing old databases have one crucial deficiency: the tree locations are recorded as street addresses instead of geographic coordinates.
This makes them much less accurate and harder to use in practice. The goal of our project is to address this challenge: we provide an automated way to assign geo-coordinates to old tree inventories using Street View images.
The hope is that our approach on retrofitting existing street tree inventories with geographic coordinates will enable large-scale longitudinal studies, where data of the same population of trees can be analysed over decades.
Why are street trees important?
Although street trees make up a small proportion of the urban forest, they provide considerable ecosystem services to the urban landscape. These benefits include for example improvement in air quality, a reduction of the heat island effect and increased carbon capture and storage. Additionally, they can contribute to rising property values and an improvement in individual and community wellbeing.
How to approach such a problem?
Our idea: We want to use the existing tool developed by Wegner et al. (2016) to automatically find trees on Street View images and then match them to the trees in the old databases based on their addresses.
Our general workflow is the following:
- The first step is to find some approximate coordinates for each tree by using a geocoding API. This means each address is converted from a textual address to a lat/long coordinate pair. We used the Geocoding API service provided by Google, which returns the geo-coordinates centered on the building of a particular parcel. This gives us a rough idea of where those trees are located.
- Next, we download the Street View images in the vicinity of those coordinates.
- And detect the trees in these images using our trained deep neural network.
- To get their coordinates, the detections are projected into geographical space.
- As a final step, we need to match the trees found in the images with the existing ones corresponding to the addresses, using a global optimization approach. This assignment enhances the old databases with the coordinate information from the detected trees.
Finding existing trees in Street View images
This part of the workflow follows the ideas proposed by Wegner et al. (2016) and Branson et al. (2018), with a simplified algorithm in order to make it easier to scale to huge datasets.
Trees in Street View panorama images are detected using Faster R-CNN (Ren et al., 2015), one of the most popular CNNs for object detection. The output of the tree detector is one bounding box per detected tree in each Street View panorama.
Like all supervised learning algorithms, the network has to be trained with sufficient ground truth data. We have manually labeled 6783 individual tree instances in 718 Street View panoramas to get accurate and reliable results.
After downloading all the images near an address, they are fed into the tree detector which outputs several bounding boxes per image.
The metadata of the Street View images includes the camera position and heading, which lets us calculate the geographic coordinates of the detected trees by projecting the bottom center point of the bounding boxes.
Matching detected trees and existing addresses using global optimization
Now we have got a database of geocoded trees. The last step of the puzzle is to match them with the geocoded addresses from the existing inventories. We cast this matching task as an optimization problem where the total distance between all pairs of trees is minimized across an entire municipality. We want to solve the following problem:
- x_ij is our binary decision variable, indicating if an address i is matched to a tree j (1 if i is matched to j, 0 otherwise),
- dist_ij is simply the geographical distance between address i and tree j,
- M is a distance threshold to avoid matching trees to addresses that are too far away (50 meters in our experiments),
- We also introduce two constraints to ensure that only one street address is assigned per tree and that the number of assigned detected trees per street address does not exceed K, the total number of trees at that address given in the database.
By minimizing the sum over the misalignment between the geocoded trees and the geocoded addresses, we seek a globally optimal solution per city. The mathematical problem is solved using Linear Programming.
Results for around ~1 Mio trees in Californian cities
We ran experiments with tree inventories from five cities in California, which come with already existing coordinates such that we can validate our results. In total, our approach correctly assigned geographic coordinates to 22’363 individual street trees out of 57’938 input trees, which represents 38%. Results for the full processing pipeline are shown in the following figure:
The green symbols denote the correctly assigned trees, the rest is categorized into the different error reasons. Those include erroneous geocoding of the addresses, missing Google Street View images, and incorrect assignments.
Particularly difficult cases are street addresses of large buildings with a high number of trees such as these two examples:
Our system comes at virtually no cost if we put aside costs for running computers and downloading Google images. Given appropriate hardware and a fast internet connection for downloading images, the method scales to arbitrarily large data sets.
To demonstrate this, we ran our processing pipeline on 48 municipalities containing altogether 1’100’952 input trees. Our method assigned geographic coordinates to 56% of the individual street trees.
Conclusion
We have presented a novel approach to assign geographic coordinates to street-trees given street addresses and Street View panorama images. We were able to assign correct geographic coordinates to 38% of the street trees, which is a good starting point for long-term studies on the ecosystem services value of street trees at large scale.
There are different reasons, which account for this percentage being relatively low. One could be the varying geocoding quality provided by the API. Also, the detection of trees can fail due to missing panoramas, occlusions or an error by the detector. The incorrect assignment is another major reason for mismatches, which is generally difficult to fix. It is due to mixing up trees close to parcel boundaries or also when several trees are on the same property, which is very often the case.
We hope this work encourages further research to generate longer time series (over several decades) of street-tree inventories with the ultimate goal of better understanding street tree population dynamics and corresponding changes in ecosystem services at a very large scale.
For the interested reader, we refer to our full research paper: here
Laumer, D., Lang, N., van Doorn, N., Aodha, O.M., Perona, P., Wegner, J.D. “Geocoding of trees from street addresses and street-level images” ISPRS Journal of Photogrammetry and Remote Sensing 162(2020)
References
Branson, S., Wegner, J., Hall, D., Lang, N., Schindler, K., Perona, P., 2018. From google maps to a fine-grained catalog of street trees. ISPRS J. Photogramm. Remote Sens. 135, 13–30.
Ren, S., He, K., Girshick, R., Sun, J., 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems (NIPS).
Wegner, J.D., Branson, S., Hall, D., Schindler, K., Perona, P., 2016. Cataloging public objects using aerial and street-level images — urban trees. In: IEEE Conference on Computer Vision and Pattern Recognition.
