One Concern
Published in

One Concern

Leveraging computer vision to resolve missing data sets

Katie Brown, Anand Sampat and Abhineet Gupta
Research by Todd MacDonald and Abhineet Gupta

The team presented part of this project at the 2019 SCEC Annual Meeting in September 2019. For more information, please visit


Advanced AI and machine learning algorithms are increasingly deployed in nearly every industry, from entertainment to financial services to healthcare and, most recently, in the response to COVID-19. At One Concern, we are one of few companies deploying this technology to improve the world’s understanding of disaster risk and resilience. With world-class physical models as our foundation, machine learning helps us train our models, learn from local environments and efficiently process massive amounts of data.

One of the more significant hurdles in our work is access to data that is crucial for estimating disaster impacts, like structural characteristics of buildings. Whether it is unconsolidated data sources, gaps in datasets, or nonexistent data altogether, researchers and data scientists must find novel ways to expand model parameters and piece together missing information.

As a potential solution to a missing data set that helps us estimate earthquake vulnerability of buildings, we recently explored widely-available image data from Google Street View. Since this data is “unstructured,” meaning there are no parameters to build a traditional machine learning model, our engineers and data scientists used deep learning and computer vision to analyze it for our purposes. Deep learning computer vision models are able to efficiently process unstructured data by creating their own parameters to understand relationships between the image input and the desired output classification.

Increasingly, computer vision models are being used in the field of structural and civil engineering, in order to mitigate the expensive and time-consuming task of field data processing. We view computer vision as an essential tool in the development of our technology, and are continually looking for new opportunities to apply this technology to projects in the real world.

The Problem: Soft Story Building Identification

During the 1989 Loma Prieta earthquake in the Bay Area, some of the buildings that collapsed were older structures with garages on their first floor, referred to as soft story buildings. Soft story buildings have fewer walls or frames on the first floor (usually due to a garage), which makes them highly vulnerable to earthquake damage. Identifying soft story buildings can help cities in passing retrofit ordinances to improve their structural performance and reduce loss of life in future earthquakes. But without existing data on soft story building inventory, the only historically available option has been to go to each building and individually record each soft story structure.

In an effort to mitigate this inefficient and expensive process, One Concern leveraged Google Street View imagery and categorical data, combined with human and machine intelligence, to assess the presence of soft story buildings in a given jurisdiction. We implemented this project in the city of San Jose, CA to understand its viability.


The city of San Jose has over 200,000 buildings encompassing nearly every classification. In order to provide a more relevant, manageable dataset to test our calibrated computer vision model (ResNet 50) for soft story identification, we needed to break our process out into a series of steps.

Step 1: Our team first employed a numerical machine learning model, called Random Forest, which was trained to identify soft story buildings in three cities in California from which we already had soft story data. This data included: construction type, stories, construction year, foundation type, land use, area, property value, number of bathrooms and total rooms. After dividing the data in these cities into training and holdout test sets, our Random Forest model had a test recall of 93%, and a test precision of 21%.

Our models are tuned for high recall, which allows us to catch as many instances of soft story buildings as possible. Although it can result in false positives, it is ideal for our specific use case, where we would like to minimize the number of misses in identifying soft story buildings.

We then used this model to classify possible soft story buildings in San Jose, and were able to reduce the buildings of interest from about 200,000 to about 18,000.

Step 2: Next, the team trained the ResNet50 computer vision model on the same data set used to train the Random Forest model, but this time using Google Street View images from about 2500 buildings in those cities. To do this, the team took the Google Street View imagery for each address, and augmented the images using horizontal flips, width and height shifts, and zooming.

This model was then implemented on the Street View imagery for the 18,000 buildings from Step 1 to identify soft story buildings in San Jose. In this process, the model identified about 3000 buildings as a soft story.

Step 3: Since none of the images used in training the computer vision model in Step 2 were from San Jose, we incorporated building data from San Jose during Step 3.

To do this, we hand-labeled each of the 3000 buildings identified as soft story, and some of the buildings identified as non soft story buildings, to augment the training dataset. This “feedback” process helps in improving the model by using the outputs from the more rudimentary model in Step 2 to re-train the model with a larger training set.

Although usually not a feasible task, hand-labeling was possible in this case because of the relatively small data set size of 3000 buildings, compared to the 18,000 buildings remaining after Step 1. After dividing the data into training and test sets, our ResNet50 model had a test recall of 91%, and a test precision of 86%. We implemented this new model again on the 18,000 buildings from Step 1.

Step 4: The last step involved hand-labeling the buildings identified as a soft story in Step 3, in order to increase the level of precision in our end result. The team incorporated this step because it allowed us to maximize the quality of the output with relatively low effort.


With human and AI model collaboration, this process identified about 1400 buildings in the city as soft story structures. As a final check on the efficacy of this approach, it was important for us to test these results against on-the-ground data.

On-The-Ground Validation: Results and Limitations

Our on-the-ground team visited buildings in San Jose based on the model outputs, and walked around all sides of the buildings to identify 130 buildings as soft story, and 65 as non soft story. After comparing our model outputs with the collected data, we found that our modeling approach reached a precision of 90% and a recall of 37%. This meant that while the approach worked very well to identify a true soft story building as a soft story building, it also missed 63% of the soft story buildings and instead classified them as non soft story.

The team then dug deeper into this discrepancy to understand the gaps. This allowed us to make judgments about Google Street View as a potential data source for this task, and gave insight into the limitations that come with using this kind of data.

The most significant limitation stemmed from broader challenges faced in hazard modeling as a science — unknowns within the data. The team identified two types of issues — when the model would identify a soft story where there was not a soft story (false positive) and when it would not identify one where a soft story actually existed (false negative).

False Positives

There were very few false positives and the team identified two scenarios for them. In the first scenario, a building may have appeared to have a soft story through the Street View image due to presence of a garage, but upon physical inspection was declared to likely be a non-soft story because of newer construction or having been retrofitted.

Presence of garage indicated soft story in the model but building appears to be new construction, and likely not a soft story.

In the other scenario, Street View images were not associated with the correct address, resulting in ambiguous imagery for the model to interpret.

While center building does not have a garage, incorrect geolocation used the adjacent building images as model inputs resulting in the model identifying center building as soft story.

False Negatives

The false negatives were more predominant and also less acceptable in this task. The biggest reason for false negatives was that only one Street View image was used for each building and that image may not include that face of the building which has the soft story. In San Jose, the field collection team found that most buildings had a garage either along the sides or the back, and not on the side facing the street. As a result, if a soft story building’s garage is in the back, hidden from the street, it would not be determined as a soft story by the model.

Garage was not present in the Street View image that was associated with this building, but the garage can be seen in a different image, so the building was incorrectly classified as non soft-story.

In these exercises, identifying your limitations is crucial — it gives you new questions to ask, new problems to solve, and new insights into how you can improve your overall analysis of hazard risk. One of the other limitations that we did not explore in this project is the generalizability of this model to other cities. Different geographical regions may use different building designs, and buildings may have varying structural features. Ideally, the model should be trained with local data to improve performance.


Google Street View is unlikely to be a viable data source for determining soft story building locations. At least in its current state, the data is not sufficiently comprehensive to be able to use on its own. But this doesn’t mean the data isn’t useful — identifying which buildings to hand-label is a massive improvement from hand-labeling every building in a city. And the shortcomings we have gathered from this dataset give us important insights into how we can gather more complete data going forward.

With more complete data, these kinds of projects — and particularly those using computer vision — become more tractable. Cities are already responding to this need: Santa Clara, for example, is performing a Lidar survey of the entire county, which can help with 3-D construction of building exteriors. Such high-quality image data could potentially lead to better soft story identification, giving us more precise knowledge of soft story building whereabouts in a given city.

With this information, cities would no longer be held to estimation and guesswork about the most earthquake-vulnerable areas of their jurisdiction. And perhaps most importantly, this information presents emergency management professionals not only with the knowledge of vulnerable buildings, but of the vulnerable people, businesses and communities within them.

By evaluating new data sources and testing the boundaries of our technology, we are continually moving closer to the potential to transform entire cities — from earthquake-vulnerable, to disaster-resilient.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
One Concern

One Concern

We’re advancing science and technology to build global resilience, and make disasters less disastrous