Finding vulnerable housing in street view images: using AI to create safer cities

Mark Wronkiewicz
May 8, 2019 · 6 min read

This is an ongoing project to support the World Bank’s Global Program on Resilient Housing.

Hard to find, easy to fix

Identifying vulnerable housing is tedious. Currently, teams of trained experts must pace neighborhoods with clipboards to scrutinize building resilience (e.g., looking for construction materials, integrity of the design, approximate age, completeness of construction, etc.). This manual strategy can take a month or longer and is prone to human error, which drains valuable resources from the retrofitting process. The challenge is bringing the expertise of a structural engineer to every corner of a city in a way that’s accurate, fast, and repeatable.

Link to United Nations’ SDGs

Development Seed is supporting the World Bank to identify dangerous housing through the use of artificial intelligence. In this approach, the World Bank collects images in at-risk urban neighborhoods using vehicles outfitted with a multi-camera platform (similar to Google’s street view cars). A machine learning (ML) pipeline then ingests these street view images and autonomously identifies risky structures. Finally, this information is organized and displayed in an online map for decision makers to help them direct retrofitting efforts. Instead of teams of engineers, this approach only requires a driver to weave through a neighborhood with a car-mounted camera. We aim to empower governments and local communities with this technology (especially those affected by climate change); they understand that increasing resilience is important but lack knowledge on where or how to act. Insights from this project will help communities find and protect infrastructure before a disaster rather than rebuild it after.

Automating for speed

Figure 1. Detecting building properties in street view imagery. Sample detections for building completeness (left), design (middle), and construction material (right). Notice that the models work reasonably well even with obstructions (like this bus).

Next, we register these building detections in a building footprint map by combining a few pieces of information. The car’s camera platform captures its GPS location and direction of travel every time it snaps a picture. Since the camera has a known field of view and points in a fixed direction relative to the car, we know each image’s extents (in terms of compass direction). We find the center of each bounding box to determine the exact compass direction of every detected structure; for example, a painted building at 270 degrees. This allows us to draw a ray out from the car’s location in the direction of the detected building (Figure 2, right). Finally, we make the final link between the street view images and our top-down map: we search for the building polygon in our building footprint map that overlaps with each detection ray to assign the detected property (e.g., “material: painted”).

Figure 2. Registering street view detections to the map. The continuous stream of images often leads to several detections per structure (left). Knowing the car’s location, heading, and camera field of view, we create a geospatial line for each bounding box detection originating at the car and pointing outward (right). We assign this detected property (and confidence) to the first building polygon (blue box) intersected.

We apply the same logic for entire neighborhoods. The camera records multiple images per meter traveled, which leads to 5–10 detection rays for each building. We combine this information with the building footprint map traced from drone imagery. Mapping the buildings and detection rays together, the scene resembles a magnifying glass focusing rays of light on each building polygon (Figure 3). For every building, we distill these ML detections into one determination per building attribute, similar to an engineer making a single determination after inspecting a building from different angles. Repeating this for each of the three properties, we generate a neighborhood-wide building database that supports our visualization tool and populates the building profiles.

Figure 3. Mapping street view feature detections (dark lines) from image capture locations (black dots) while the car was driving past buildings (blue polygons). We assign detected features to buildings by finding the line/polygon intersection closest to the car position. The overhead building footprint map is derived from drone imagery.

Interacting with the data

Figure 4. Our web tool combines street view navigation with ML-derived housing information. The animation shows how to navigate through our pilot region (in Colombia) and open the “Housing Passport” pane containing building information and risks.
Figure 5. Users can also navigate using a building footprint map. Both the street view and building footprint maps allow users to select and view a building’s Housing Passport.

Going forward

Additional resources

Development Seed

To understand a changing planet we create, analyze and…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store