Motivation
Philadelphia is a unique city. At its peak in 1960, Philadelphia was home to over two million people in 1950; due to deindustrialization starting in the 1960s, along with numerous other factors, the population shrunk over the next four decades, crashing its housing market.
In the last seven years, however, Philadelphia home prices have risen almost 46%, although the consequence of its previous population decline means much of the city’s housing stock is in bad shape. As a result, there are a large number of buildings which fail inspections, in addition to the constant new construction and renovation of a city undergoing a revival — all of which fall under Philadelphia’s Department of Licenses and Inspections (L&I).
Objective
My aim was to create a robust model which actively uses the Philadelphia Licenses and Inspections (L&I) permit application process as a monitoring system for potentially dangerous buildings.
This model combines two aspects: the building permit process, and the building’s historical inspection, violation, and permit data. The most egregious code violations are currently found by either L&I inspectors making the rounds, or complaints stemming from people’s observations of a building’s exterior. However, not many people who have access to a building’s interior — whether it’s the owner, contractor, or tenant — have the incentive or the structural knowledge to realize and complain about dangerous conditions or fire hazards. By using the permit process, we may be able to find unsafe buildings that have been overlooked by the usual system.
How the model works: When fed a new permit application, the model predicts how dangerous or unsafe the building could be, and whether L&I inspectors should immediately follow up. By predicting which buildings have a higher likelihood of being condemned and what buildings are the most dangerous, L&I can prioritize the buildings that inspectors should focus on first — and by doing that, increase the safety of Philadelphians. One doesn’t have to look far in the news to find Philadelphia’s unsafe housing risk, which is why a streamlined and accurate inspection process is crucial.
The Permit Process
In 2019, there were over 14,000 permits issued for building and renovation in Philadelphia. These permits range from electrical work being done by contractors, to demolitions, to minor alterations done by homeowners. The large volume of permits each year means that L&I can’t possibly send inspectors to check on every permit or inspection request in a timely manner.
A large number of buildings in Philadelphia fail their inspections and further inspector followups, while some buildings have more drastic action being taken, and are declared unsafe and uninhabitable.
Modeling Process
I built this model using data from Philadelphia’s open source data repository: Open Data Philly. Using the city’s APIs, I pulled in four large datasets: permits data, inspection data, code violation data, and unsafe violations data. I then cleaned, aggregated, and blended the data together before building the model.
In addition to the variables and aggregations from previous permits, inspections, and violations were included in the dataset, I also built custom variables in order to improve the model’s prediction accuracy.
When I started comparing the unsafe buildings vs “safe” buildings, the dataset was clearly unbalanced — due to the sheer volume of permits coming through the system, <1% of buildings were unsafe. Blindly throwing this data into a model might result in a high accuracy rate— of course it’s easy to guess a building is safe when 99% of the time you’ll be correct. However, the model will be terrible when actually trying to find the unsafe buildings.
In order to correct for this, I spent some time finessing the dataset further by resampling the data. I looked at both undersampling and SMOTE in order to re-balance the data.
I ended up pulling a large number of features, as I wasn’t sure which ones would later play an important role: therefore finding variable importance was key in slimming down and improving the model. I used Boruta for feature selection, which was used to improve model quality. Below is a graph of the most important variables used in the model.
When designing the actual model, I reviewed a variety of models; I then used ensemble learning to see if combining models would improve the results. I’ve written up more details on the modeling process here, in addition to my Github project repository.
Results
While I looked at a range of metrics in order to determine the best model for this problem, the chart below shows the accuracy rate of my model.
The accuracy rate shows the success of the model, which predicted which buildings would be flagged as unsafe by Philadelphia’s Department of Licenses and Inspections.
The final result below is the accuracy rate of the test data. Before I built the model, I split the data randomly into training and test sets. I trained the model on the training set and use it to make predictions for the test data. By comparing my model’s predictions to actual data, I am able to tell how accurate the model is.
My most recent model predicted a 85.9% accuracy rate on the test data. Figure 4 shows how these predictions were made. False negatives happen when the model predicted an unsafe building [false], but the actual result was a safe building [true]. False positives are when the model predicted a safe building [true], however the building was unsafe [false]. True negatives and true positives correctly guess the test classes.
Conclusion
My work isn’t done yet — I plan on adjusting the model and trying to improve its accuracy even more by improving upon the ensemle model. I believe that open access to government data is essential, and can spur changes in government policy and methodology.
Finally, although this model tackles a small, yet serious problem, the dataset and model that I built can be tweaked for a wide array of uses. It can also be adjusted to predict buildings that will fail inspections multiple times, or find neighborhood trends and building patterns. It could also recommend successful contractors to permit applicants based on their past successes, and monitor contractors who repeatedly incur code violations.
