# Improving the Zestimate: An Experiment

Dec 11, 2018 · 6 min read

Foxy AI set out to improve residential real estate valuations by incorporating image data. To do this, Foxy AI research developed FoxyNet, the Convolutional Neural Network that powers house2vec. House2vec takes a raw image and returns an image feature vector, embedded in a high dimensional space, containing information on the quality and condition of the property for use in valuation models, among many other applications.

We decided to pair house2vec with arguably the most famous and divisive state of the art valuation model, Zillow’s Zestimate. The results of several experiments will be discussed in a multi-part series.

# Overall Results

Low end: <\$175k

Mid-range: \$175k — \$750k

High end: >\$750k

In the low-end bracket, the house2vec + Zestimate model predicted sales price outperformed Zillow’s Zestimate 60% of the time.

In both the mid-range and high-end brackets, the house2vec + Zestimate model predicted sales price outperformed Zillow’s Zestimate 40% of the time.

Let’s dive into some specific examples of where Foxy AI outperformed Zillow and where Zillow outperformed Foxy AI.

We learn the most from studying our failures so let’s first look at an example where the Zestimate outperformed our combined model, and attempt to understand why we weren't able to improve the prediction.

In the following example, Zillow managed an impressive prediction within 3.5% of the sale price. Foxy’s combined model was 43.5% off of the sales price. (Note: Zillow has since updated their Zestimate to incorporate the new sales price.)

Zillow’s Zestimate: \$106,265

Foxy + Zestimate Predicted Price: \$194,832.92

Sale Price: \$110,000

Let’s dig into this listing and try to figure out what happened. Upon initial inspection of the listing, we can see there are a substantial number of individual property level features listed, as well as a decent set of property photos. Also, this property is a condo located in a professionally managed community. The community features a pool, tennis courts, a clubhouse, and professionally landscaped grounds. The image set includes pictures of these luxury amenities.

We believe these community amenity photos throw off our combined model. This case presents itself often, when listings include images outside of the actual property being sold. If we show a human a set of photos that contained a house, a large pool, as well as an additional structure on the property, without knowing these features are actually part of a larger private community, it is reasonable to assume that he or she would guess a higher property value. We believe this is exactly what happened here.

In our second example, Foxy AI outperformed the Zestimate. Foxy’s combined model was 3.87% off of the final sales price vs. Zillow’s 18.01%. An impressive improvement over the Zestimate alone. (Note: Zillow has since updated their Zestimate to incorporate the new sales price.)

Zillow’s Zestimate: \$201,246

Foxy + Zestimate Predicted Price: \$171,635.20

Sale price: \$165,000

You can see from the listing that we have a robust set of photos, and according to the listing description, the property was recently renovated. So why did we outperform the Zestimate and why did we predict a price lower than the Zestimate?

The listing says the property was recently renovated and it appears to be in good condition. As mentioned previously, our network is trained to understand both quality and condition.

Upon inspection of the photos, one could argue that the kitchen and bathroom floor tile are “builder grade” quality. Similarly, the kitchen countertops, although granite, appear to be Ubatuba, which falls in the low end on the granite price spectrum and would also be considered “builder grade”. There is little landscaping to speak of, and while the interior features on Zillow list hardwood floors, we can clearly see there is, in fact, no hardwood at all.

Foxy AI picks up on these subtleties and factors them into its predicted price.

Real estate pricing is inherently a subjective matter. What I think is nice granite, you might consider ugly, but at the end of the day, the value is ultimately what a buyer is willing to pay. The FoxyNet is a CNN trained on millions of property photos. The beauty of a deep neural network is that it “remembers” these nuances.

When compared to the first example which has a substantial number of features listed, this listing has relatively few property level features. Aside from some basic information like square feet, the number of bedrooms, lot size, etc., there is little data for the Zestimate to factor into its price estimate.

We believe that the quality and condition data provided by house2vec, is ultimately why Foxy AI’s combined model outperforms the Zestimate alone. House2vec can especially help in listings missing categorical data and provides that final frontier of missing information the Zestimate doesn’t seem to take into account.

Both of the preceding examples were of properties in the low-end bracket, where Foxy AI performed the best.

What we learned from the low-end bracket applies just the same to the other pricing brackets. However, we found that listings in these pricing brackets were more likely to have a robust set of property level features, which is why we outperformed the Zestimate only 40% of the time.

# Research Limitations

There are several limitations to this research. The first is that the sample size is small. Today’s machine learning models need data and lots of it. After splitting the data into training and test sets, we have even fewer than 1,000 listings on which to train our downstream model.

The second limitation is that we are working with relatively few model features. Zillow’s valuation model likely takes into account hundreds if not thousands of features, but it is all boiled down into the Zestimate. That compression causes loss of information.

Written by

Written by