Evaluating the Effect of Property Primary Photo Changes
As a technology company innovating in hospitality industry, Airy adopts agile development and lean startup method. Through build-measure-learn loop, we focus on being agile during build step and keep validating our product by measuring its value and learning from the results. Although learning is the last step in the loop, it is actually the first thing we do before starting the build step: we initially set business hypothesis and generate ideas. If we do not measure and learn from each iteration, we might end up building things which are not aligned with our business value [1].
Prior to measurement, we should already have a hypothesis. On one of our previous studies, we hypothesized that changing the angle of Airyrooms primary photo on search engine results page (SERP) to diagonal view will increase the performance of our conversion funnel (from SERP to property detail page). By displaying photo with diagonal view, we assumed visitors could get a broader view of the room and find their preferred rooms easier.
This hypothesis could be validated by conducting a research. There are two common research designs [2]:
- Experimental study
In this study, researchers attempt to analyze the effect of an intervention by splitting the subject of interest into control and treatment groups.
- Randomized controlled trial (RCT)
It is commonly conducted in online platform (known as split test or A/B test). On this experiment, users are assigned randomly to different treatment groups, then the researchers observe whether treatment A is better than B [3]. While RCT has low bias due to randomization, it is expensive and time consuming. - Quasi-experimental research
It shares similar characteristics with RCT, except on the randomization. Quasi-experimental research allows researchers to assign subjects to treatment or control group using other criterion. Although it is subject to confounding variable, it reduces the time and resources needed to conduct the experiment [4].
2. Observational study
Observational study focuses on observing the effect of a treatment, without trying to change which subject is exposed to the treatment. This study is usually conducted when there are any ethical concerns or logistical constraints if the researchers proceed with experiment.
Considering that the impact of changing the angle of primary photo is lower than the required time and man-hour needed to set up randomized experiment, we decided to conduct quasi-experimental research. Using property as the subject of analysis, we assigned some properties from top-searched locations as treatment group (with diagonal view as primary photo) and kept other properties as control group. We sampled properties from top-searched locations to ensure that the properties are discovered on SERP during the experimentation period.
Our hypothesis statement is formalized below.
H0: properties with diagonal view photo have lower or equal conversion rate compared to properties with bed-front view photo.
H1: properties with diagonal view photo have higher conversion rate than properties with bed-front view photo.
There were two batches of experiment, with different properties listed in the treatment group. How did we measure the results?
Causal impact analysis
To evaluate the impact, we compare observation from the treatment group and the counterfactual. Since we could not know the counterfactual (i.e. observation that would have occurred without conducting the experiment), we construct a synthetic control group, which resembles the potential outcome given there is no experiment. The effect equals to the differences between actual observation (post intervention) and the synthetic control group.
There are three sources of information to construct a synthetic control group [5]:
- Time series data of the observation prior to intervention.
We used historical conversion rate of the properties before the experiment was conducted. - Similar time series data which are predictive to the observation.
To determine the similarity, we could pick other data which have positive correlation to the observation. We picked other properties, i.e. those without photo changes, as the control group because its conversion rate has strong Pearson correlation with treatment groups from both batches (0.75 and 0.58 respectively). Those properties (red line) also have similar conversion trend to the treatment groups (cyan and yellow lines, see Figure 3).
- Available prior knowledge about the model parameters.
The synthetic control group is based on state-space time series model, which allows us to incorporate prior knowledge about volatility of the data. On this analysis, we do not set prior parameters and let the model determines the parameters itself.
We used CausalImpact package on R, which resulted in 3 plots (Figure 4 and 5).
- Original: shows the actual observation (the solid line) and the predicted value (dashed line).
- Pointwise: shows the difference between the actual observation and the predicted value per each data point.
- Cumulative: shows the cumulative difference between the actual observation and the predicted value.
First batch
On most days after the experiment, the conversion rate is higher than the control group (pointwise plot on Figure 4). We observe 1.27% increase in the conversion rate after the intervention (with a 95% confidence interval [0.90%, 1.67%]), and the result is considered statistically significant.
Second batch
On this batch, we observe 0.72% increase in the conversion rate after the intervention, with a 95% confidence interval [0.45%, 0.99%]. In spite of having lower increase compared to April 4 batch’s result, it still shows statistically significant result.
Conclusion
Although both analysis shows statistically significant results, we should get back to the stakeholders and discuss whether the observed 1.27% and 0.72% increase are considered significant or not in business view. We expect the experiment will increase conversion rate at least by 2%, then it doesn’t prove the hypothesis even though we have statistically significant experiment results. However, we still applied the changes because it is still positively impacting business.
Changing the angle of a photo sounds simple, yet it actually could improve conversion rate. This idea was proposed by one of our employees. At Airy, we encourage everyone to be bold and innovative. As an employee, everyone could propose an improvement idea (with justification, of course!) along with the hypothesis. A proposal might be implemented if it is aligned to the current company’s OKRs and is feasible to do. If you have lots of ideas and are eager to improving hospitality industry, no worries, we’re hiring!
Reference
[1] Burns, B. (2018, April 24). Lean Startup vs Agile. Retrieved from https://medium.com/giglabs/lean-startup-vs-agile-dab65069392e
[2] Cohort studies, case control studies, RCTs*. (2016). Institute for Work and Health. Retrieved from https://www.iwh.on.ca/sites/iwh/files/iwh/at-work/at_work_83.pdf
[3] Tamburelli, G., & Margara, A. (2014). Towards Automated A/B Testing. International Symposium on Search Based Software Engineering, pp. 184–198. Retrieved from https://home.deib.polimi.it/margara/papers/ssbse14.pdf
[4] Shuttleworth, M. (2008, Aug 13). Quasi-Experimental Design. Retrieved from Explorable.com: https://explorable.com/quasi-experimental-design
[5] Brodersen, K., et. al. (2015). Inferring causal impact using Bayesian structural time-series models. Annals of Applied Statistics, vol. 9 (2015), pp. 247–274. Retrieved from https://ai.google/research/pubs/pub41854
