A data-driven approach to redesigning app features
How BUX used data to prototype a new feature and A/B test its performance
At BUX, we do our best to not only rely on our gut feeling about the directions we take but also follow a data-driven approach to decision making. If possible, we combine the user data we obtain via different forms of in-app activity tracking with qualitative feedback gathered from our users (CRM tools, questionnaires, meetups organized by BUX, etc.).
In this article, I would like to describe one of the bigger projects the team worked on recently, which resulted in redesigning one of the key features of the app (the change should already be visible in the latest versions of the app!).
Background and motivation
Ever since the first version of the BUX X app (previously known just as BUX), the home screen is the screen that welcomes the users to the app.
The upper side of the screen contains the latest news from the financial markets, prepared by our Brand & Communications Team. Below, we can see the part of the screen that serves as a logbook — we see all the trades of BUX users that are happening right now (as well as information on joining channels containing trading tips, etc.). Additionally, we can switch between ALL (the entire userbase), STARS (the users who are the best performers in each asset class this week) and FRIENDS (intuitively, our friends in the app).
The home screen is a very vibrant place, showing the users how much is happening in the app at any given time. However, at the same time, we heard feedback that it might be a bit overwhelming, as it is constantly changing and it might be a bit difficult to keep track of what is happening. That is why we brainstormed how to modify the existing home screen to improve the user experience.
Goals
As a data scientist working with the BUX X team, I tried to help out with designing an algorithm that would improve the experience. Some of the goals we had in mind when coming up with potential solutions:
- the new home screen should contain aggregated information about what is currently happening in the app — what products are being traded, what is “hot”
- additionally, the new home screen should provide insights about market sentiment, whether the expectations are bullish or bearish
- at the same time, the user cannot be overwhelmed by the number of cards at the home screen
- we want to keep the recent history available (up to the last 150 cards), so the users can see what recently happened on the markets (proxied by the trading activities of BUX users)
- the algorithm must be fast enough to work in a live (real-time) environment
The new approach
We tested out a few different algorithms, but ultimately we settled for a relatively simple one, based on moving averages and standard deviations. Let’s break it down into smaller steps.
On a daily basis, we calculate the average (and standard deviation) number of positions opened per product and per hour. We calculate the metrics on the product level, as the volume of trades varies a lot between the assets. And in BUX X we currently offer ~300 products, and the number keeps growing!).
If in any given 60 minute period there were more positions of a certain product opened than the corresponding average + 1 standard deviation, we publish a card. Each card aggregates the number of trades and shows the sentiment as the percentage of trades going long or short. Additionally, they also show the time frame in which the trades happened - since the first trade in the moving window.
We also implemented a cooldown period, as we did not want to repeatedly post similar cards when a large number of trades is still within the moving window. After showing the card, we do not present another card for the same product again for one hour. Unless, in any of the following one 1-hour windows, the next threshold was exceeded (either 2 or 3 standard deviations above the mean). If that was the case, we show another card (with updated numbers) and the cooldown period restarts to 1 hour. In the case when the 3rd standard deviation threshold was breached (either 2 or 3 cards presented in total), there is a mandatory cooldown period of 1 hour and the threshold is also set again to 1.
Prototyping the new home screen
Before asking the Design and Dev Teams to implement the new feature, we wanted to quickly prototype it. This way, the Product Team could inspect how the feature would look in practice (well, the prototype is not that nice and shiny, but the logic behind it is the same) and decide if they were happy with moving it to the next steps. By following such an approach, the other teams do not need to waste their time on developing features that in the end might not be used in the final version of the app. And before that, a lot of work had to happen: designing the new look of the cards, front- and back-end development, extensive tests on the beta environment and, finally, deploying to the live version of the app.
For prototyping the new feature we used a combination of SQL, Python, and Tableau. We started by extracting a snapshot of trade-related information from our data warehouse. Then, we wrote a Python script recreating how the new home screen would have looked like — at what time certain cards would appear and what information they contained.
By parametrizing the script, we were able to run it multiple times to quickly create different scenarios: weekly/monthly metrics, how many standard deviations above the mean are considered to be triggers, duration of the cooldown period, etc. As the prototype did not have to run in real-time, we stored all the resulting information in the data warehouse, available for a quick lookup.
On top of that, we built a dashboard in Tableau. By selecting appropriate filters, the Product Team could easily compare how the home screen would look like for any day of the considered half a year period.
The recreated feed already provided a lot of useful information about how the home screen would have looked like in the past — how crowded and what kind of information dominated the feed. However, it is also important to have a more general overview. We started by looking at the number of cards over the first five months of 2019.
By implementing the new algorithm, the user would see around 400 cards per day. This is still a lot, but significantly less than with the old implementation (where the number was in thousands). We can quickly arrive at three conclusions:
- The number of cards generated by the algorithm is fluctuating over time and depends on the volatility of the market.
- With each threshold (1/2/3 standard deviations above the mean), we displayed fewer cards.
- The vast majority of the cards come from the star traders (the best performing users in a given week).
The reason for the third point is that the star traders are not aggregated and we display each of their trades. The feedback we received from the users indicated that they eagerly follow their favorite star traders and this is a feature they would like to keep.
Additionally, we looked at the distribution of cards over an average day. The two peaks happen around the opening hours of the European and American markets.
A/B testing the performance
The second part of the process happened after the feature was already built and tested. The new version of the home screen was released to a fraction of live users in order to A/B test the performance. The main metrics, based on which we evaluated the success of the endeavor, were related to the number of trades opened from the home screen. We hypothesized that given a cleaner overview of what is happening on the market, together with a measure of market sentiment, our users would open more trades from that particular screen.
Additionally, we evaluated metrics such as retention and social activity in order to monitor the general usage of the app and make sure that we inadvertently did not break something else.
After evaluating the results (following the best practices mentioned in [1]), both in terms of numerical comparison and statistical tests, we came to the conclusion that the new version of the home screen was performing better, both for new and existing users. We introduced the additional split to see if the behavior was different for users who already experienced the previous version of the home screen and those who never saw it before.
Conclusions
The results of the project can already be seen in the latest version of the app. We hope that introducing this feature will help our users in having a better overview of the current market dynamics and making more informed trading decisions.
In the image below, you can see the new home screen. The left one shows the aggregated cards on the new home screen, the right one gives a preview of what happens when you actually click on the card. By doing so, you can inspect some additional information such as the current price, the percentage change and a shortcut to opening a trade :)
I would like to thank all the people working on the project, it was a pleasure. From my side, I must say that it makes me proud to see that as a data scientist I can have an impact on the product and help the company make informed decisions.
If you would like to join our amazing team, please have a look at our career website and reach out in case of any questions :)
References
[1] https://www.evanmiller.org/how-not-to-run-an-ab-test.html
