A New Feature For Dropbox Paper And Its Experimentation
Here’s the story. One day I was in the office and ran into my colleague David in the hallway. Since David always had great ideas and a positive attitude, I decided to bring up to him an idea that was hovering over my head for days ——
The Calendar View for Dropbox Paper
As Dropbox users might have noticed, Dropbox does not provide users the option of having a grid view for their files. As expected, Paper, though armed with powerful full-text search, does not support grid view either. It sorts docs by recency, just as your Dropbox does to your files.
I’m positive that the product managers at Dropbox had already deliberately considered it and decided not to roll out the grid view feature. However as a user, I constantly find myself subconsciously trying to find the view-switch button every time I land on the file page. I also found similar sentiment on the Dropbox forum where users were asking for a grid view to display their photos.
Instead of coming up with the grid view that other document repositories have, I attempted to find something more revolutionary. The idea hit me when I was using Buffer, a scheduling tool for social media postings. Everyone is used to checking meetings or coffee chats in a calendar. Why not view docs in the same way?
As to what I imagined, it would look something like this:
The mockup is not 100% perfect. But you get the taste. By clicking the “Calendar View” on the tab bar, you would be able to view all the docs you have in your possession inside a nifty calendar. The calendar serves as an arbitrary time marker for your docs. You can choose “date of creation” or “last modified” to manage their appearances.
When your cursor hovers over a doc slot, you would be able to view the details of the doc, such as how many people this doc is shared with, and how many pictures, videos, or lines of code are included in the doc. You could even see how many emojis were used during this fun team working process.
When the cursor floats in the blank area on the calendar, it would show a transparent new doc box suggesting you that you could click and create a new doc, similar to creating and sharing a new event within your calendar.
You haven’t forgotten about my teammate Dave, have you? Dave is an awesome guy who has an inborn talent of encouraging people around him. So he said:
We called up our teammates and planned out the workflow to test the new idea. We basically followed an experimentation approach adopted from Twitter (source: The What and Why of Product Experimentation at Twitter). The experimentation lifecycle is pretty similar across most tech companies. I’m using Twitter’s approach here because it’s explicit.
I will put this lifecycle into perspective, giving the context of the calendar view feature experimentation. (I omitted the qualitative research since the emphasis of this article was on experimentation. But qualitative research is no less important than experimentation and normally is before quantitive analysis.)
Break It Down
Build A Hypothesis
Our hypothesis is, by adding the calendar view for Paper feature, we can significantly change user’s behavior to increase their usage of Paper.
Define Success Metrics
Now we have our hypothesis. But what defines success for this experiment? Who exactly are the “users”? How significant is “significant”? How do we measure “increase their usage of Paper”? When we can confidently declare that a user’s behavior has been “changed”?
We need to put these ambiguous words into clear, specific metrics to make sure that everyone is on the same page and is looking in the same direction.
1. Who is eligible for this experiment?
For now, the Dropbox Paper beta is a web-only app that is only available in English. Since not every Dropbox user has access to Paper, the user base is fairly small.
To increase statistical power and reach statistical significance, we want to include all Paper users in this experiment and split them into two groups. We would randomly assign 50% of the users to the test group (feature-switch on), while the other 50% remaining as the control group (feature-switch off).
2. What metrics should be tracked?
The first thought that comes to our minds is the metrics that are directly related to the experiment hypothesis, which are the metrics that measure the user’s usage of Paper. The key actions users take within Paper and the usage increases we want to achieve are:
- Access(and edit) existing docs (+28.2%)
- Create new docs (+22.2%)
- Share docs with others (+8.3%)
Therefore, we need to track how many docs users create, access and share, on average, every week for the control and test group, respectively. We then look at the corresponding metrics to see if there’s a significant increase in these three key actions in the test group, as compared with control group.
We should see something like this:
Besides that, we also want to know how users interact with the feature as they dig deeper into it. Here are some metrics we could track:
- How many % of users click on the calendar view feature? —— After users click on the feature, we know no more information if no further interactions are committed. So we may also want to look at how long users spend in the calendar view to get a better sense.
- Among those who clicked on the feature, how many % of them clicked a second time? A third time? A forth? —— Measure it in another way, how often do users click on the calendar view feature?
- Among those who clicked on the feature, how many % of them accessed a doc from within the calendar view?
- Among those who clicked on the feature, how many % of them created a doc within the calendar view? —— This is the most important metric we’re looking at. Don’t forget that our ultimate purpose is to drive up the usage of Paper.
- Among those who created docs within the calendar view, how many % of them shared a doc within the calendar?
These metrics are absolute percentages or numbers that should be compared with the corresponding metrics of other views of Paper docs, such as the “Recents” view or “Created by Me” view, so that we can define what are good (or bad) numbers.
Considering that the “Recents” view is the landing page when users log in to Paper, we would expect that the numbers for the calendar view to be considerably less than the “Recents” view, while comparable to the second pages such as “Created by Me” or “Shared with Me”.
3. What are the goals?
Although the defined metrics do not strictly follow AARRR, we could still use a funnel to plan our metrics and goals. I am going to assume that users who requested access to Paper are active users who use Dropbox on a daily basis. But I’m taking a humble guess here:
After the most need-to-be-thought-out part of the experimentation, we can proceed to execute the main body:
- Implement the new feature
- Instrument appropriate logging
- Perform sanity checks and health monitoring
—— to make sure that would not happen.
Learn (and Communicate)
What are the actionable insights we can find from this experimentation? A lot of data analyses go here. Based on what we’ve planned beforehand, we first would look at the outcomes of the metrics at an aggregated level.
We can also combine the funnel analysis with cohort analysis, if, given enough users go into each cohort, so that we can take into consideration the customer lifecycle and look at user engagement over time.
We don’t have the real data. But after having a deep dive into the data, we may find ourselves asking questions like these:
- The click through rate of the calendar view feature is low. Is it because users saw it within Paper but were not interested, or because the feature is not easy to find? If it is the latter, how can we improve it?
- What are the most popular time slots for users to create docs within one week span, and why?
- As soon as a user accesses docs 10+ times per week for 8+ weeks, she’s 80% more likely to stick with the new feature and foster a stable user behavior. How can we prolong users’ action of accessing docs 10+ times per week for 8 weeks?
To answer these questions, ad-hoc analyses are likely to be conducted. We could even propose hypotheses to those questions and design new experimentations.
Last but not the least, clearly and timely communicate the insights with teams. They include but are not limited to product managers, engineers, designers, analysts, marketers and other stakeholders of the product.
Ship (Or Not?)
After all the delicate setup, testing and examination of the numbers, we finally felt we validated the hypothesis that the new feature did in fact increase the usage of Paper. So we decided to ship the new feature. Woohoo!
—— Well, that’s just my dream. Since this is all hypothetical, we don’t have enough data to make the decision. However, in the real scenario, I would expect this feature to not be shipped considering Paper is still in the beta phase.
As Paper is going to be released to a broader audience and optimized to be more stable, new features like calendar view would more likely to be added after delicate qualitative research and user feedback.
Wait, there’s more (cycle).
The experimentation mentioned above is not a one-time shot. It’s a cycle that would go on and on, where we generate more hypotheses for more improvements, incorporating new insights from the experiment and continuously improving the product, until it reaches the point where we feel that the product has matured.
As the user base keeps growing larger and larger, we could do A/B testings on some variations of the current version to optimize the feature. As for the analysis, we could further apply user segmentation or mix-and-match several analysis methods, such as trend analysis, statistical analysis, regression models, etc.
A/B testings on Small Features
Let’s suppose we rolled out the calendar view feature and started to optimize it, there’s a lot more to be imagined. I’m just throwing some random ideas here:
- The placement of the “Calendar View” button
- Display different designs of the “Create” button
- Show a time sidebar instead of addressing accurate time in each doc slot
- Span docs that last for a period of time, just as your 3-day hackathon span on your calendar
- Create a separated bucket from calendar for docs that don’t really have a time associated with them
We would split the test group into the above mentioned treatment buckets, each with one variable to be tested. With multivariate experiments running at the same time, we should prepare ourselves to see something like this:
For user segmentation, we may want to avoid demographic but to use behavioral metrics. There are many dimensions we could use to segment users. Here are some examples:
- Subscription: Free, Pro, Business
- User activities: Inactive, active (base on the user’s activity log)
- User types: business, family-and-friends, independent users
User segmentation is only valuable when different segments of users behave very differently. Of the first batch of users who have access to the Paper beta, it’s more likely that they are active users of Dropbox and early adopters who have similar user behaviors. However, as Paper roll out to larger audience, it’s reasonable to start segmenting users.
A Huge Leap
From a cloud storage company to a collaboration company, Dropbox is taking a huge leap. What does it mean? It means Dropbox is moving from the home for your files to the home for your ideas and knowledge.
While Dropbox is the users’ go-to option to store, Paper will be the catalyst for users to create. —— Let’s keep our eyes widely open.
I’m not a Dropbox employee and I’m not associated with Dropbox in any way. The “my colleague Dave and I” plot was made up. It was just a fun writing experience to play around with the awesome product.
The conclusions were arbitrary due to the lack of data. Pitfalls and tons of details were not covered due to the limitation of length. However, if my way of thinking piques your interest, feed me some real data. I would make sense out of it.
Many thanks to Brian Eng, product manager at Math Camp, who provided many constructional advices along the way, and David Zhang, software engineer at Dropbox, who helped critique my thoughts of the calendar view feature and brought up some aspects I would have otherwise neglected. Also shoutout to my friend Jeffrey Chen, software engineer intern at Yelp, who proofread everything I wrote in English to make sure I, as an international, didn’t make a fool of myself.
It is my first time writing about product and analytics. Faults were very likely made in this article. So comment below or DM me on Twitter. I’m happy to learn from anyone. ☺️