Journey to Find Key Metric for Academic Search Engine
Have you ever searched a paper?
Researcher’s fate is to search papers to explore precedent research. Academic search engines, such as Google Scholar, SemanticScholar and Scinapse, help the research by providing publications from journal databases.
Scinapse is the fastest growing academic search engine. We currently focus on developing the paper exploration experience.
As a product manager, I have to define these three chapters:
Product Manager should be able to answer:
1. How customers use the product?
2. What to build, What not to build?
3. How the product make customers use Scinapse?
To answer the questions, product managers should have their own decision framework. Metric is a great approach for a decision framework, and key metric would be the main role. So let’s see metrics I explored to find a key metric.
Metric: The Growth of Usage Count and Ratio
I thought that the growth of usage count and user ratio are important as a basic. So I made the chart below to watch the daily changes of usage count.
This chart gives an insight for the changes of daily usage and the visual effect of feature update. As you can see, at the end of 2018, we updated a feature and the usage count grew from then.
But it was not enough, because I could not see who the users are. Are they returning users? or just new users? If they are all new users, the visual growth might not be happy as a product manager. Because it means that nobody use the feature again.
So I needed to distinguish users by categories.
Metric: Categorized Users
By distinguishing users by categories, product managers are able to set a sharp strategy. Firstly I made a chart, categorizing users by a signed up date.
By categorizing like above, I could see a fact. Most users were new users. OMG. So I realized that two more facts. 1) The feature is working for new users as an attractiveness of our product. 2) I need to focus on catching the old users.
In converting the table to chart, I aggregated all old user categories as a returning users, because the table let me realize that the keen segmentations are not helping me.
This chart gives me an insight to see the growth of feature users and that of user categories.
Specifically, the chart only for returning count provides the result of effort to catch old users. But these charts gave me a question. “Do the returning users love our product?”
Metric: Heavy Users
I just wanted to make sure that there’re hundreds of real fans who are willing to take the effort to refer our product. Let’s call them HeavyUsers.
To find HeavyUsers, I needed to set the standard to define the weight of HeavyUsers. The standards might different in the point of views, so I set that by features or experience flows.
At first, I wanted to scale how heavy they are. I got HeavyUsers list with quick standards, and had made the sense of proper standards. Finally I charted by the standards for each features.
The growth of HeavyUsers was really helpful to be proud of building product. We could believe that we’re building a product which people love.
The discovery of the existence of heavy users had raised the curiosity of why they became HeavyUsers. So I tracked their usage patterns.
Metric: The Usage Patterns of HeavyUsers
The usage log of HeavyUsers gives an information how our product can be used. I believed that bounced users or exited users give the information of the reason why people leave our product. It does provide, but sometimes HeavyUser would be better for the clue. Bounced users would not have a will to use our product in depth, so they will leave even if we did something. But HeavyUsers’ log have patterns of usage or stop.
I researched the log line by line to 3,000 users. Then I founded some usage patterns and picked the best case patterns.
These give me an insight to what we should build. A group of HeavyUser already jumped our our intention and created their own way. That inspired me. Then finally I got a proper metric to cover all ongoing projects.
Metric: The Primary Objective of A Tool
A tool is used to operate better. Easier, Faster or more Successful are the primary objectives of a tool.
Academic Search Engine is also a tool, to search publications. It means that ‘Easier search’, ‘Faster search’ and ‘Successful search’ are the primary which that have to provide. Thanks to the enlightenment, the key indicators could easily be found.
Exploration success duration and success rate were metrics I found as key metrics. All the efforts to refine our product might lead to the success.
How was my journey? and how are your journey. Please comment and share your journey too. I also want to learn from your experience.
Thank you for reading,
and be happy with product management :)
