Theory of Hypothesis Testing
Check Eligibility vs. Apply Now
What is a great hypothesis?
If there was to be a Maslow’s heirarchy of hypothesis, my guess would be:
(1) Learning + Serial Impact (bonus: cross funnel) → Top of the pyramid
(2) Learning + Serial Impact - Current funnel Impact
(3) Learning + Current funnel Impact
(4) Learning + No Impact
(5) No Learning → Bottom of the pyramid
It may be fair to assume that product managers/ owners grow their way up the pyramid and consistent “top of pyramid” hypothesis generation could be a hallmark of a great product manager. Or appropriately, a self-actualized product manager
Here is an example of a “top of pyramid” hypothesis we managed at my current company, Lendingkart.com
Hypothesis
Users seek a light weighted approach toward understanding what specific loan terms to expect
This means before any activity a user can undertake in the loan process, he must be made aware of his specific eligibility. While this may seem obvious in hindsight, taking loan is equivalent to buying a shirt was not obvious i.e. user will mostly do anything to get a loan was the predominant belief
How do we possibly know this?
(1) User research: Lendingkart sees two personas of users — Type O (Orthodox) & Type T (Tech Savvy)
Type Os are married with kids and have been frequently rejected by banks and financial institutes for loans. Hence they tend to be extremely skeptical to share any data and seek eligibility before progressing to apply
Type Ts on the other hand are aspirational youngsters who freely explore multiple sources without going deep into any process. Spray & pray. Hence, they seek eligibility before progressing to apply
(2) Data: This is strongly supported by how the conversion funnel shoots up post providing his loan terms. >50% to <25%
Product Manifestation
AB: Home page “Check Eligibility” vs. “Apply Now” button
Why? If the hypothesis is true, then it is very likely that it is the most important first action the user will take upon landing on Lendingkart
Sample: Only new users to avoid surprise in experience
Statistics: Sample Size = 3000. Alpha = 99%. 1-Beta = 95%
Business Impact
The Click-through-probability went up from a sub-10% to a mid-30% and it all took 0.5 man days of tech effort
Learning & Serial Impact
Strong validation of hypothesis opened up avenues to further optimize the “Start of Journey” experience. Few small steps taken include:
(1) Replace a registration process with Check Eligibility process. Eligiblity = Reject/ select
(2) EMI calculator based onboarding and exit popup. Eligibility = Reject/ select, Estimated EMI/ IRR/ Tenure
(3) Pre-approval loan offer based on PAN only (Government ID). Eligibility = Loan offer
