How a Failed A/B Test Turned Me From an Intern Into a Major Feature Developer
I Came to Photomath in November 2018. More than 3 years ago, at the start of my 3rd year of college.
At the time I knew no Objective-C, a little bit of Swift, some iOS development techniques and a whole lot about copying the first Stack Overflow answer that solves my problem.
Today, I am an author of one of our latest major features bumping Photomath to version 8 so let me tell you how a failed A/B test pushed my personal growth at this company into a higher gear.
When I first came here the whole student onboarding process was new as the whole company counted ~90 employees. Besides me there was just one other student (who is now at Google btw 😄).
I still remember my first task.
It was to change boolean string formatting for a single label. In an internal debug menu nonetheless. That was exactly what I did. I mean it’s pretty straightforward.
Still, I received a single comment suggesting a single ternary if operator instead:
In my book, that’s my first “production level code”. It ended up in a real application, right?
Taking baby steps I was assigned to somewhat bigger and more demanding tasks in the coming months (and years).
As the company grew it slowly started shifting focus to, well, growth itself.
Especially when the pandemic hit we started doing a lot of A/B testing.
It turned out to be a great strategy as we saw over 2.5x increase in Plus members since the start of COVID and a total of 10x growth in 2020.
A/B Tests At Photomath
AB tests are a sure found way to decide the direction in which to take an app or a single feature. It’s based around a single or multiple hypothesis alongside a control group. Meaning that a certain percentage of users is split into multiple “buckets”, one being measured as a control group without any changes and the rest being experiment hypotheses or experiments.
Aside from changes, experiment definition includes the measurement which is meant to conclude what experiment variant is most accepted by users. For example, user retention or trial to paid conversions.
While we’ve written about A/B tests at Photomath, here’s a quick rundown.
Most of the time, there are multiple AB tests defined on a released app in which any given user can be a part of. Oftentimes, different AB tests may overlap in subject meaning that these tests need to be explicitly set as mutually exclusive as they may influence each others results. All of this involves a lot of juggling by the developer and a certain pitfall for the product manager or anyone defining a test.
Aside from the aforementioned type of AB tests, certain “bigger” and more impactful AB tests have been ran on Photomath 7 which greatly influenced the look and feel of Photomath 8
We use Firebase A/B Testing as it enables us to define users with a number of filters such as country, state, language, platform and more. On Experiment completion Firebase provides various statistics to back up experiment results.
Photomath has the privilege of running an extensive number of AB tests due to our tens of millions of active users.
The Test Called Balthazar
Zagreb, a Croatian trademark
Exact AB test I will discuss here will be referred to as Balthazar. Now, the purpose of the vast majority of AB tests is to determine how it will impact users and how they will react back. Purpose of Balthazar was to also determine how will the whole company react to it on a bigger, semi-realistic scale.
With development time spanning across multiple months and across all engineering teams at the time, design, marketing and content teams it was no small feat.
My experience and attachment with working on the feature was particular as it was my first solo “major” feature on the iOS team as I was still an intern (in my final year of college). I was given an opportunity and trust to work on something long term and implement several features which were new to everyone on the app team. Naturally, I grew emotionally attached to the feature having spent countless hours perfecting certain aspects of it till I was pleased.
Results
Unfortunately, after some time, the product team decided to take a different approach than the one in Balthazar. Balthazar was gradually turned off and the whole thing was scrapped in the coming months.
At first, it may seem as a major misdirection to scrape such a big feature, it did to me at first, especially as I grew attached to the feature, rooting for its success.
But there are several important points to make here:
- Testing the feature on a small percentage of users saved the company an enormous amount of time and resources is it wasn’t viable to maintain
- Releasing it as an AB test meant turning it off was not much more difficult than turning off the remote config flag
- Various bits and pieces of the feature can now be found within Photomath (but I won’t reveal where 🤫)
Back to my personal development
We usually try to make a major build bump that closely matches the start of a new school year so in may / june we started looking at various designs and prototypes.
One prototype was particularly interesting for me as it used some refinements of a feature I built for Balthazar.
I was then appointed to work long term on that feature alongside an Android colleague with whom I also worked on Balthazar.
Most of this was a first for me, including:
- Having frequent meetings and kickoffs regarding the feature
- Making internal builds and discussing them with the product
- Working with an external focus group discussing the usability
and more.
In the middle of the development process I formally transitioned from an Intern to a full fledged iOS developer in a now larger team counting 5 whole people.
After numerous setbacks, changes and updates we finally released Photomath 8 on January 21.
So why am I telling you all this?
Especially when coming with little to no real experience it’s not about the onboarding speed and fast personal development as it’s about quality long term results.
Looking back, each one of those “tiny” tasks played a role in me ending up working on Balthazar and ultimately on Photomath 8 with me owning a major feature and earning a tremendous amount of knowledge and experience in the process.
There’s five of us and we are looking for a sixth colleague so we can’t be counted using a single hand.
If you like what you’ve read learn more about #LifeAtPhotomath and check out our job postings: https://careers.photomath.com/
