How Thinkific’s Growth Team Experimented on our Homepage
Who am I, what is this, and why should you even care?
My name is Robert, and I joined Thinkific in May 2021 as a Growth Engineer on the team affectionately known as Spider-Man (because we work on the web…get it?…no?…let’s move on). The Growth team consists of several sub-teams that run experiments in different areas with the overall goal of growing our customer base. My job as an Engineer is to run experiments that focus on getting a visitor to sign up.
Experimentation is a big part of our culture which is why we have a team dedicated to this area. It helps us learn and grow by giving us the space to try — and sometimes break — new things. We are fanatical about our customers’ success in order to deliver a wonderful experience; there’s no better way to achieve this than experimentation.
In this post, I’ll walk through an experiment we ran on the homepage, which is probably the most important page for any business. I’ll talk about what initiated this, some challenges I ran into, and the results. I learned a lot from this experiment and I’m excited to share it with you!
The Stack
Our site is currently run on WordPress, and while it may not be my favourite CMS option, it fulfills the requirements that we need to run our tests. The current experimentation stack is a combination of Google Optimize and Mixpanel. Optimize allows us to run simple A/B tests, while Mixpanel allows us to analyze how users are interacting with the page.
There are plans for the team to move away from WordPress and onto a headless CMS, which would allow us to fix any gaps we previously had in our tech stack that affect our experiments such as site speed. We will also be looking into alternative testing solutions such as Optimizely and VWO, which will allow us to conduct more server-side experiments. We are constantly looking at ways to improve our infrastructure and will be doing so in the near future.
For this experiment, visitors coming to www.thinkific.com were randomly divided into two groups — control and test, where the control group is sent to the original page design, and the test group is sent to the new design. We tracked two main events for the homepage, impressions and link clicks. Impression events are fired when a user views a page that contains the test or control, while the clicks events are fired when they click the link, which allows us to gather sign-up data.
Why now?
2021 brought big changes to Thinkific — we grew a ton as a company, both in terms of users and team members. Growth as a department has existed at Thinkific for a while, but one of the changes from last year was a new subteam dedicated to managing the website and all of the tools associated with it. The team was formed in late 2021 and what better way to end the year than a new homepage.
The previous homepage design was built sometime in 2018 and though it fulfilled its purpose, it was due for a bit of a refresh, technologically and visually. More importantly, we wanted to increase visitor engagement.
Visually, the page did not scale sufficiently on both large and smaller screens, information was out of date, and some links were not working.
The page was incredibly slow because it was very image and video-heavy and had inefficient script loading, in addition to any page loading issues that naturally came along with WordPress. We previously featured customers’ videos using Wistia, which would break by taking you off of our site to play the video (or not play the video at all).
With the new homepage, I wanted to also increase our Core Web Vitals — which are Google’s page experience metrics that can affect page ranking in search results. At the time, ours were hovering around 65–69, while our target was to be above 90.
In addition to the potential tech and visual changes, Black Friday was coming up — this was an important time for sales and our Marketing team would be running promotional campaigns that would drive more visitors to our site. With this in mind, the idea was to run the experiment, analyze the results, and make a decision before Black Friday, as we knew it would bring lots of opportunities to increase our conversions. Since we had not run any homepage experiments for the year, and because this would be a large change, we fully expected success.
This new design would bring in the change we wanted.
Problems
Let’s talk about one of the problems we encountered.
As a reminder, we were a newly formed team that inherited the already-built website — that meant there were some complexities to the site we were not aware of given the size of the website and the annoyances of WordPress.
The Expert Marketplace is an important area of our site that highlights our Thinkific Experts — specialists that help our users build their businesses. This page was built with a search engine that would query all of the experts and display the results. The issue was that it generated the results from the homepage, therefore it would naturally break since anyone in the test group would be sent to a different page. We were unaware of this until we were several weeks into the experiment. This eventually resulted in the team building a brand new Expert Marketplace, with a better search function and new design.
Results
The goal was to increase visitor signups and I had high hopes for this going in; everything improved both technologically and visually, so there was no doubt in my mind that this would be successful. However, the data showed
no significant change between the test and control at the end of the experiment, which was a bit of a surprise. Through discussions with other team members, we concluded that the experiment simply was not big enough to make the impact we expected. We needed to go a bit deeper in our thinking and do things that make more of an impression on our audience.
You may notice that if you visit Thinkific’s site today, you’ll see the new design we came up with. Our experiment failed, so why did we release the page anyway?
While the initial goal wasn’t achieved, there were some pretty big wins. I fixed a large number of issues with the previous page, the biggest one being page speed. As of writing, our Core Vitals score is at an 85 which is a significant improvement from the 69 it was before. I also fixed a lot of script loading issues, removed the videos, and further optimized our images.
Technical improvements aside, the biggest win came from a metric we were not initially looking at. With our experiments, we track deep within the funnel, beyond just visitor signups, in order to see what those signups lead to. Working with our team’s Data Analyst, we found that the test group had a 97% higher probability of trying a paid plan than the control, which was pleasantly unexpected. Because of this, we released the page to all visitors coming to thinkific.com.
Learnings
This was one of the first big experiments that I was able to work on at Thinkific, and I learned quite a lot from it. A few of the lessons I took away are:
- A homepage experiment that consists of mainly visual changes may not move the needle enough to be impactful; we need to take bigger chances if we hope to see better results. Additionally, when conducting an experiment, it is important to verify that other areas of the site do not unintentionally break.
- We don’t simply track when a visitor to our site signs up, we can identify their activity further down the line. This is an incredibly important opportunity to see how people interact beyond the obvious and is a good reminder to keep an eye out for wins and successes we didn’t initially have in mind.
- The need to move to a newer site and experiment stack is more important than ever; the flexibility would allow us to use other testing technologies would be hugely beneficial and would help us better identify any breaking changes that may inevitably occur.
I worked with so many amazing people on this project, ranging from design to engineering to data, and more. I would not be able to do this without their help and support. This experience has given me a ton of information that has helped improve the quality and speed of how we deliver experiments.
If you would like to know more about our many experiments or would like to join our amazing team, take a look at our Careers page or reach out to me directly at robert@thinkific.com.
