How to set your goals to level up your Experimentation Program

How do you take an experimentation program from good to best in class? I am the Program Manager for the experimentation program at Optimizely on our Product organization. In the last year, we’ve greatly expanded our experimentation practice and now I am working more specifically to make our program best in class.

As part of making our program best in class and myself a best in class program manager, I wanted to map out concrete items I could commit to this quarter to work toward this larger vision. Additionally, I felt it was important to commit to creating excellence within our experimentation program via my OKRs to commit myself, and others, to this goal. This not only commits me to following through on these goals, but also helps protect my time against other initiatives that may come my way later in the quarter.

For starters, I needed to define my north star goals to provide me with structure as I was thinking about the more tangible actions and initiatives to commit to this quarter. We were already doing a fair amount of experimentation and utilizing rollouts within our product organization, but I wanted to think about how we could elevate our program to be best in class. To that end, here are the north star goals I developed along with my executive sponsor:

• Create a best in class Experimentation Program (multi-quarter)
• Become a best in class Experimentation Program Manager (multi-quarter)
• Understand blockers for Product Managers, make running an experiment easier

The above are my north stars. Here are a few other north star metrics we’ve seen our customers leverage:

• Create and expand a dedicated experimentation team
• Co-lead quarterly product planning meetings
• Create case for expanding into Full Stack experimentation

Now, I’m fortunate that I already have a bought-in executive sponsor in our SVP of Product and our product teams already have an active interest in experimentation and experience with running experiments.

So, I come to this already having the following:

• An executive sponsor who can help with organizational blockers
• Baseline organizational buy-in
• Teams trained to design and build effective experiments
• A program owner who drives the organization forward and socializes what’s been learned

If I did not have the above pieces in place, I would focus on these areas first as our program is working toward experimentation excellence versus just getting off the ground. If you’re just getting your program off the ground, I’d recommend start with some of the resources on the Optimizely Knowledge Base [here].

Now that I had my north star goals in mind for creating a world class experimentation program, I wanted to think about how to create specific key results that would track toward these goals this quarter, which are outlined below and categorized with their north star goal.

Create a best-in-class experimentation program

To work toward a best-in-class experimentation program this quarter, I’m focused on creating more visibility within the organization at all levels. That means presenting at large company meetings, creating a more detailed leaderboard for experimentation (both physical and digital to involve remote offices and teams) and presenting externally with my executive sponsor. We are also focusing on higher quality experiments and committing each product team to a specific number. Additionally, we are committing to all features being a rollout, versus a hard number of rollouts which we found was not the right measure.

Visibility

• Webinar on Executive Sponsorship and Program Management with Claire Vo, SVP of Product [here]
• Update to Executive Team x 2
• Present at Company All Hands x 2
• Present at Engineering Weekly x 2
• Quarterly Company-wide email
• Build a physical experimentation leaderboard
• Build a digital experimentation leaderboard

High Quality Experiments

• Run 6 Growth Experiments, 2 per Product Team
• 100% of Features are Rollouts

Become a best in class Experimentation Program Manager

To drive more mature thinking around experimentation, I’m consistently meeting with my amazing Optimizely account team for feedback and brainstorming around the program. They also share articles, documentation, do trainings and share insights from other experimentation teams, which is hugely helpful. Additionally, taking the certification ensures all my own knowledge is up to date and commits me to utilizing the great resources on the knowledge base. Finally, building a monitoring dashboard helps me better monitor all activity within the program.

• Pass all Optimizely X Certifications
• Meet with my Customer Success Manager and Strategy Consultant for bi-weekly check-in
• Build experiment monitoring dashboard

Understand blockers for Product Managers, make running an experiment easier

While our Product team is very invested in experimentation, they have the same struggles that many teams have: lots of projects and priorities. So, while we are running experiments, making this process as streamlined and baked into the product development process across all product teams is important. To that end, I’m running user research sessions to better understand how we can make experimentation easier for our Product Managers and Designers. I hope to share this in a later blog post once I’ve synthesized my findings.

To help increase efficiency, there are specific types of re-usable components we still need to be built with our new branding such modals. Building these using Extensions (aka templates with easily modifiable components) is a great way to streamline running experiments, especially for less technical folks. I’ll be working on these during our company hackweek.

Finally, there are some ongoing efforts that will help with making experimentation easier, that need to be re-visited almost every quarter. Documentation which reviews our overall program structure and how to run an experiment on the product team. Archiving any unused components within our Optimizely instance and having a current, agreed upon process for adding new components that are verified by the engineering team.

• Map out additional Extensions that are needed
• Update our internal documentation
• Clean up Product’s Optimizely Web Instance
• Update internal documentation with nomenclature and approval process defined for creating new components with Product’s Web instance
• Conduct User Research with Product Managers and Designers about Experimentation Program

Overall, my goal is to kick-off the new year strong — new year, new me! Or, rather, new experimentation program. I hope this helps you think about what key results you could commit to when upleveling your experimentation program.

What key results are you committing to this quarter? Would love to see them in the comments below or tweet them to me @bexcitement.