Creating a Beta Program

Challenges and benefits of a long-term customer engagement program

What do we do when faced with an unknown? As designers and researchers, we jump straight into that vortex, equipped with a number of hypotheses to validate and a list of questions to ask. But what if we don’t know what types of questions to ask?

My team faced this conundrum earlier this year, in helping our largest Enterprise Customers on their journey to the Atlassian Cloud. Initially, the space seemed well known… but moving from a self-managed bare metal (and most of the time, heavily customized) on-premise environment to Cloud is a complex and multi-facet transition. We were well prepared for regular-sized companies and their needs, but Enterprise customers? Not really.

Naturally, we did “regular” research: we went through available insights, interviewed a couple of Solution Partners and customers, but it was not enough for our purposes. We could see the customers were going on the journey with old maps and tools not built for their size.

In June 2021 we realized we need to unpack what we don’t know about the situation our customers were in… and that we needed something more than a burst of case-by-case interviews.

And that’s how the Beta Program was born.

The program was born out of necessity, as we were seriously lacking customer exposure and continuity, but we were inspired through our Customer Advisory Board (CAB) sessions organized by key stakeholders.

We invited several Enterprise customers to partner with us and design the journey together. Our goal was not only to build a regular cadence but also to create a bond. We offered early access to features in development and the ability to influence our roadmap. In return, we received first-hand insights and unique customer perspectives. Monthly meetings showed us how our customers’ migration journey was progressing and whether the tools we were developing were truly helping.

As with every project, things didn’t go according to the plan, especially at the very start of the endeavor. We addressed the challenges as we went, shaping the program along the way.

What didn’t work: Chaotic beginnings and empty agendas.

We wanted to run before we could walk - we wanted to get in front of customers ASAP, without much preparation and just a bunch of ideas. We had good kick-off meetings with several interested customers… and then nothing. We had nothing substantial to put on the agenda and nobody to keep track of where we were. This forced us to do several last-minute scrambles during the first month of the program when we were desperately looking for topics to present.

What works 1: Tight schedules and a dedicated driver

It soon became obvious that we needed a well-organized person to keep tabs on the program, make sure the meetings are scheduled and agendas were prepared in a timely manner.

What works 2: Going beyond conversation

Our initial ideas for the agenda were simple: exploratory user interviews, feature demos, and user testing… but soon we pivoted to hands-on activities. We started using Mural to quickly map out the customer’s setup which evolved into illustrating various customer setups, from migration strategy across time to necessary connections between products.

What didn’t work: Low attendance and misaligned topics

Even with an organized schedule and prepared agendas, we encountered low attendance. Enterprises tend to be more susceptible to changes and internal movements, meetings get canceled or rescheduled at the last minute — even if the attendees have confirmed, a super urgent issue on the customer side can mean a delay or sudden cancelation of a session.

We also learned that sometimes we presented a topic that was not interesting to the customer. The confirmed meeting attendees might come from a different business area to the one that confirmed, or the particular customer had no need for a given feature.

What works: Have plan B (and C)

On encountering a misaligned meeting topic, we would have a more generic presentation or a demo ready, just in case. Something like a high-level overview of a problem space or a whiteboard activity. Anything that would benefit both the customer and us.

What works: Geo-distributed team

In addition to covering more ground time-wise, having team members in different locations helped with meeting schedules (US-based companies don’t need to align with EU hours, etc). Jake Barnett, our resident New Yorker, runs a number of research initiatives on the basis of the Beta Program. We’ve also had a couple of teams from outside of Cloud Acceleration join the program as guest stars, which worked very well.

Today, we have 10+ large customers in the program and we’ve facilitated some 45 meetings in the past 5 months, not counting the follow-up calls devoted to particular features in development or regular user testing. Thanks to feedback and insights from these customers, we’ve been working on features that enable a secure connection between products in the cloud and on-premise or allow for user data clean-up before migration.

None of these would be possible without the engagement from our field teams, especially Technical Account Managers, who effectively bring new Enterprise customers to the program.

After all, one of our program’s attendees once told us: “We’re noisy because we want to remain your customer!”.

We’re here to hear that noise and remold it into tangible user value.