Developer Sandbox

At Button, we practice a philosophy called “omotenashi.” It’s most commonly used in Japan’s travel industry to refer to extreme hospitality. However, when applied to building software it’s stretched to include anticipating your users’ desires and needs. Omotenashi is the basis of our Contextual Commerce product — to provide consumers with what they want and need at the exact right moment.

In the Summer of 2017, our product and design teams took stock of areas where we could provide more omotenashi to our customers. After conversations with Sales, Partner Success, and Partner Engineering it was clear that we needed to make it easier for customers to test their integration.

Challenge

Any action on Button today is “live” which isn’t ideal when Merchants and Publishers are integrating and setting up. It also impacts Merchants who perform ongoing testing.

• In many cases, Publishers are forced to spend real money to test integrations.
• When Merchants test, their transactions incur real commissions. This pollutes the data and makes distinguishing real payments challenging.

This challenge impacts almost every piece of our technology stack. Dashboard, SDK, API, developer documentation, guides, Merchant and Publisher test apps.

Goals

Together, our product team sat down and outlined the goals we wanted to accomplish with the new developer sandbox. They included:

• Enable Merchants to place test orders that are visible in the Dashboard for confirmation, but do not incur commission cost.
• Separate test and live orders in the Dashboard, API, and Webhooks.
• Enable Publishers to test orders without spending real money.
• Testing doesn’t impact more than one partner.
• Reduce the integration time for partner developers.

Non-Goals

Using the feature to onboard all users

Being a pain point in the integration process, we wanted to get this out quickly. Instead of using Sandbox to onboard all partners into the Dashboard, we’ll only turn the feature on for a few partners. Those users will see “new feature” onboarding.

Ability to test with real partners

It’d be ideal to allow Publishers and Merchants to test with real partners on each side of the marketplace. However, this would require more engineering work on our side and additional permissions from each organization.

Research

Design isn’t normally involved in the integration process so the main focus is to learn from those folks who are on the front lines. I put together a list of questions to ask our Partner Integrations team.

• Walk me through the last integration with which you assisted.
• What’s the biggest hurdle for a developer in our current integration?
• How did the developer learn about Button?
• At what point in the sales process was the developed engaged?
• Tell me about the last time a developer tried to integrate and failed. What was your response.

From there, I reached out to several contacts at both Publisher and Merchant companies to understand their needs.

• What did the process look like from sales to integration?
• What was your first experience using the dashboard?
• Who was involved in the integration process?
• How would you describe your integration process?
• What would’ve been most helpful during integration?
• Does your engineering team use separate environmental builds like “staging?”

I took my findings and presented it to our product team as well as a larger group of observers and team members.

Surprised by:
I was most surprised by the number of requests we’ve had for this feature. Sid, who leads our Partner Integrations team, said every technical kickoff call he’s had has seen developers ask about testing.

More confident about:
Every partner organization uses a similar development structure (staging, production) and thought of Sandbox as a “pre-launch” process. That really helped us make the decision to implement Sandbox as mode rather than a feature.

Learned
I learned a ton through this research and gained a lot of appreciation and empathy for our Partner Integrations team. However the most interesting thing was seeing how existing partners were solving for not being able to test. One Publisher was even placing cheap hotel reservations in Cambodia!

Approach

I developed three possible solutions informed by the patterns recognized during past integrations.

Present the Sandbox as a separate feature

• Pros: Create a space just for developers, easily scalable patterns
• Cons: No marketing means no links testing, navigation doesn’t make it easy to discover

Quick wireframe of what this approach might function like in the Dashboard.

Present Sandbox information inline with existing features

• Pros: Easy to discover
• Cons: Too easy to make mistakes between test and production

The inline introduced duplicate information that made the interface confusing and crowded.

Sandbox becomes a “mode” of the Dashboard

• Pros: Makes a natural handoff from test to live, logical step for approval, eliminates mixed data
• Cons: New model to our Dashboard product, doesn’t work with some features, makes the product more engineering-focused

Steered by our research, we made the decision to make Sandbox a mode of the Dashboard. It builds on the user’s familiarity with the existing interface in a way the other two don’t and made the path for self-serve approvals really clear. The negatives identified turned out to not matter a great deal to the users we interviewed. All features would still be accessible when not in Sandbox mode. Plus, this portion of the integration is done by developers so it makes sense to use a paradigm with which they’re familiar.

Solution

Navigation

In order to create the sense of a global testing “mode” in the Dashboard, we needed display that choice at the highest level of the interface, the navigation. Looking at other dashboards in the affiliate and payments space, we narrowed in on a switch or toggle component. We mocked up different variations of this, specifically focusing on the language used.

The components below were tested first with our internal engineering team and then with engineers from Merchant and Publisher partners who’d be using this feature.

Iterations on top of iterations. The UI outlined in blue tested best with both our internal and external developers.

Different states of the Sandbox toggle in various screen widths.

Onboarding

Developers were really familiar with Sandbox environments in 3rd-party dashboards. In fact, our Partner Engineering lead said that most developers specifically asked about it. With that in mind, we felt it was best to keep feature-level onboarding simple and obvious.

We went back and forth on what the call to action should be in the onboarding. Knowing that most unsuccessful integrations were due to developers not reading the docs we launched with “View docs.”

Dialogue triggered when hovering over the Sandbox mode toggle.

Onboarding dialogue that’s triggered when a user switches Sandbox mode on for the first 3 times.

Dashboard

One of the key benefits of using a global Sandbox mode versus a separate feature or inline solution was the ability to only need minimal changes. These included a persistent way of knowing you were in Sandbox, badging labels, and the occasional copy edit.

The difference in interface when a user toggles Sandbox mode on.

In-App Testing for Publishers

Publishers follow these steps to test their integration in combination with our Developer Sandbox.

1. Publishers insert the auto-generated code from the Dashboard into their app.
2. A Button for the Test Merchant appears once the code is implemented. The Button takes the user to a mobile website where they can purchase from a list of items.
3. Once the fake order is placed, users are encouraged to check the Dashboard in Sandbox mode to see the transaction and verify the price, currency, and list item.

Example of a Publisher rendering the Test Merchant Button in their app and the flow that follows.

In-App Testing for Merchants

We took the Sandbox project as an opportunity to update the visual style of our Partner Test App to better follow Material Design and iOS 11 guidelines.

While Publishers don’t need the app to test, we created a new feature called Integration Test just for Merchants. It works like this:

1. Tapping the “Place Test Order” takes the Merchant to the staging or developer version of their app where they can place an order.
2. Upon checking out, a notification is triggered by Button when we received the purchase in our system. The Merchant’s developers can confirm the amount and currency of their test purchase in the Button Test App.

Integration Test feature in the newly designed Button Test App.

Test flow for a Merchant (Jet).

Launch

Most of our Dashboard changes at Button are rolled out slowly to users. Sandbox was no different, especially as part of the onboarding relied on Partner Engineering educating developers during the first technical call. We officially launched Sandbox and the updates to the Button Test App on October 3, 2017.

Results

The introduction of a dedicated Sandbox environment in the Dashboard has had a positive impact on the integration experience for both Publishers and Merchants.

• The time between the technical kickoff call and code submission to Button decreased from 12 days to 5 days.
• The feature has been widely adopted by our partners and 100% of integrations since the launch has utilized Sandbox.

Appreciation

The Developer Sandbox was truly a team effort. I want to thank the following folks for their invaluable contributions. Without them, this would not have been possible.

• Chris Maddern, Exec Stakeholder
• Eden Hammond, Product Manager
• Daniel McGrath, Engineer
• Pavel Pantus, iOS
• Wes Smith, iOS
• Chuck Greb, Android
• Liwei Mao, Data Scientist
• Wilson Tsao, Data Scientist