Using Metrics to Measure API-Driven Ecosystem Value

Businesses building API-driven ecosystems face a challenge: finding effective ways to measure their success against business or organizational goals.

Such measurement is critical to understanding the success of API initiatives, regardless of whether those initiative are internally- or externally-focused.

API initiatives to build internal or external ecosystems can vary widely, and so can the possible measures applied to assess their value. Examples of this range include

  • accelerating internal innovation by reusing capabilities. While it’s easy to measure the traffic from and consumers of a new API, determining if the API is actually accelerating development can be more difficult.
  • evolving toward a microservices-based architectural model for scalability, maintainability, fault tolerance, etc. Capabilities exposed via APIs in this effort may not align directly to readily measurable business capabilities, products, services or KPIs.
  • supporting internal or external applications such as websites or mobile apps, where value is measured through the apps themselves rather than the API.
  • securing partner or third-party developer engagement. Measuring traffic levels and partner counts is easy, but measuring the business value generated by a third party requires an understanding of the business value expected. Third parties do not often allow their user-facing apps to be instrumented by API providers.

As the above illustrates, depending on how a business’s KPIs are defined, it may not be straightforward to identify the value derived from an API program or ecosystem. Numerous factors throughout the organization can contribute to this challenge, making it harder to unify the disparate pieces into expressions of value. Revenue growth per product or service, for example, may be measured by the organization for each product, but not necessarily for each channel; further, APIs may support multiple channels — websites, voice assistance, mobile apps, third-party ecosystem participants, etc.

There are a number of traditional web-based technologies available, such as Google Analytics or Mixpanel, for measuring direct customer engagement with a brand. However, to capture metrics around abandonment, feature engagement, and so on, these solutions generally depend on being able to instrument the client itself — i.e., the website or mobile app. Unfortunately, this level of instrumentation is not always possible with partner, third-party, or ecosystem interactions.

If a business engages with third parties or end-user interaction channels via an API (as many do), the business can address the instrumentation challenge by leveraging the API itself to collect metrics and funnel them to a system that enables additional insight. To enable value to be measured and apportioned as needed, organization should capture this data at the API tier.

To elaborate, let’s explore two of the key interaction types organizations may measure as API providers.

Ecosystem Interaction Types

Partner-Originated Interactions

In the partner-originated case, the partner owns the customer and has its own apps and systems instrumented for its own purposes. As an API provider to this partner, it can be difficult to get insight into the value the partner’s engagement provides. Here are some tips for overcoming this challenge:

  • If the relationship with the partner — or the nature of the API — allows tracking of specific end users, a business can correlate value created via the API directly to an end user. This, then, can have tremendous value in matching activities to specific user demographics, which is valuable when supporting the partner’s efforts or the provider’s own initiatives.
  • Different API calls have different value to both the API provider and its partner. A call to the provider’s API to retrieve product information, for example, is somewhat less valuable than one that actually places an order for goods or services.
  • Variations in traffic can indicate how engagement is changing, when partners are most active, whether the nature of a partnership is changing, etc. What’s more, traffic patterns may alert a business to illicit activity, such as partners or third parties that are caching information and using it inappropriately.
  • The number of developers and their type can give a business insight into how and where its partner network is growing, what types of partners are joining, and the success of developer evangelism initiatives. Hackathons, blog posts or user community, trade group, and conference participation may help to drive new partner sign-ups. Additionally, tracking metadata about developers can help a business segment its partner community such that it can better track where partners came from and their expected contribution to the business.
  • Information in partner requests can sometimes provide insight into the identity or nature of the partner’s customers, such as their geographic location (e.g., ship-to or bill-to addresses) or their preferences (as indicated by products or services ordered). This may help drive collaborative marketing campaigns or help a business make better logistical decisions.
  • The pattern of partner request may reflect the value partners perceive from the API provider (e.g. many product information requests but few orders may indicate the partner likes the provider’s data but not its prices).
  • Request patterns can also help the API provider rationalize the cost of supporting a partner. High-touch partners that do not generate a correspondingly high level of value may not be worth the focus placed on them. A business may gain insight by comparing the value and costs of partners that use its API to those of partners that don’t.
  • Capturing the value of goods and services provided through partners can help a business understand which partners are contributing the most to its bottom line and which are providing consistent recurring revenue.

A business can derive insights and improve partner engagement by tracking partner interactions with its developer portal. Questions to investigate include:

  • How long does it take to onboard a new partner? Streamlining the sign-up and legal/business vetting process may accelerate onboarding and improve engagement. Many businesses grant partners a “sandbox” version of their APIs, allowing developers to get started before the onboarding formalities are complete.
  • Are developers continuing to engage with the portal through documentation, community activities, blog posts or other channels? Ongoing engagement with API documentation, for example, may indicate a continued — or expanded — commitment.
  • How are partners informed about new or planned API enhancements? Using blog posts, personalized reports that reinforce the mutual value provided, and other strategies may be valuable.
  • Businesses should measure whether an API reduces partner interaction time and effort. If an organization’s API provides good documentation, a platform for experimentation, and tiers of service that allow growth, the business may greatly reduce the manual effort it expends managing partners.
  • Tracking the nature and location of interactions may provide insight into the expansion of a partner’s business, which can in turn support the API provider’s own planning process.

Self-Originated Interactions

In the self-originated case, a business interacts with partners to obtain services that the business then presents to its own customers. Customers often do not distinguish between a business and the partners on which that business relies, so the ability to measure the quality and responsiveness of partner services is crucial. Key questions to investigate via measurement include:

  • How often do customers request info about partner products?
  • How responsive/performant is the partner service, and how does that impact the end customer’s experience?
  • How reliable is the partner service? Does a business need to retry requests often? Does the partner service experience a high error rate?
  • Do the organization’s developers struggle with a partner’s poorly-defined or inconsistently-implemented APIs? Do the developers find that too many API calls are needed to accomplish goals?
  • Is cost-per-call for a partner’s API affecting the bottom line? Costs may change as partner interactions change.
  • Can multiple partners that provide the same service be “multiplexed” in a transparent way through the API layer?

Internal ecosystems — where the “partner” is another business unit — also open opportunities for measuring changes in the effectiveness of APIs in fostering innovation or supporting rapid development. Understanding internal interactions — and having an API product manager who can have conversations to explore opportunities for more organization-wide impact — can be critical.

[Looking for more insights about APIs and digital transformation? Sign up for our Virtual API Security Jam!]