Telling Our Stories: Public API Service
Facilitating the development of public APIs at Box
Welcome to the second part of our series that focuses on the Women in Tech (WIT) Employee Resource Community (ERC) at Box. Check out the first in this series here.
Get to know our Author, Marie Rogers:
In my years as a software engineer on the Platform Team at Box, I have had the opportunity to create features and products for one of my favorite personas, developers like me. As I have considered the needs of developers building on Box, I have become interested in the API story we are telling. I have advised as a member of our API Working Group, which reviews all new API proposals, and API Architecture Group, which considers long-term, high-level direction for APIs at Box. During my time at Box, I have also been able to invest in communities that I care deeply about, namely our Women in Tech Employee Resource Group, where I have acted as a Global Chair and a member of our Leadership Committee. Building out the Public API Service has been one of my key accomplishments on the Platform Team, and I am excited to share more about the positive impact of building public APIs and how the Public API Service facilitates that development.
Background
API First Architecture has become a buzzword in the software industry as the importance of mobile integration, developer experience, and microservice architecture grows. Here at Box, as we move from our monolithic to a microservice architecture, we find ourselves becoming an internal API first company, with many internal APIs being created to facilitate feature development, but only a few being translated into publicly documented and available APIs. Unfortunately, this narrows the benefit of our APIs and cuts off the growth of important Box products including Platform, Mobile, Integrations, and UI Elements. Though some of our deficit in public API development comes from a historical under-investment in public APIs, we found clear areas for improvement in our internal developer experience as we looked to cut down the friction and overall effort required to build public APIs on top of existing internal APIs. We built the Public API Service to help alleviate some of the pain points around public API development here at Box, and as we move forward, we are using the Public API Service to solidify and grow our public API offering.
So what were the pain points in public API development at Box?
For one, we provided no general support for API development: no team owned API development, no tooling existed in our microservices to build public APIs, and documentation was poor or non-existent. We were putting the burden of engineering design and investigation of public APIs onto the shoulders of our developers, and with multiple internal frameworks that support API creation, there was not a clear, consistent process. We also did not maintain proper public documentation which led to a worse developer experience and frequently broke our autogenerated SDKs. So, we decided to build a service that provides a paved path for development while managing three key aspects of our public API layer: scope checking, payload and schema validation, and request orchestration.
Payload and Schema Validation
The Public API Service is built on our standard BFF (backend-for-frontend) Javascript framework which provides built-in payload validation including checking for missing fields, incorrect types, and enum values. On top of this, it is an appropriate place to include additional, endpoint-specific validation. We hold two separate standards for our internal and public APIs, and the Public API Service provides support for translating these internal APIs into standard public APIs. Often for our newer endpoints the translation is straightforward with limited changes to naming and casing, but having this support is especially useful for some of our older or less standard APIs. For example, when Box acquired the company Sign Request to integrate its e-signature offering, we needed to map a complicated set of API schema fields that did not match our Box public API standards, and the public API service was an ideally centralized location for this schema mapping.
We have always faced the issue of internal developers making API contract changes without updating the schema documentation. As we have grown our SDK offering and developed SDK auto-generation capabilities, outdated schema documentation has become increasingly more impactful as it can directly break our SDKs and cause friction in the auto-generation process. To improve this process, we now keep a single source of truth of our documented public API schemas in our Public API Service repository to be maintained and updated by our developers. These schemas are reviewed and copied into our Box Open API Specification, the publicly-available schema repository that powers our Box Developer Documentation and SDK auto-generation. Overall, the service greatly improves the documentation maintenance workflow, eases the burden on our Technical Writing Team, and stabilizes our SDKs by binding the internal development process closer to our external documentation.
Scopes
Scopes are a key part of any API, as they limit what endpoints a client is authorized to use. In our legacy scope framework, we bind each scope to a set of permissions, resulting in a tight coupling of responsibilities that leaves us with an inflexible and limited framework that can cause difficulties for both API providers and consumers. Our developers that consume APIs struggle to find the appropriate scope for a given API action, and they sometimes have to settle for scopes that are too permissive. Meanwhile our API providers are often limited in creating the most fine-grained scope for an API endpoint, leading them to use scopes that may give access to multiple API endpoints. This lack of visibility into what scopes are needed to perform an API action also translates into less clarity for our admins surrounding what actions an authorized app is allowed to make. For example, the root_readwrite scope not only grants access to Box Item endpoints such as /files and /folders, but also other non-item endpoints such as /comments, because the scope has been mapped to various permissions in monolith. When an application mints a token with the root_readwrite scope to upload a file, it will inadvertently have the permission to post a comment on the file, which is not ideal.
In the Public API Service, we are working to expand the functionality and granularity of our scopes by binding a scope to an API operation. For example, it is now much easier to authorize an app with scopes specific to the sign request or the retention policy endpoints. This granularity also clarifies ownership over the scope. Right now, multiple teams — from Platform to Permissions to feature teams — work on pieces of the scope framework, and very few people have substantial context on how it all fits together, but this shifts ownership squarely to the team that owns the API endpoint.
Orchestration
As Box has grown, our backend architecture has expanded into hundreds of microservices, making it increasingly important to properly route and hydrate API requests and responses. Previously, without an orchestration layer, we saw a proliferation of custom BFFs in order to interface with the necessary internal APIs. These internal APIs are spread widely across microservices and the monolith, and we have multiple mechanisms for connecting them. Now, in the Public API Service layer, we have a single point to enforce standards and clearly outline the components necessary in forming the public API response. We also properly format and return errors from downstream services, and we hope to eventually connect with GraphQL to provide an even cleaner hydration mechanism.
Future
Beyond these initial aspects, we see the opportunity for incredible growth in the value proposition of the Public API Service. On the product side, we have increased buy-in on an executive level to push forward API development as a key piece of feature development. We are seeing increased interest in the developer experience rather than focusing solely on the user experience, and public APIs are a central tenet of the developer experience. We have formed the new API Management Team to continue spearheading the growth of public APIs at Box, and we have many ideas to build out a more robust service:
- API Versioning: Without a versioning strategy, the process for ending support of API fields is difficult and leads to stagnancy in our APIs. We are investigating if Public API Service is the appropriate place to implement a new versioning strategy.
- Contract Testing: There is often a disconnect between internal feature development and our public APIs, and in order to prevent backend changes from breaking our public APIs and SDKs, we need to run contract tests that call out these misalignments.
- Improved Error Messages: Our third party developers often have to decode unclear error messages that link to generic documentation. Public API Service may act as a centralized location to add additional color to our error messaging and couple them more directly with useful documentation.
- Paved Path migration from the monolith to microservices: As we move away from our monolithic architecture, we will need to migrate our APIs, and the Public API Service may simplify this process by providing a clean, paved migration with appropriate tooling and generated code.
- Load Testing: To maintain quality as we move over some of our core APIs from the monolith into microservices, we may use load testing to provide key insight into the performance of our APIs.
Conclusion: Why do we care about public APIs?
Public APIs drive forward both Box’s business and technical goals. Customers who build on Box Platform are stickier and buy into the Box ecosystem, and we improve and expand the features we can deliver when we create APIs. As we bring public API development into the planning and design of a new feature, we reduce the technical complexity of trying to build out a public API later on a system not engineered for it. We also ensure that our APIs meet the high standards of our clients and are properly documented and maintained. Box has grown from a small content storing company to a cloud content management platform, and as we grow in feature breadth, we need public APIs to help us deliver our vision.
