Api Design & Best Practices

Overview:

API’s should represent the view point of the application developer. The API should be crafted to maximize the developer’s productivity and success.

API-First design is an architectural approach where systems are designed and built with API consumers as the primary consumers.

Simplistic representation of an API and its flow

API proxies decouple the app facing api from the backend services, shielding those apps from backend code changes. As you make backend changes to your services, apps continue to call the same API without any interruption.

Enabling partners (B2B) with API as the starting point and never let the codebase go out of sync with the documentation is the guiding principle behind annotation-based software development. Tools like swagger will help visualize the API and validate the request/response that can be easily shared with the teams to facilitate a collaborative development across the organization.

Api Strategy:

Business Orientation: APIs are strategized based on the business orientation of the organization. This exposes the purpose of business operations, to be achieved by the API implementation.

Attracting innovation and disruption: Organizations expose data and operations (organizational IT assets) through APIs to attract innovation and disruption from outside the organization, injecting new thinking and skills into the business. Organizations get the benefit of insights on their business data by exposing them to customers.

Monetization: Organizations with valuable data and business operations sell them directly via APIs

Value Chain: An API implementation touches different levels and layers of an organization. Modern API implementations often include external stakeholders and other value providers like partners, suppliers, customers, and developers. APIs integrate and facilitate the digitization of business flows by connecting different stakeholders with organizational IT assets. The term “API value chain” refers to the entire ecosystem and the affairs between assets, API providers, and API consumers.

Strategic Components of API

Strategic Component view of API

Tactical/Implementation Features:

API’s consist of key features that help manage and maintain their consumption. While designing a well-formed API will make the ease of use as a priority, sharing, hosting and managing the runtime will help deliver the best customer experience. Some key aspects and service level agreements that need to be captured and tailored for every API’s are as below:

• API Proxy: Proxies help shield the real app api behind the network and help manage the SLA’s of the customer
• Proxies as a Circuit Breaker, Load Balancing, High Availability: Circuit breakers are a functionality when we see a backend api is being erroring out and we want to contain the damage from causing ripple effect on dependent services. Scalability can be achieved by configuring the load balancing algorithms and spanning multiple backend systems to handle the load. High Availability is another view of the scalability, when infrastructure sees issues in handling load, an elastic system can autoscaled based on the health checks and load indicators from the proxies.
• Pre-flow & Post-flow filter or payload enrichment/truncation and message routing: Most of the API’s have common functionalities or SLA’s they need to implement; a pre-flow and post-flow orchestration will help a externalize the common functionalities of an API and facilitate reuse
• Common service callout as part of API flow process (Java, Python, Javascript): API’s need common functionalities that need to callout third party integrations before processing or post processing of an API call. Service callout helps in generalizing these calls or help consolidate common integrations.
• Virtual url / Url masking / Endpoint offloading: provides consistent customer facing usage endpoint: Customers and business Partner’s long for consistency. Any changes to the API or versions of the API need to be transparent. Masking and offloading help conceal the changes to the external users and help conceal the intricacies of the implementation and provide a seamless view of the API
• Message transformation (xml to json / json to xml), Key Value maps: Message formatting and transformation can be decoupled from the customer and help integrate legacy or disparate systems internally. Key value maps help changes from bubbling up to the customer
• Message validation (Payload message content and data type validations): Payload validation with the data and the datatypes can be externalized and help run with common corporate validation rules.
• Protocol conversion (REST to SOAP, HTTP to JMS): HTTPS is the most common way of exposing the API consumption, in cases where the backend systems cannot support the protocol, an intermedia layer of protocol conversion help serve the internal systems without complex re-engineering efforts
• Encryption and Decryption: Wire and message encryption help data to be secure at rest and in-flight. This is a common concern and uses a common functionality on implementing certificate management for PKS, block ciphers, stream ciphers, shared key AES etc.
• Analytics and Performance metrics: Real time intelligence on the number of api calls, load, bottleneck can be analyzed using tap on the data flow. Helps in real-time analysis and mitigation
• Threat protection (Payload, IP, Geo location): Blacklist and Whitelists are an easy way to make sure filter the machines or DNS providers to safeguard our systems. Also, intelligent systems that can trace out the network jumps, virtual nets, block geo locations and help protect the API’s
• Caching and Persistence & CORS support: Caching content that is not real-time can be cached and persisted at edge systems and help deliver a faster customer experience for API customers
• Security Policy (JWT, Accesstoken, Oauth): API’s should be integrated with the security token providers for access and authorization control and the support systems should be able to help with integration to IAM, OpenID Connect, Oauth, SAML provider systems.
• Rate Limiting (Requests per minute): Some times, a looped call from a single client could bring down the API’s causing outage to all the other customers. Rate limiting policies help facilitate monitor and block API hogs and help provide a stable environment
• Concurrency Rate Limit (Parallel requests per minute): Load testing customers could span multiple clients and API’s should be protected to manage and monitor for support to concurrent users.
• Spike Arrest (Spike in volume under a time period): Spike in API calls cause the API to have patchy experience and outages, the policy to spike arrest blocks the user from causing such an affect.
• Logging and Audit: Compliance and auditing requirements can be fulfilled with this functionality of the API implementation
• Fault handling: Fault handling and appropriate notification to the monitoring systems help resolve the issues in a proactive manner.

HATEOAS is way of expressing the API without a static publishing site with the documentation and details of the API. We are fundamentally using a starter API to find out the other API. The upside is we can build rapidly changing and dynamic API url’s and their relationships. A downside is it’s a complexity only makes sense when we want our API to be dynamic and customized per user. HATEOAS is a site that provides information to navigate the site’s REST interfaces dynamically by including url links with the response. With HATEOAS, the output makes it easy to glean how to interact with the service without looking up a specification or other external document. In the below example ‘rel’ mean relationship. In this case it is self and the link is the ’href’

HATEOS links simple example

Best Practices for URL design:

• A simple and intuitive base url lays ground for a simpler API design and easy usage. Industry standards are something like `api.subdomain.domain.com\org\xx` (ex: www.equityanalytics.com would have API for ADVR group as `api.advr.equityanalytics.com` or globally organization wide `api.equityanalytics.com`)
• Design API urls for collection and a single. (ex: stocks represented as collection of all stocks `/stocks` whereas a single stock as `/stock/googl`)
• Keep verbs out of the base URL and nouns are good to represent models. (ex: Invalid use of verbs in API design. As this get difficult to grok by reading the api: `baseurl/getAllStocks`, `baseurl/getAllTopStocks`, `baseurl/getAllShortedStocks`). Use nouns for design to make it more structured and easier to consume. (ex: `baseurl/stocks`, `baseurl/stocks/top`, `baseurl/stocks/shorted` )
• Use HTTP actions as verbs to act on the model of representation. GET/Read, POST/Create, PUT/Update, DELETE/Delete
• Move complexity of association with the noun’s as parameters. Make it simple by sending the associations as a query parameter after ? in the url. (ex: /stocks?criteria=sp500&top=10&cap=large)
• Handle errors as Error Codes of the HTTP. Check with the industry standard error codes that an API should not override (http://en.wikipedia.org/wiki/Http_error_codes). Limit the number of codes that the application uses. 20x codes are success related codes. 30x are redirected, 40x are exceptions in handling the API, 50x are internal server errors because of infrastructure related issues that were not handled and should not be thrown in the code (P1 issues. A codebase should handle all the known and unknown exceptions and should never throw 50x error). Make sure to pass in a business error code and message as the response for exception for better debugging. Ex: Success = 200, Resource Not Found = 404, Unauthorized = 401, Forbidden = 403 Other custom errors = 40x
• Pagination & Partial responses: Partial information helps the api consumers of the information dump and help build a low bandwidth and faster responsive applications typically in the mobile and wearables. Ex: /stocks/tops?fields=ticker,shares,volume
• Pagination can be achieved by limiting the response length and by using offsets and limits for tracking. Ex: /stocks/tops?offset=50&limit=25
• Support for multiple formats (xml & json). This can be achieved in the API by specifying in the content-type in the headers or as another parameter Ex: Accept: application/json OR Accept: application/xml
• Provide an SDK and sample implementations in the supported languages of the API. SDK and reference implementations help for the faster and optimized ramp-up of the happy customers. Provide a public access for the documentation to the API and if possible a sandbox for the users to play in the sandbox with the pre-formatted data. This is help the developers understand the features and the functional capabilities for them to map their requirements to our API.

Versioning:

• One of the core features of an API is to have a version attached to the URL even though you have just one version of them now, because there will always be bugs, features and releases that needs enhancements to the API. Versions in the API help manage an existing integration and their migration and retirement.
• Versions could choose to implement the pattern of ‘major.minor.patch’. Heavily used API can use major and minor but for normal usage a major version suffices. Versions can also be part of the parameters of the url as a design choice based on the preference

Ex:

• salesforce.com/services/data/v20.0/soobject/Account
• api.equityanalytics.com/stocks/data/v1/us/portfolio/trade
• api.facebook.com/data?v=1.2

Release Management:

• API’s have lifecycle. API’s gets documented, built, published, consumed, modified, enriched and re-released as a lifecycle
• Sunsetting older api, migration of the existing customers and releasing and onboarding of the new API’s are part of the release management and product planning and the attaching a version to the API helps in the management of them
• Product management should be involved in the design of the API as they are customer artifacts and plan for their release and consumption.
• Mock implementation with the API for validating the business on the content and the value that will help the business understand the usage of the API.
• Limiting the content based on business using Grafhql to ovoid overloading and under-fetching and n+1 problem(client has to make additional requests to fetch everything they need).

Virtual URL/Masking:

• Companies have many divisions and groups and API’s are managed by individual team but they need to hosted and visualized and one single API from the company. The Virtual URL help in web redirecting & forwarding to internal API hosted systems and hence create a uniform vision for external consumers.
• Separate the different environments of (dev, test, sandbox, prod) via virtual url’s (also network security features like perimeter security, vpn networks, secure tokens etc., can be used for isolation)

Management:

API’s once started building can proliferate and hence need for the governance. Interactive API design and building tools in the market can help in their management. A common repository like `SwaggerHub` will help better understand the API and their shared usage.

Security:

All the API’s have to be protected with the corporate security standards with time bound invocation management. HMAC encryption using timestamps, Oauth token as headers, Cross Site Origination request blocking and filtering, CSRF tokens validations, Redirects & Forwards help protect the company assets/.

Protocols:

API’s should be able to support the widely used JSON and or XML protocol formats as they are the most consumed by the users over internet. GRPC is also considered as a faster binary based protocol that can be used for large payloads consumption and transmission. Since it is a binary protocol there are advantages of using it since it saves on the bandwidth and supports lower level communication protocol of the OSI layer like TCP

Exceptions & Retry mechanisms:

RESTful API’s need to be designed for Idempotency. The same action could be done multiple time but the outcome should be consistent with no partial transactions or unstable system. Retry mechanisms should be taken care in the API implementation to make sure we can handle system outages and other issues from the implementation point of view while being the issues are transparent to customer.

Testing Tools:

Postman is a great open source tool with enterprise support for testing and validating the API. It helps manage the tokens, parameterize the API and build common pre-flow and post-flow steps in a common place.

Analytics:

Measuring the traffic and sizing the needs is a constant feedback loop process. If we don’t know or can’t measure usability of the API, then we can’t build elasticity into the system.

The main stakeholders of the API users are; Product Owners, Builders, Maintainers and Influencers of an API. Metrics that matter most and called the KPI list include as below:

· Uptime(9’s)

· Resource Consumption/Constraints

· User request Load

· Traffic per User OAuth Key

· Acceptable Latency

· Error/Drop Rate

· Usage Growth Rate

· Monthly Unique User’s

· Top Consumption User

· User Retention/Stickiness

· User’s per Business Domain

· API Time to Market

· Time to Onboard new Consumer

· API Version Adoption & Retirement

· Geographic User Distribution

References & Credits:

Web API Design (Crafting interfaces that people love) pdf — Apigee

How to design a good API and why it matters. Ppt — Google Joshua Bloch

Api Design practices — Apress Book