REST API 101
This story is a quick reference to REST APIs. The main reference for this material is the book Undisturbed REST by Mike Stowe and the research paper by Dr. Roy Fielding
API — Application Programming Interface. An API specifies how software components to interact. It is a set of clearly defined methods of communication between various software components. Examples are Google maps API which provides directions and a weather application providing weather report through an API. API provides a specific contract the client and the server agree and provides a layer of security in between
REST — Representational State Transfer
Six Constraints of REST
Dr. Roy Fielding defined the REST constraints as per the paper
Client-Server: Client and Server should be separate and be able to evolve without impacting the each other. The changes on each side should not impact the other
Stateless: Each call should be independent of each other and should contain all data necessary to complete the call. REST API should not rely on any data stored on the server. Any data that is stateful must be stored on the client side
Cache: When handling large loads, the API should be designed to encourage the use of cached data and response should indicate that its cached data with an expiry time. This will allow clients to store data locally and not strain the servers
Uniform Interface: API should provide a uniform interface that should allow the application to evolve without needing to change the application services and modules. The interface should provide an unchanging communication standard like HTTP/CRUD and JSON
Layered System: In REST, different layers of architecture helps to create scalable and modular applications. This helps shield legacy applications with modern interfaces. Layered systems help move systems in the architecture in and out as technologies and services evolve without impacting the clients. Layered systems also help add security and other utility layers
Code on Demand: Code on Demand allows for code to be transmitted via the API for use within the application. It creates a smart app that no longer just depends on its own code structure. This is least known principles as transmission of the code raises security concerns
Planning the API
- Identify the right customers. Understanding the right customers is a very key step
- Identify the actions expected by interacting with the customer.
- List the actions and group them. Many times, same actions may appear in different users. This exercise will find the right home for each action
- Identify how existing applications will use the API
- Forecast the growth of API and plan how to maintain along with support strategy
- Have a versioning strategy, it will be like an emergency kit as a last resort to break backward compatibility
- Create solid and accurate documentation
- Plan how developers will interact — like access, security, throttling
API Design
API should be built using Spec-Driven-Development with significant but worthy steps before development starts similar to having a blueprint before starting a building construction.
Versioning
Versioning should be thought of a necessary evil
- Versioning may not be thought of an accomplishment, good APIs are rock solid without having too many version changes
- API development should not be handled as a traditional software development which keeps upgrading itself to stabilize but build a solid foundation first
- Versioning should be done for backward-incompatible changes, APIs that are not extensible and out of date SPECs.
Long-Term design
According to Dr. Fielding, “People are good at short-term design and usually awful at long-term design”.
By leveraging advances in technology, its possible to find flaws in design before actual development. Leveraging a specification is key to a successful long-term design.
Spec-Driven-Development
In Spec-Driven-Development, the design is iterative, development is not. If there are flaws in API, the design needs to be corrected and fixed in development. The following picture from MuleSoft shows the idea behind.
Constraints of Spec-Driven-Development
- Standardized: Using a standard specification like RAML or Swagger makes the design portable and thoroughly testable by the community.
- Consistent: Ensure resources are well formatted and methods all operate in similar format with regards to request/responses
- Tested: Spec has to be carefully tested by internal and external users. Using user feedback, it should be iteratively tested and eliminate large majority of issues
- Concrete: Spec should encompass all aspects of the application and be a blueprint to the developers to code.
- Immutable: The Spec may not be altered by the code. Spec must be the ultimate authority on the API design
- Persistent: Spec can evolve. but must be carefully thought out as the original foundation. Every change in API must start with design stage, test validate and so on
Popular Specs
- RAML: RAML is the newest popular spec backed by Akana, MuleSoft, PayPal and Intuit. It was developed to model API, not just document. It comes with RAML/API Designer, API console and API notebook. RAML uses YAML format.
- Swagger: Swagger is the oldest and mature of specs. It uses YAML since V2.0. It has an API designer and API console. Swagger has a large community. It lacks support for design patterns and code reusability.
- API Blueprint: API Blueprint is designed and document API. Compared to RAML and Swagger, it lacks the tooling support and uses a specialized markdown format
Prototyping
In this phase, after the Spec is ready, Mock implementations (prototyping) will help interact with potential users to test the prototype. Also, developers must test every potential aspect of API.
MuleSoft, Mockable.io, and Apiary provide several mock tools
Authentication and Authorization
Early days of applications did basic authentication and relaying the credentials to the API. This created huge security risk regardless of the trustworthiness of the application receiving it due to hacking/breaches. To help with this, OAuth a token based authorization format was introduced. In this case, the application relays the authentication page to the software provider who then generates a token and issues back to the application.
Access Tokens can be deleted, restricted with limited permissions and can have an expiry. As a best practice, Access Tokens must be unique to the user and the application.
OAuth has 2 variants, OAuth 1 and OAuth 2. OAuth 1 is more secure with Signed certificates while OAuth 2 uses SSL, but is flexible with multiple possible implementations. OAuth 2 also has 1 token compared to 2 tokens in OAuth 1. Check this link for a detailed comparison of the two approaches.
OAuth 2
The following picture depicts the flow of OAuth 2
OAuth2 Steps
- User clicks the Login link (e.g. Login using Facebook). The link contains crucial information to complete handshake that includes what application is requesting the token, the URI to which the API should respond back
- User is redirected to the API provider website where they can login and determine permissions.
- API provider generates an access token and it is tightly coupled with user and application. Token must have a set expiration
- API must validate that application uses SSL to ensure security of the token and optionally validate against the application’s domain
Best Practices
- The application must safely store the tokens
- Add additional security measures when OAuth application is being accessed from a public computer
- Watch for several requests from the same user repeatedly
- Hire a security professional in case of insufficient expertise
Designing Resources
Resources are the primary way clients interact with API, hence adhering to best practices is very important to ensure that the API is long-lived. in REST, resources are the primary gateways to areas of your application.
Decoupled Architecture
In REST, resources are decoupled from architecture. Resources are decoupled from actions so that multiple actions can be performed on a single resource. By decoupling, we can ensure that we can change backends services / technologies without impacting how client interacts with the API
Use Nouns
Use appropriate nouns like /Users and avoid things like /getUsers which ties the action to the resource. Resources should also take advantage of plurals except in cases there will be only one occurrence. The design should take a futuristic approach
Content types
Resources should be able to support multiple content-types like JSON/YAML. By allowing clients send their data type using Content-Type header, you can extend your architectures to support multiple types.
It is also possible to process the Accept header to request the client to send the type they expect. However, as a best practice, use Content-Type to know the client’s data type and use the same to send a response to avoid confusions.
Building the Architecture
The architecture should build 2 specific areas to support multiple content types, an Incoming Content Handler, and Outbound View Renderer.
Keep the Data Architecture same regardless of the format. This will help break down the incoming data using an incoming content-type handler, formatting it using a usable object that can be easily interpreted by your API service and passed on to the necessary services to generate the data necessary for response.
For output, use a View renderer. This will let you build the architecture in such a way that the response format is separate from data processing
By creating your content-handler and content-response views in a layered format, you will be able to quickly add new response types
Popular Content Types
XML is both machine and human readable. Popular formats leveraging XML are SOAP, RSS and XHTML. It encourages strict schemas and has been a popular choice for the format for enterprises. It takes more bandwidth than JSON and limited language support. But there are many libraries for interpreting XML and many times these libraries are add-ons rather than being core libraries.
JSON was designed to be an alternate for XML with Key-Value pairs in a human-readable format. JSON can also be restricted with JSON schemas though it’s not very common
YAML was designed to be simpler and more human-readable format. YAML utilizes whitespaces to identify properties eliminating the need for opening/closing braces. Support for YAML has been slow. While YAML adoption is still behind, it has become the format of choice for RAML and Swagger
Versioning
You should try to avoid versioning as a core strategy but there can be situations, its the only available solution. Avoid minor versioning altogether. There are three Versioning Strategies.
URI
This method includes the version number in the base URI for example,
api.example.com/v1/customers.
This forces the awareness of the version when used by developers. This method is not friendly for hypermedia driven APIs and also when URI is hidden deep into code, can make developers not explicitly know the version
Content-type Header
With this method, developers will append the version to the content type, for example,
Content-type: application/json+v1
This lets developers to quickly modify the version when needed and also reinforces the principles of communicating with the server. It also allows for more dynamic and hypermedia-led API. Its possible to use Accept header to expect a different version of response though it’s not a recommended practice.
In this method, the API documentation must be clear and do validation to force the developers to explicitly add version.
Custom Header
This is same idea like Content-type header and satisfies people who don’t believe that version belongs at Content-type. Example
Version: 1
This avoids the use of Accept to expect a different version of the response.
Caching
While the API can build caching mechanisms at server side to optimize performance, it’s critical to provide mechanisms for developers to cache the response will eliminate unnecessary calls. Expiry can be set by max-age or by explicit header as below. public tells it can be cached by CDN and client
Examples
Cache-Control: public, max-age=3600
Or
Cache-Control: public
Expires: Tue, 06 March 2018 18:00:00 GMT
If you don’t want to cache
Cache-Control: no-cache, no-store
Designing Methods
REST utilizes methods within resources similar to methods within classes. For web-based REST API over HTTP, we can take advantage of standard HTTP methods.
CRUD
CRUD stands for Create, Read, Update and Delete. We can apply these standard principles to our API.
Items vs Collections
It’s important to distinguish between single item and collection of items. By understanding this, some actions may not be appropriate for collections for example Delete on a collection will remove entire items and this action may need to be more carefully designed. By using OPTIONS method in HTTP, users can identify which methods are available.
HTTP Methods
While HTTP methods GET, POST, PUT, DELETE, PATCH and OPTIONS have formal purposes, there are some overlaps and the API documentation. For a comprehensive overview of all HTTP methods, review https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html
GET
GET method is designed to explicitly get data back from a resource and is the most commonly used HTTP method. GET response usually returns a status code of 200 (or ok) unless an error occurs. It relies on query string to pass on data to server
POST
POST was designed to create or manipulate the data and commonly used in Web forms. POST relies on the body of the payload, not the query string. POST must be predominantly used to create an object and return status code 201 and URI of the created resource
PUT
PUT is less well-known and was designed to update a resource and create new when not available. PUT must be used predominantly edit an item by overriding with the incoming object and use caution if it were to be used to create a new object in the absence of an existing item. API should have proper documentation of PUT usage and remain consistent across all usage by using right return codes.
- PUT shouldn’t be used to create resource directly without a reference to the object.
- PUT should return 200 for successful modification, 201 if it creates the object and 304 if the implementation decides to force the user to use the POST method to create.
- PUT shouldn’t partially modify the object, should do a complete override. Check the PATCH method for such use-cases
PATCH
PUT is also less well-known method. Use PATCH method to only modify the data provided, an example is, modify city of an address resource. PATCH should return 200 for successful modification, or 304 if the resource doesn’t exist.
DELETE
DELETE is a straight-forward method and also dangerous. Use of DELETE against a collection must be restricted. DELETE must use the following return codes
- Return 200 if object is deleted and you are returning a content back to user
- Return 204 if object is deleted and you are returning nothing back
- Return 202 if the action is accepted and queued
OPTIONS
OPTIONS is not a CRUD method. It is designed to interact with the API to find what HTTP methods are supported. The Allow header provides the list of supported methods.
OPTIONS should return a 200 if returning a body, 204 if just Headers
Handling Responses
A well formatted, coherent body response is extremely important. You should provide developers with what happened to the call and in case of failures, provide descriptive error messages with what went wrong and how to fix it.
HTTP status codes
Using HTTP status codes will make things easier for developers who are familiar and speak a common language.
Refer to https://en.wikipedia.org/wiki/List_of_HTTP_status_codes for comprehensive review of all status codes
Handling Errors
However may be a good documentation, developers are likely to skip over and discover that the calls that worked before no longer work. Errors are a reality.
Generic errors are probably more expensive as they could result in frustration, support requests etc. By providing specific error messages, the time to debug reduces enhancing the productivity.
Some sample Generic vs Specific error messages
Restricted Access — Invalid API/Token. Pls register for one at https://dev.domain.com/register
Permission Denied — Your API key doesn’t have permission to access this resource or the method. Pls contact support if you need access
Request Failed — The request is missing required data such as email. Pls check the documentation at the following link
Error Formats
There are popular and standard ways of error formats already available
JSON API: This allows multiple errors to be returned. In this case, the error should be the primary response and no other data should be returned. For comprehensive overview of structure/elements check http://jsonapi.org/format/#errors
Google Errors: Google lays out in its style guide. The error response is made up of 2 parts. A generalized error code and an array of what caused the generalized error message. Its property based and can be used in multiple formats including JSON and XML. Check https://google.github.io/styleguide/jsoncstyleguide.xml#error
error.vnd: This is another standard. This allows for error to be included as part of larger response. for more details check https://github.com/blongden/vnd.error
Adding Hypermedia
Dr. Fielding described HATEOAS (Hypermedia as the Engine of Application State) as one of the 4 sub-constraints for Uniform Interface. Adding Hypermedia to the response provides the possible links for the developers based on the action they just did. For example, when creating a user, the response provides the User Information along with links to how to delete/modify the user. This avoids guesswork/read documentation about further actions. This also can be used to specify only the allowed methods based on API credentials.
Common Hypermedia Specs
Check the links for deep dive into each
- Collection+JSON: http://amundsen.com/media-types/collection/
- JSON API: http://jsonapi.org/
- Hypertext Application Language(HAL): http://stateless.co/hal_specification.html
- json-ld: https://json-ld.org/
- Cross-Platform Hypertext Language(CHPL): https://github.com/mikestowe/CPHL/blob/master/README.md
Hypermedia Challenges
Though Hypermedia can make sure that API users get accurate links for all actions, it will pose challenges to users who hardcode/store the links. The specs are not standardized making it difficult for developers to learn new specs often
Managing the API using API Manager
There are several aspects of API that warrants a Proxy or API manager. Popular commercial platforms are Akana, MuleSoft and AWS API Gateway. It’s also possible to build a custom solution which won’t be trivial. The following are some of the major functions of an API Manager
- API Access
- Integration using OAuth
- Throttling
- SLA Tiers
- Analytics
- Security
- Cross-Origin Resource Sharing (CORS)
- IP Whitelisting/Blacklisting
- XML/JSON threat protection
Documentation and Sharing the API
Good documentation should be clear and concise and also visual. The following are some of the best practices.
- Clear explanation of resource/method
- Share important about warnings/errors
- Sample call with media type body
- parameters/rules/formatting details
- Code examples for multiple languages
- SDK examples as applicable
- FAQs
- Links to additional resources
- Interactive experience to try/test e.g. API Console
Several Open Source and commercial offerings available for Documentation tools for e.g. readme.io and ram2html.
Support Communities
Setting up communications channels like Slack /IRC/ StackOverFlow is key to help developers get self-service support and ticketing system for commercial offerings. When setting up public forums, its key to watch for spam/trolling and keep the forum effective while keeping the free speech of developers.
SDKs and Client Libraries
SDKs help reduce the burden on the developers by letting them use the API quickly without knowing all nuances of API. But it requires you to maintain and developers to update frequently to leverage new features.
SDKs doesn’t require the developers to know about various method differences like PUT/POST/PATCH, they work with the SDK. If you decide to provide SDKs, providing accurate documentation and maintaining is key. Very likely you need to provide multiple languages support.
There are commercial tools like APIOMATIC.io and REST united create SDKs out of SPECs.
