Handling inter-parameter dependencies in REST APIs with IDL4OAS
Inter-parameter dependencies can now be specified in the OpenAPI Specification!
Summary: In a previous post, we discussed what inter-parameter dependencies are and what types of dependencies can be found in real-world REST APIs. In this post, you’ll discover an alternative for handling inter-parameter dependencies in the OpenAPI Specification (OAS). We propose the Inter-parameter Dependency Language (IDL) and an OAS extension (IDL4OAS) for describing and analyzing these dependencies automatically. Potential applications in areas like code generation, automated testing and monitoring are promising!
Hey, could you refresh my memory? What was an inter-parameter dependency?
An inter-parameter dependency is a constraint between two or more input parameters of an API that must be satisfied to form a valid API call.
If you want to know more about these dependencies, I suggest you to check out my previous post.
Okay. So you were saying they are now supported in the OpenAPI Specification?
Indeed! So far, these dependencies had to be described manually in the API documentation, leading to ambiguous API specifications. In fact, the OAS documentation states the following:
“OpenAPI 3.0 does not support parameter dependencies and mutually exclusive parameters. […] What you can do is document the restrictions in the parameter description and define the logic in the 400 Bad Request response.”
This lack of support made the community actively ask for this feature! Cedric Warny put it nicely:
This issue has become the most upvoted of all times in the OAS repository. And it’s still open. To date, there’s no API specification language supporting this kind of dependencies. Until today!
In this post, I’m happy to introduce you to a tool suite for the specification and analysis of inter-parameter dependencies in web APIs. More specifically, I will present three resources: (1) a domain-specific language (DSL) for the specification of dependencies; (2) an OAS extension for their integration in the OAS ecosystem; and (3) a Java library for the automated analysis of dependencies.
Inter-parameter Dependency Language (IDL)
Inter-parameter Dependency Language (IDL) is a DSL tailored for the specification of dependencies in web APIs. It is based on our previous review on the presence of inter-parameter dependencies in industrial APIs [1], therefore it provides support for the seven patterns of dependencies consistently found in practice. This includes dependencies such as “if a parameter is used, then another parameter must be set to a certain value” or “out of three parameters, only one must be used”. If you want to know more about these patterns, have a look at my previous post.
Next you see examples of IDL dependencies for each of the seven patterns, taken from real-world APIs such as YouTube, Twilio or Google Maps:
IF videoDimension THEN type=='video'; // Requires
Or(title, description); // Or
OnlyOne(From, MessagingServiceSid); // OnlyOne
AllOrNone(subject_id, subject_type); // AllOrNone
ZeroOrOne(radius, rankby=='distance'); // ZeroOrOne
max_id >= since_id; // Relational
offset + limit <= 1000; // Arithmetic
IF type=='video' THEN OnlyOne(embed, data); // ComplexBesides simple dependencies like these, IDL also supports more complex ones using logical operators or combinations of existing dependencies, like the examples below:
IF rankby=='distance' THEN Or(keyword, name, type);
IF intent=='browse' THEN OnlyOne(ll AND radius, sw AND ne);
IF radius THEN intent=='browse' OR (intent=='checkin' AND
(categoryId OR query));
NOT OnlyOne(Or(p1, p2), Or(p3, p4));
AllOrNone(p1=='value1'|'value2' AND p2>-1, p3>3);
IF p1 THEN p2==p3;If you want to get a deeper understanding of the capabilities of the language, do not hesitate to visit the GitHub repos of IDL [2] and IDLReasoner [3] (more about this down below!). There you will find dependencies from real APIs and more complex artificial examples.
As a last remark, IDL is specification-independent, therefore it can be integrated into any API specification language such as OAS, RAML or Blueprint. In this post, we describe how IDL can be integrated into OAS specifications thanks to the IDL4OAS extension.
IDL4OAS: An OAS extension
IDL dependencies can be straightforwardly specified in OAS specifications using IDL4OAS, an OAS extension. You just need to include the keyword x-dependencies at the operation level, and include the list of dependencies as an array. Take a look at this simple example:
The operation GET /example/route accepts three optional input parameters, p1, p2 and p3. Despite being optional, at least one of them must be used, according to the first dependency (Or(p1, p2, p3);). Also, if p1 equals true, then p2 becomes required (dependency IF p1==true THEN p2;).
IDLReasoner: Automated analysis of dependencies
Okay, so now I know that dependencies are common, they are problematic and I can specify them with IDL4OAS. What else? Can I actually leverage these dependencies somehow?
Of course! Since IDL is implemented with Xtext (a standard framework for the design of DSLs), it is machine-readable and can be automatically analyzed. In particular, we have developed IDLReasoner [3], a Java library for automatically analyzing IDL specifications (including IDL4OAS!).
The figure below depicts how IDLReasoner works. In a nutshell, the API and IDL specifications are provided as input, and the inter-parameter dependencies are translated into a constraint satisfaction problem (CSP). Then, a number of analysis operations can be invoked in this CSP. In the next paragraphs, we explain these operations and provide example scenarios where they can be useful.
Checking the validity of IDL specifications
The analysis operation isValidIDL can be used to check whether a given IDL specification is valid or not. An IDL specification is valid if it does not contain dead or false optional parameters (more about these down below) and at least one valid API request (i.e., a request satisfying all dependencies) can be generated. This operation can be useful when editing service specifications, for easily spotting errors and inconsistencies. As an example, the IDL4OAS specification shown in the code snippet above is valid.
Dead parameters
A parameter is dead if it cannot be included in any valid call to the service. The operation isDeadParameter can be used for automatically detecting them. Dead parameters are caused by inconsistencies in the specification or the design of the service. They may be hard to detect when the inconsistency is caused by several inter-related dependencies. The IDL4OAS specification above does not contain any dead parameter.
False optional parameters
A parameter is false optional if it is required (i.e., it must be included in all API calls to satisfy inter-parameter dependencies) despite being defined as optional. The operation isFalseOptional can be used for automatically detecting them. Just like dead parameters, false optionals can accidentally arise when manually editing service specifications, and they should be avoided. Our sample IDL4OAS specification does not contain false optional parameters.
Generating random requests
IDLReasoner also allows to generate valid and invalid requests (i.e., requests satisfying and violating inter-parameter dependencies, respectively) through the operation getRandomRequest. This operation, in conjunction with test case generators such as RESTest [4] can be extremely useful for automated testing of RESTful services. In fact, it allowed us to uncover bugs in the APIs of YouTube, Foursquare and Yelp, among others [5], but let’s leave that for a future post 😉.
Checking the validity of API requests
The operation isValidRequest provides a straightforward way to check whether a given API request is valid or not. Such an operation offers many interesting possibilities. For instance, it could be integrated into API clients and servers, even those automatically generated by Swagger Codegen, thereby achieving a new level of automation in source code generation tools. As another example, API gateways implementing this operation could detect and monitor invalid calls without the need to redirect the request to the target service, providing faster responses and reducing the consumption of user quota.
The operation isValidPartialRequest can be used to check if an API request is partially valid. A request is partial or incomplete if some other parameters should still be included to make it a full valid request. This operation is useful for the early detection of inconsistencies. For example, interactive API documentation portals such as Swagger UI could warn the user about inconsistencies as soon as a dependency is violated, without having to wait until constructing the full request.
Other analysis operations
The operations described in this post are those that we consider most useful in practice, however, more are possible. In fact, IDLReasoner supports a total of ten analysis operations. If you come up with some (still unsupported) operation that you think would cover some interesting use case, do not hesitate to submit an issue to the IDLReasoner GitHub repo [3]!
Wrap-up
In this post, we presented the first comprehensive alternative for specifying and handling inter-parameter dependencies in the OpenAPI Specification. These dependencies can be expressed in the IDL language and be integrated in OAS documents thanks to IDL4OAS, an OAS extension. Additionally, IDLReasoner allows to invoke a number of automated analysis operations for several purposes such as checking the validity of API requests.
This post is based on a paper recently accepted in the journal IEEE Transactions on Services Computing [6]. A super short summary of that paper (just 2 pages!) was also published at the International Conference on Software Engineering [7].
References
[1] A. Martin-Lopez, S. Segura, A. Ruiz-Cortés. 2019. A Catalogue of Inter-parameter Dependencies in RESTful Web APIs. International Conference on Service-Oriented Computing. [Available online].
[2] Inter-parameter Dependency Language (IDL) [GitHub repository]. Link: https://github.com/isa-group/IDL
[3] IDLReasoner [GitHub repository]. Link: https://github.com/isa-group/IDLReasoner
[4] RESTest: Automated Black-Box Testing of RESTful Web APIs [GitHub repository]. Link: https://github.com/isa-group/RESTest
[5] A. Martin-Lopez, S. Segura, A. Ruiz-Cortés. 2020. RESTest: Black-Box Constraint-Based Testing of RESTful Web APIs. International Conference on Service-Oriented Computing. [Available online].
[6] A. Martin-Lopez, S. Segura, C. Müller, A. Ruiz-Cortés. 2021. Specification and Automated Analysis of Inter-Parameter Dependencies in Web APIs. IEEE Transactions on Services Computing. [Available online].
[7] A. Martin-Lopez. 2020. Automated Analysis of Inter-Parameter Dependencies in Web APIs. International Conference on Software Engineering. [Available online].
