8 simple tests that will give any REST API service client outstanding connectivity reliability

These tests are so powerful because they’re built on an OSI model understanding. Each test simply emulates a problem at the each OSI level on API side — which gives you a chance to find any unexpected behavior on a client side of that API.

So, let’s get started!

1. Machine (API server) is ICMP unavailable.

A server black hole case. This happens when a whole server is down. Transport level of a client simply can’t reach API server (machine) to establish a connection with it.

To emulate this case without switching API server down, you may execute in terminal on API server:

iptables -A INPUT -s X.X.X.X -j DROP

where X.X.X.X — is an IP-address of a client, that uses an API.

To get API server from this state, execute on API server:

iptables -D INPUT -s X.X.X.X -j DROP

where X.X.X.X — is an IP-address of client, that uses an API.

2. Machine is ICMP available, but port is not opened

This situation may happen, when API service is not started on machine or API service crashed.

To emulate this situation simply stop API service on API machine and try to do something via client:

  • try to start client;
  • try to interact with client started before API service stop;
  • try to shut down client;
  • try to start API service in each of above situation and check if client handles this situation right as well.

In every case client must not to crash, be responsible, not to throw unclear error messages to end-user, etc.

3. Machine is ICMP available, port is opened, but service not responding

This is a service black hole case. Server is fine, but service is not. This happens, when service hangs or has too long time to respond of a client request.

There are several ways to emulate this case: using SIGSTOP and SIGCONT for a PID of API service or using iptables. I’ll describe with iptables.

To emulate this case, you may execute in a terminal on API server:

iptables -A INPUT -dport XXXX -j DROP

where XXXX — is an API service port, that a client tries to reach.

To get API server from this state, execute on API server:

iptables -D INPUT -dport XXXX -j DROP

where XXXX — is an API service port, that a client tries to reach.

If your API server has several ports that are used by client, then you need to execute this script for each port (simultaneously).

4. Machine is ICMP available, port is opened, service is responding, but HTTP error code is not 200

This happens due to an internal error in API service. You can emulate it either by manually breaking an API service (forcing it to respond with HTTP 5xx, 4xx and 3xx errors) or using specially written stub http service, that will do the stuff you want. For example:

#!/usr/bin/env python
"""
Original code: https://gist.github.com/bradmontgomery/2219997#file-dummy-web-server-py
====================================================================
Usage:
python server.py
Send HTTP GET request:
curl http://localhost
Send HTTP POST request:
curl -d "foo=bar&bin=baz" http://localhost
====================================================================
"""
import time
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
class S(BaseHTTPRequestHandler):
def _set_headers(self, content_length):
self.send_response(200) # means: HTTP 200
self.send_header('Content-Type', 'application/json')
self.send_header('Content-Length', content_length)
self.end_headers()


def do_POST(self):
delay = 0 # delay in microseconds
time.sleep(delay/1000000)

# payload JSON is made as a string - so you might form an invalid JSONs as well:
content = '{\
"integer": 1,\
"string": "2",\
"null": NULL,\
"boolean": False\
}'
content_length = len(content)
self._set_headers(content_length)
self.wfile.write(content)


def do_GET(self): # example of GET processor implementation
self.do_POST()

if _name_ == "__main__":
port = 80
httpd = HTTPServer(('', port), S)

print 'HTTP server has been started.'
httpd.serve_forever()

5. Machine is ICMP available, port is opened, service is responding, HTTP error code is 200, but error code inside the response is not positive

This case happens, when API service has some troubles to process your request and that’s why it responds with non-successful error codes. These codes often described in specification for API, but I’d advise you to test more codes and also literals instead of digits in code.

To emulate this situation you need either auto tests or a stub http server, that will respond with everything you ask for.

For example, in response you expected:

{"code": 200}

but received:

{"code": }
{"code": 0}
{"code": ""}
{"code": 500}
{"code": "error"}
{"code": null}
{"not a code": "at all"}
{}

More about possible values you may read in my article: Always test these 5 conditions in software.

6. Machine is ICMP available, port is opened, service is responding, HTTP error code is 200, error code inside the response is positive, but content sucks

This strange situation also may happen, when API service has some missed bug. In this situation API may respond, that everything is fine, but with empty content.

For example, you expect, that in case of success API will respond with:

{"code": 200, "state": "on"}

but it responds with unexpected:

{"code": 200}
{"code": 200, "state": 0}
{"code": 200, "state": ""}
{"code": 200, "state": "anything here"}
{"code": 200, "state": "on", "state": "off"}

More about “content sucks” tests you may read in my article: Always test these 5 conditions in software.

7. Machine is ICMP available, port is opened, service is responding, HTTP error code is 200, error code inside the response is positive, content is fine, but API server responds with not acceptable delay

We’re almost there. API server or network could be loaded and significant delays may appear. You should test timeout thresholds in your client, and you can use netem utility for that.

On API server execute:

tc qdisc add dev eth0 root netem delay 1100ms loss 5% 1%

and this will cause 1100 ms delay with 5% loss to outbound traffic from eth0 interface of API server.

To undo this rule, execute on API server:

tc qdisc del dev eth0 root

IMPORTANT! You may set some values that may destroy your control communication channel with API server. That’s why, when testing, it’s always better to call undo rule after some timeout:

tc qdisc add dev eth0 root netem delay 1100ms loss 5% 1% && sleep 60s && tc qdisc del dev eth0 root

this will apply delay of 1100 ms with 5% packet loss to outgoing channel and will automatically undo in 60 seconds.

8. Machine is ICMP available, port is opened, service is responding, HTTP error code is 200, error code inside the response is positive, content is fine and delay is acceptable

Hurray! Bingo!! That’s a sweet case! This is a smoke check case: at last we’ve got what we wanted and expected! And everything works great. Yeah!

BUT!

But now you know, that everything works great not only in this last sweet case, but in all that 7 previous ugly cases as well — and that’s why this case becomes even sweeter!

Conclusion

Another important thing in these tests that you may find, that your client may need more responsibility (more clear and helpful messages). And that’s good!

Anyway, no matter what your client is: mobile app or infrastructure component or 3-rd party API client — these tests will make your client super responsible for almost any possible situations that may happen with API.

We design API clients with these rules in mind for years and they just work. Do you need more from an API client? We don’t.

I hope you enjoyed reading and found this article useful!
If so, please like to recommend others!

Thank you!

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store