Finally, my first testing blog. It is about 3 years overdue, as I did a lightning talk at a NYC testers meetup back on May 22, 2017. I had been meaning to blog about it since, and I did so internally at my company, but I was not in a rush to publish it to the outside community. I always thought my first blog post would be about time management, and how to avoid procrastination, but I never found time to write about that topic (irony?). Hence, my overdue, ‘first’ blog post … slightly re-worked since my original writing.
A few years back, I spoke at a NYC testers meetup. It was my first talk to a group of Testers from the larger community. I gave a lightning talk (5 minutes) on the topic of a heuristic approach to using your API to test your UI. I called the talk “(h)APIness is not enough” For those of you who missed it, here were my talking points:
I started testing back in the year 2000, at the turn of the millennium. During the first dozen years or so I was always testing things that had a user interface. Whether it was an installer with screens where users choose options and set configuration settings, or a desktop app that gets installed locally onto the computer, or a mobile app that runs on a phone. I had a pretty good handle on how to test through a user interface to discover problems. And for the past 8+ years, I’ve been testing web applications and web services
Sometime around 2012, I was assigned to a new project which was tasked with building a Cloud Platform. This project consisted of building web services, each of which manages a set of API resources. Basically, the data is input and processed as web requests. At this point, no UI had not yet been developed. I was left to test the API’s on their own, and I learned a valuable lesson — make sure your API’s are resilient before you try to write a UI around them. What I also discovered was that each service has its own API documentation as part of the code, and they have a published API contract. The contract is a more human readable document describing how to interact with the API. These documents describe things such as status codes, input value types and lengths, required and non-required parameters, among other things. I was left to do all my testing through making web requests using an http client from the terminal. Interesting, and valuable, even if it’s not exactly exciting (at least at the time).
After testing the API using the web client for about a year, a UI was in development to consume these API resources in order to make it easier for users to configure “things”. I had requested to be part of that team and would bring my knowledge of how the API layer works with me. I was now understanding ways to use the API testing combined with the UI testing. In some cases, I had no choice, the UI was not hooked up yet, so I was making API calls to create, update, and delete objects (with almost no validation rules in place, yet). Then I would check through the UI if those objects were indeed viewable, or updated, or deleted. I saw an opportunity to test some of the UI rules using the API.
Once the UI was hooked up, I started to consider interesting ways I could test the system. For example: What if specifications in the contract do not match the UI constraints? or the UI does not match the DB constraints?
Does the UI denote the same required/optional fields? The UI can have fields with a character limit of say 30 chars, or it may or may not allow certain characters. The UI may prevent you from saving bad data such as this by disabling the Save button. Or it may stop accepting characters from the keyboard beyond the max UI limit for that field. I noticed the API contract had a different limit set for some fields (i.e. 255 chars). Now this may not necessarily be a bug, because the API may be consumed by an external source and they may not want to have the same limit on these fields that our UX team prefers.
There are other controls in a UI which constrain the values a user may input:
• drop-down fields for things like Countries, States, Regions, Gender or other pre-defined lists of values
• radio buttons which allow only 1 option to be selected
• check-boxes which have only specific allowed values.
This led me to more ideas, and without realizing it, I started developing a heuristic for using the API to ‘test’ the UI.
Some ideas I had were:
• Creating objects with the max value of characters the API would allow. Check how does the UI handle displaying this data, does the page load at all? Is the page readable?
• Are required fields which are denoted in the UI really required when creating through the API?
• Edit an object through the API, maybe set a required field to an empty value, does the API stop you? If not, how does the UI handle it?
• For drop-down controls, specify a value through the API that is not a valid option in the UI drop-down. Check the UI, how is this handled?
• For something like radio buttons, are you able to use the API to set multiple values true? Or update the object and set all values to false? How would the UI handle this?
• For controls like checkboxes, or other defined lists of choices, does the API have the same limited choices? Are values case-sensitive?
I am aware that there are likely many more ideas, but the object of this talk was to state that testing your API’s is not enough, to achieve true API-ness, you should see how the consuming UI works when people do unexpected things through an interface to the API layer. Consider the malicious user that you do not yet know about.
