A QA Q&A With DocuSign: Making API tests as quality gate for developers.
I recently sat down with Anjali Pilgulwar, a Senior QA Manager, and Nick Macellaio, a Senior DevOps Engineer, from DocuSign’s real estate team for a Q&A on how they use Postman. Over the course of the past few years, they’ve taken a painful and bulky QA flow and converted it into a sleek, well run testing machine. Using a combination of Postman, internal tools, and ingenuity they implemented automated workflow testing. Read on to find out how they accomplished this, what they gained, and how Postman helped every step of the way:
Anjali Pilgulwar — Sr. QA Manager @ DocuSign: I am a Senior QA manager at DocuSign . We started using Postman 3 years ago . Postman helped us shorten the feedback loop between Dev and QA .
Nick Macellaio — Sr. DevOps Engineer@ DocuSign: I’m a DevOps Engineer at DocuSign. I’ve been using Postman ever since I started here. I helped to get it into our quality gate, so that we run all of our Postman tests on every build.
Paul Logan — Developer Advocate @ Postman. I’m interviewing these fine folks today.
Paul: I assume you are using the desktop app? What exactly are you using it for?
Anjali: In the start we used the Chrome plugin, now we use the desktop application.
Nick: We mostly used it to create workflows and to develop sets of tests. Most developers have Postman and can update a set of tests.
Anjali: I was looking for an easy to use API testing tool so we could build an automation suite faster. After doing some research, I found that Postman has all the features and capabilities we were looking for. At first, our scope was just to verify endpoints as we were already smoke testing using Selenium. Considering the response time of the tests, we thought it was good idea to have some of our end to end tests in Postman. Dependencies on more than one end point took us in the direction of workflow testing. We now have good coverage of end to end tests in Postman.
Paul: What does workflow testing look like?
Anjali: We try to compare it with our smoke tests. There is more than one operation happening, so we must come up with a test scenario instead of an individual test case. For example, a manager can approve a task list that includes four or five endpoints and create that as a workflow.
“Dependencies on more than one end point took us to the direction of workflow testing. We now have good coverage of end to end tests in Postman.”
Paul: What about smoke tests?
Anjali: We use Selenium for UI smoke tests. We initially ran both together when QA was pushing the code to test environments the API tests would run first, then the smoke tests. However, several times we had a major functionality break and we wouldn’t find it until QA ran API tests later on. That’s when Nick implemented the quality gate which helped QA get better builds. To make sure critical functionality was not broken we leave it at the developer level, so when they run the build all the API tests run and we find the issue sooner before it gets to the other branch. Nick can expand more.
Nick: At first we had created one collection that had folders with all the different workflows, but that collection file turned into one big JSON file that was sort of unmanageable. So, we broke up all of those folders into around 50 different collections that represent workflows. When you develop them in Postman you need to download them; we check the collection JSON in with source, because that’s what the server will run. To make modifications you need to download them and check them in with source. That’s how the build server runs all the tests.
P: Are you using the Postman API to achieve this functionality?
N: Yes, we use the Postman API to download the collections. We’ve made it automatic so that a developer can say “I want to download all of my Postman tests into the source directory” so that later they can commit it.
P: What about uploading?
N: That’s not happening automatically. Each developer has their own account.
A: It’s a two step process to download and then upload.
N: It would be nice if Postman could read your collection off of a file system instead of the cloud. Then we wouldn’t have to go through the process of pushing new updates to the cloud before downloading via the API to check in changes.
P: Have you worked with Newman?
N: Yeah, we’re using Newman to run the tests on the build server. In order to modify the tests you need to access the cloud on your account, so modification is a pain point right now. Developers don’t always understand “When I pull the latest from GitHub, I need to update my Postman collection in order to make changes, then re-download them in order to save them and commit them.”
P: Do you have any other pain points?
A: Collections would be more readable to devs if they could also be saved in C#. That would be a great feature enhancement.
/*
Sorry folks, this isn’t on our public roadmap for now.
*/
N: When we run tests we are just checking that the response is a 200. It would be nice to make it so that we can test against an object that has already come back, like confirming a first name and a last name field exist. Those tests take a while to write.
/*
You can validate schema and response body objects efficiently in our Javascript sandbox with Tv4. Check our documentation on the sandbox API for more cool javascript tricks and libraries!
*/
P: So how else are you using scripting in the sandbox?
N: Usually we grab data for future tests in the workflow, set up global variables, and confirm the status code. Sometimes we validate the body, but not often.
A: We sometimes grab the returned messages as well. It depends on the tests really. The code snippets are pretty useful from a QA point of view. When we have to do more intricate work we go to Nick.
P: Can we get a deep dive into the technical aspects of this story?
A: We came up with a process in which developers will write happy path tests in Postman for any new endpoints or changes to existing endpoints. QA will then add negative test cases on top of that. Whenever we have a new endpoint in the code, developers make sure to add the Postman tests and run it locally to ensure nothing existing or new fails. Then, QA sits with developers to add more robust acceptance criteria
P: How does Newman fit into all of this? What tool are you using to run this entire CI process?
N: We use an internal build tool similar to Jenkins. Newman allows you to specify a collection file to run against, we pull those out of GitHub. Every time there is a pull request opened or merged into master, it triggers this flow.
P: How do you treat the tests themselves? Are they reviewed the same as code in Git?
N: If you modify a Postman test, that shows up in the pull request. We have reviews on every pull request before we’re allowed to merge.
A: It’s treated just like source.
“We use Postman for security compliance in our everyday flow. We make sure certain users can’t call protected endpoints, we see if users are getting unauthorized returns, 404’s, etc.”
P: How much time has Postman saved you after all of this?
A: A lot.(laughs)
N: Think about it — the APIs we care about aren’t breaking every time we commit.
A: We are getting much cleaner builds, the feedback cycle became very small. The majority of the functionality is working, and the only areas where QA is focusing is new functionality, not things that already existed.
“P: How much time has Postman saved you after all of this?
A: A lot.(laughs)”
N: There’s a benefit to unit tests, but unit tests don’t build and run like the entire application. Postman tests do. There have been several times where a bad branch that otherwise would have been merged into master wasn’t, because we caught it in Postman. For example, because of some kind of dependency injection looping. There are things that unit tests can’t catch which Postman tests are able to.
A: It’s a good implementation of shifting left.
P: What’s that?
N: Fail early, rather than late. Before QA even puts their hands on anything it has to pass these tests. That way it stays on the developer’s lap and doesn’t loop through several people before you figure out you broke the API.
P: So Postman tests can function as a definition of done?
A: We’d call it a quality gate. If everything looks good with the unit and API tests, only then can you merge to master.
“There’s a benefit to unit tests, but unit tests don’t build and run like the entire application. Postman tests do. There have been several times where a bad branch that otherwise would have been merged into master wasn’t, because we caught it in Postman.”
P: What are some other times when Postman has helped you?
N: We’re moving away from just serving a webpage to instead providing an API so that anyone can use it and build their own UI. Postman has helped in that scenario. Anything you can do with the web you should be able to do with an API. So, we’re moving things into a React or Angular frontend using just an API backend with Postman’s help.
A: We’re becoming platform based rather than web based.
P: Have you used Mocks? They seem like they can help you out.
/*
I may or may not have waxed poetic about using mock servers to speed up your development workflow and help isolate your front end for dev and testing, but I’ll spare you that here. If you’re curious check out Joyce’s more in depth blog post on mocks.
*/
N: Wow! That’s a great thing that would allow the UI to continue working during the development process without an API, and QA could use that as a way to test.
A: That’s cool! Lots of times when the full functionality isn’t there we can’t work on half of the stories in the release. We could still test using mocks. Can we use mocks with Newman too, or do we need to use the Postman UI?
/*
Again, I spent a while rambling about how you can use the Postman API to programmatically interact with Postman objects without using the UI. We’ll save that for another article.
*/
Both: Cool!
P: What are some other examples of how Postman has saved the day?
N: We had dependency injection loops; there have also been bad database implementations that wouldn’t have been caught by unit tests. Bugs aren’t really top of mind. They become the developer’s issue before they get merged.
A: This is habit to everyone now. Things will fail a good amount of time on [development’s] end, but that doesn’t come to QA. It’s taken care of at their level.
N: We also use Postman for security compliance in our everyday flow. We make sure certain users can’t call protected endpoints, we see if users are getting unauthorized returns, 404’s, etc.
“Bugs aren’t really top of mind. They become the developer’s issue before they get merged.”
P: What’s your tech stack like, how did you start to use Newman?
N: We use cake, which is like make for C#. We build commands that allow us to run the Postman tests using Newman. Instead of typing the Newman commands we use Cake to run them. We also use Cake to set up Postman tests, so we take the ones that are in source and push them into the Postman cloud. You can then modify them in Postman and download them into source using Cake. The build server will pull the source when it gets a new pull request, and using the same Cake command will run all the Postman tests and mark that build as success or failure based on those results.
P: Where do the information and logs from those builds live?
N: It’s recorded in the build server. We also use HipChat notifications to alert the developer that this build failed and point to the Postman tests with the HTML and JSON results. We parse them to grab the exact error from the request or response body.
P: How are you interacting with Newman using Cake? Just writing command line scripts?
N: There’s a Cake.Newman Plugin that we use.
P: What does the future of this project look like?
N: Like Anjali was saying, reading tests out of source instead of the cloud, and maybe having the tests in C#. If they stay in JSON, breaking them up and making them easily readable.
N: I also just want to say, this makes nice technical documentation for customers — “Hey you want to use our API? Here’s a Postman collection that you can run and test with right now.” This is real living documentation that works.
P: Awesome guys, thank you so much for walking through such a cool process with me!
Both: Thanks!
“ ‘Hey you want to use our API? Here’s a Postman collection that you can run and test with right now.’ This is real living documentation that works.”
There you have it. Nick and Anjali started out workflow testing in Postman after being inspired by smoke testing. A lot of QA time was going to bugs with existing functionality, so they implemented an automated flow that tested all the existing functionality before handing it off. Using Postman and Newman together with Cake and an internal CI tool, the build is run every time a developer wants to merge back into master. As a result, QA is using time efficiently by only testing new functionality, while errors in old functionality get kicked back to the developer who caused them. It’s saving the DocuSign team a bunch of time, and hopefully will become even more streamlined in the future with the information they learned here!
We’ll be doing more Q&A’s with industry experts in the future, so keep an eye out!
