How Manual Testers Should Adapt to the DevOps Era

The industry is shifting towards a more automated software development and testing processes. This trend, however, threatens to eliminate the need for testers whose main job is to manually execute large amounts of tests. Although testers have important contributions that would never be substituted by advanced automation or machine learning (ML), they have to embrace new trends in the software world — and adapt accordingly.

Anna Royzman, the founder of TestMasters academy, hosted Dror, TestCrafts’ CEO on our most recent webinar: “How manual testers should adapt to the DevOps Era”. Dror discussed what has changed in the software industry since switching to Agile and DevOps, and how it affects Software Testing and QA professionals. He further explained how testers can up their game to increase their value and become essential in the modern DevOps process.

This webinar had a focus especially on ML assistance on automated testing. ML or artificial intelligence (AI) tools are the hottest topics right now in the testing industry, however, QA managers and manual testers have little to no knowledge about it (see our poll below). Therefore, Dror gave priority in lecturing the attendees on the relevance of ML assistance for a manual tester — with real examples, statistics, and applications.

Source: Poll conducted on the Webinar. There was a fourth option: “Very familiar” but none of the attendees checked this one.

Dror closed the webinar with a Q&A with the viewers. This Q&A caught us by surprise with the number of thorough questions. For your benefit, here are the answers to the questions provided:

Q: The example that you gave for ML of a button moving from one side of the screen to the other — this can be addressed by having custom attributes. How can ML address such problems?

A: That is true, custom attributes could be used — you can achieve that with plain Selenium but you would need vast resources to get to that level. ML, on the other hand, is a learning algorithm. If there is an element that has changed and the text failed, we ask the user for help (by pointing to the correct element on the screen) and add that to the algorithm’s body of knowledge. Next time that we are going to face this challenge, the text will fix automatically — this is how ML is being used in that sense. We do have an option today to alert the users on elements that are more probable to break because they are written in a certain way. We know that in advance by analyzing all the elements in the test and we give the user the option to select the strongest attribute to that element. Again, we add this information to the data of the algorithm to improve it going forward.

Q: Does TestCraft support regression testing? Or you can you say load testing?

A: Our focus is on regression testing or functional testing on web applications, not so much on load testing.

Q: Is the TestCraft system 100% codeless? Absolutely no code maintenance?

A: There is no code involved whatsoever. The way it works is that the user uses a visual means to create tests either by dragging and dropping elements on the screen or sort of tracking the user actions while using the application. Under the hood, we turn that into Selenium code and it’s specifically designed for testers with no coding skills. Having said that, we do give testers with coding background the option to add certain code assets to the test and run them as part of the test — but it is definitely not required. So some of our customers that are engineers might want to do more but it’s definitely designed for manual testers.

Q: What you said sounds mostly as a recording tool, which is around for almost 20 years. What’s the difference between a codeless testing platform and a recording tool?

A: So in general recording technology is great because it is very easy to get started and start your first test. You just record, go through the process and there you have it. But when you want to do more complex tests, you see that it is a limited technology. For instance, it does not support iframes for example. There are all sorts of limitations to the technology. But even if you were able to record a certain test, when your application is changed and is updated frequently, then you need to find ways to intervene with the recording session which is not visual. It might be easier for you, in many cases, to record again, and if there is a session with a bug it might not be able to do so. What we are doing is very different, but from the users perspective, it is almost the same. You launch the application and you go through the scenario you want to create an automated test for. You just point and click on elements on the screen during a real session with your application. What we are doing is not recording a session but collecting the single steps of the user actions using this steps in order to create a visual model of the test on a canvas. This model is built out of single steps (a click on a button, a text entry) and is very easy to maintain and edit. You can very easily remove an element, add an element, start a new branch on your test, a new scenario — so its very flexible this way. And also it supports all of the iFrames, tabs etc. that are not supported with a recorder.

Q: Which platforms does TestCraft work with? iOS, Android, web?

A: We support web applications for all of the common browsers (Chrome, Firefox, Internet Explorer). We also support mobile applications with mobile simulators for Android and iOS as well. We do not support native mobile apps.

Q: Do you have to retrain the ML model for every new site you want to test?

A: First of all the model comes fully functional out of the box. Training it only makes it better over time and this is application specific.

Q: How easy is it for manual testers to adapt to ML when the world of manual testers is already struggling with BDD and TDD?

A: As I mentioned before, you don’t need any coding skills to use our platform, so it is already enough if you understand the business process, the application and what needs to be tested. You don’t need any technical skills for using our tool. ML is used under the hood — It tracts your simple visual actions on the screen and these feed the algorithm and makes it better over time. So you don’t need to learn anything about it or understand how it works. It’s nothing like those other platforms where they do require some training or technical understanding (coding skills). It really is codeless.

Q: Tell us a little bit about the ML part of it because there are a lot of questions on that. Do you ever have to adjust the model to send it to the customer? Or is it just fine?

ML is smart enough to overcome certain changes in the way the elements are being used in the application. It is not as smart as to create its own tests, so if there is a change in the logic of the application, then the tester would need to create a test for that new logic. However, if you have a regression test where the logic remains the same but the application is going through constant changes and updates, then it is smart enough to overcome most of these changes. And as I mentioned earlier, it improves over time. So the work of the tester creating automated tests is very simple — just to go through the test’s steps. Again, under the hood, we write the code for that. We allow testers to connect with their DevOps or CI/CD processes and it’s all being done automatically including the test execution, fixing broken tests, and learning from the user actions.

Q: Does the test results’ output from TestCraft work with other tools, like bug reporting tools? And if so, which ones are partnered with TestCraft?

A: Yes. When there is an issue or bug that is identified on the TestCraft platform, then you have different options. First of all, after every test execution, you have a set of screenshots for each of the steps of the test. If there is a bug or issue, you can share that information with the development team, for example, and they will have a very clear understanding of what went wrong. Then, you can report the bug in the TestCraft platform — so you can track it. Also, you can report it directly into Jira. You can have Jira as your issue management platform and it feeds of bugs directly from our system — which will include all of the information the developer needs in order to address it, including screenshots of the test. You can also communicate it through Slack or via an email.

Q: Is this tool more for UI testing or can you code basic sets of functions and reuse them?

A: There are two different answers for this question; first of all it is focused on UI. Basically what we do is end-to-end testing but through the UI. So we test the functionality of the system through the way that it is being reflected in the UI. We do have an option to also test backend components. For example, if you made a change on the UI and you want to make sure it was reflected correctly in the database, we do support that as well.

Second answer: Any process, or even a single element that is being used in the system, you can reuse them across other tests — and this is a real reuse. Let’s take the login process for example you create it once and reuse it across different tests and that is something that is changing in the application — you only have to create it once and it will immediately apply to other instances that are using this test.

Q: Can the tool be used for mobile app testing?

A: We are not supporting native mobile applications yet, but we are supporting hybrid applications and of course mobile web applications.

Q: Can a user of this tool use designs (i.e. inVision, Sketch etc.) to create the base set of tests for a mobile app, then have the tool compare the app to the base tests to determine if designs were implemented correctly?

A: We are not supporting per se but we do have our own canvas for creating a test which you can use it to create a test and we will immediately turn it into an automated test. We cannot import from external tools but we have our own very simple drag-and-drop tool for doing that.

Q: Are the tests created automatically from the get-go, or do we need to define them?

A: There is a lot of thought that is put into this process by the testers, so the tests are not created automatically but they are maintained automatically.

Q: How many times in the last couple of months did you modify your ML algorithm by itself?

A: It is being modified all the time by the data that is being fed into it. The algorithm itself is not changing but the way it works is based on previous actions.

Q: Does this tool work with API testing as well?

A: Not at the moment.

Q: How easy it is for a CI integration? Which tools can it be integrated with?

A: Very easy. It’s very well ingrained into our platform. We support all of the common ones: Jenkins, TeamCity, CircleCI, Visual Studio, Heroku, and many others.

Q: Even though the recording works 99% of the time, there are still instances where the elements are hidden and require some coding, which is usually the case. In some cases, an ID will change between logins in another element that is just buried somehow and is difficult for the recording to see. Are you saying that this is never the case with TestCraft?

A: I wouldn’t say never. There are so many ways to implement or to create a web application. So it would be too presumptuous for me to say never. It is a major leap when you compare this to plain Selenium implementation but it is still not 100% — so there might be cases where you would still need to intervene in the code or you would need our support to do that. And by the way, we do that for our customers — if that is the case. So it's not 100% but we are getting close.

Q: In an initial effort to create those test cases, there has to be an investment — manually. How automation is better than manual testing?

A: That is easy. First of all, you only have to create the tests once. And then you have a set of test cases that are automated — you can put them on a schedule and run them on multiple platforms, browsers, different work environments. And you can do that in parallel. With manual testing, you need to go through the process over and over again, across different platforms and maybe using different data sets for each of the tests. So the permutations for a single test can be very significant. When you automate a test, you can do that with a click of a button. So it’s a significant improvement in your efficiency.

Originally published at www.testcraft.io on September 17, 2018, by Ely Mansur.