Test Automation Demystified, Part 2: Is Application Ready for Test Automation
If you decided to go the automation route, you still have a lot of decisions to make about the process itself, test scenarios, tools to use, features to care about…. In this article we will talk about the best way to start automating your software testing efforts.
Application Does Matter
When talking about Automation, the way you look at an application changes. Things that are cosmetic or minor for developers or for manual testers become very important.
If you have to make decision about automation or need to make a Proof Of Concept (POC) check.
Compare these two cases:
And this:
In the first case the AUT (Application Under Test) is installed in the remote desktop. However the test automation is not possible, because automation tool is on the calling desktop and may only see AUT as a picture. In the 2nd case we may do automation because automation engine is on the same host as the application.
DevOps and Shift Left
I believe you hear a lot about DevOps and shift left today. The idea is to make testing earlier in the product development loop.
This approach is fruitful for all kind of applications: web, desktop, mobile. In the reality development team is often not accessible and communication is limited. However, there are still possibilities that may make UI Testing Automation possible or at least better.
Some changes are easier when you are involved in the early stage of the application development. If you manage to do it then you made your life easier.
Your Situation
If you new to automation and look to available videos or listen to success stories then you have a feeling that other people leave in different world. Authors of these videos live in a world where application is properly defined. Where the tools work as a charm, scenarios created without scripting application deployed in on click.
When you then look at your application you see that there are tons of obstacles. When you think about automation you probably try to record & playback one scenario end-to-end. It probably fails somewhere and you don’t know what to do next. It is good approach. But we advise you to do something else. And we will start talking about it today.
Developer’s View
When we ask developers about an application, they would tell you a lot of info. IDEA, VS, Eclipse SQL Server, MySQL or NoSQL, protocols, load balancers, caches.
It is all good but meaningless for us. They tell what they worry about. There is actually a lot of things to care. Here is some example application component schema. You may get something like that when you ask about application:
We talk about UI testing and thus we interested in UI. Change of DB from MySQL to Postgres may require deep re-write of the program. However it does not affect UI — all tests may stay valid without modification.
In the other hand, with theme-enabled products the dev team may change name of built-in theme (one line of code) and destroy all UI tests. So when investigating an application, we need to worry about things that may harm us. We need to ask questions about UI technologies.
If we return to the schema like that we actually do need to worry only about this part:
Where to Start From
Typical approach: take automation tool and record a scenario. Then try to play and see if it works.
Well, if it does not work, does it mean you cannot do automation?
If it does work then does it mean that you may automate all your scenarios? If it does not then what you may or may not automate?
There is a recommended approach how to look at application. You may do it at the early stage when you see initial version of UI.
For example, let’s look at Material Design components for Angular JS.
There are 35 controls right now. Suppose that application is implemented using this library. Does it mean all 35 controls should be implemented and automated?
Well the trick is that usually (we know it from our practice) 50% of those controls does not affect UI automation (because they needed for logical organization of the application or design and layout).
So the reality is that you only need 5–10 controls to fulfill all the automation needs. So instead of checking the whole application we recommend to identify these controls first.
Typical parts of application to look are:
1. Date Pickers
2. Popup Pickers
3. Trees
4. Menus
5. Grids / Tables
6. Graphs
7. Output controls
For example, we have an application like salesforce.com we may go through the UI and try to identify most important controls for us.
Let’s go through and check available items. The application like Salesforce has hundreds of pages, thousands of controls and widgets. But do we need to check all of them to make decision about application automation? Here is, for example, contact editor screen:
It has 31 input controls and 3 buttons. However, only 4 different (combo box, text input, picker control) and a button.
Now check ‘New Task’ dialog:
It also has 8 controls and 3 buttons, but only one new control — ‘Due Date’ which shows a date picker:
Well, if we continue overview we will see that most of other forms contain the same list of controls. In addition to it we also need to have the following two covered:
It is grid and menu.
So overall what we need to know is that there is a limited list of controls that you need to automate. And it is essential to know what is working and what is not before doing automation.
So finally we may have a ‘TODO’ list of widgets required for Salesforce automation.
- Grid
2. Picker/Dropdown
3. Text Input
4. Combo Box
5. Date Picker
6. Buttons
7. Menu
So only a few controls are actually needed to cover such a grand app. That is a key knowledge to your success.
Rapise has dedicated support for Salesforce and we prepared for support of essential UI widgets. The whole set of widgets are available:
https://lightningdesignsystem.com/components/
So when you are evaluating the tool you may use this showcase. But in real life your application may have both supported and unsupported controls. So you goal is first to define what needs to be supported. Usually there is just few of missing ones.
Control Libraries
There are about a dozen key technologies (Win32, Windows Forms, WPF, Silverlight, Java AWT, Java Swing, SWT — this is for windows desktop only. There are hundreds of vendors of old win32 components. Hundreds of ActiveX control vendors. Thousands of Windows forms and WPF controls. And probably more web controls (DOM, Shadow DOM, Canvas). Automation tool supports most important ones. But we meet new ones every day and we have to automate those. However it is in many cases crucial to know control widgets vendor.
For example, there are plenty of web frameworks: https://www.techempower.com/benchmarks/
You may see that in addition to few leaders there are lots of smaller libraries.
The good part about library vendors is that each of them usually provide a showcase that may help to simplify our life. In this case you will have additional information from the library vendor (including samples and showcases).
Suppose that you made list of controls and found out that Rapise supports everything except one or two.
What you can do then?
1. Ask dev team about specific controls.
2. OR detect control yourself — we well talk about it today.
3. Try to automate it. If needed, send question to Rapise dev team showing an example of the control (link to a showcase or smaller POC app — will show it later).
ShowCases
Most of control libraries provide demo, samples and showcases. For web libraries the showcases are mostly publicly available.
For example, for Angular Material:
For desktop: .NET, WPF, Win32, ActiveX, Java and so on libraries you may download something from the vendor’s website, for example here it is for DevExpress:
https://documentation.devexpress.com/WindowsForms/14962/What-s-Installed/Demo-Applications
The trick here is that even for paid product the showcase demos are always part of the free trial. So even when the library is expired the samples keep working. You may have a zipped copy and keep them.
Public Showcase notes:
1. Make a copy if possible (version may change in the future and showcase may be updated without letting you know).
2. Try to get version used by the application. Or closest possible version.
Good example is Salesforce Aura library: https://github.com/forcedotcom/aura
The notice says that project is archived since March:
So anything publicly available may suddenly become hidden. That is why local copy is always a good idea.
Mini-AUT
Smaller the example — better support for the control.
We sometimes receive small apps with controls that our customers what to be supported such as:
The mini AUT contains the control (or controls) that you need to have automated. When we get such an app from the customer usually we add support really quickly — because we have all we need. It is small, there is no need to login/logout. No private data or information of your company is exposed.
Usually it is a matter of minutes to make such a POC control demo for someone from the dev team. Use this chance whenever possible!
Detecting the Controls Library
So what happens if you don’t have access to dev team (it is an legacy or 3rd party product)?
We have to detect the widget type yourself. We will talk about it more today.
How many are there widgets there? For example, how many different date pickers are there?
If we use this picture as a bird eye view we can see that there is 67% probability that you will have ‘Other’. And this ‘Other’ includes thousands of vendors. So some controls in your application are coming from those thousands of vendors. Or it is developed in house. And if you manage to identify a vendor and find control sample then you have more chances to have it supported.
There are 2 major ways of finding control type: Spy and files.
Spy
For Desktop Applications the order of Spy usage is following:
1. UI Automation Spy
2. Accessible (if appropriate)
3. Managed (if appropriate)
4. Java (if appropriate)
For web technologies both built in Spy and Browser Inspector are what you need. Most browsers have built in Inspector and this is enough for basic investigation.
What to spy for? We are interested in class names, custom attributes and control types.
Application Classes and Attributes
Desktop Application
We see that the control itself is not ‘visible’, nothing highlighted inside it, just the whole tree. In such cases automation requires knowledge of such a component.
The most interesting option for us is ClassName property ‘SftTreeOCX75SoftelvdmInc’.
One googling leads to their website:
And there we find a library:
There is a trial to download with sample apps containing this control. So you may try automation tool with this control or give it to Rapise dev team to analyze and tell your needs. We always welcome those who know what they need.
Web Application 1
There is a similar feature with web applications.
Consider this app as an example:
https://www.gwinnettcounty.com/web/gwinnett/calendar/general
We see a table with tooltips.
Let’s find out how it is done. Inspections gives us several specific items, i.e.:
The names seem to be something branded ‘data-dojo-type’, ‘dijit_Tooltip’.
So we may google for it. Thus we get to ‘The Dojo Toolkit’. And there we may quickly find a showcase (search ‘dojo toolkit showcase’):
Web Application 2
https://app.blinga.club/#/home
We may see many specific class names that are worth to be checked:
These are ‘q-item’, ‘q-router’, ‘q-link’ etc. Simple googling for ‘q-item q-link’ shows us mentions of the Quasar Framework:
https://quasar-framework.org/components/layout-drawer.html
There we may switch to desktop view, and there is a link to Showcase.
Application Files and Links
One more good way for investigating the application internals is checking its files. For desktop application the most fruitful information source are .dll and .ocx files located in the application binary folder.
Desktop Application 1
Here is an application ‘Cashflow Dashboard’:
Let’s find its shortcut:
If we go to ‘Start in’ folder we see many dll files with specific name ‘Infragistics4.Win*’.
Googling gives us:
There we may also find links for trial version containing a showcase.
Web Application
I suggest you to do this analysis yourself and then compare your findings on what we found below:
Leads to Semantic-UI:
https://semantic-ui.com/introduction/getting-started.html
It has showcase:
Always Prefer 32 bit.
Check the difference between these two applications:
It is the same application. Is there a difference? Yes. Let’s see the task manager. One is marked as ‘32’. So the other is 64. Let’s use Rapise Spy to nail down the difference.
The trick here is that some UI interfaces differ between 64 bit and 32 bit. Luckily there is a way to switch it using ‘CorFlags.exe’ utility available for free from Microsoft as a part of SDK.
https://dotnet.microsoft.com/download/thank-you/net472-developer-pack
corflags.exe ChagingApp.exe /32bit+ /force
Will make it 32 bit executable.
Sometimes you have control what version do download and install.
For example, the Microsoft Universal Serial Desktop gives you explicit choice:
https://www.microsoft.com/en-us/download/details.aspx?id=58204
So choosing i386 (i.e. pure 32 bit) may give us potential benefits.
Another example is Microsoft Dynamics Nav. It comes with 2 entry point and it is up to you which one to launch:
Why we need 32 bit? Well, there are some older APIs available only in 32 bit (such as ActiveX interfaces, some native interfaces).
Another example is .NET application built in ‘AnyCPU’ mode. It defaults to 64 bit but we may change it. For example, we have a PhotoViewer application. If we run it in 64 bit mode then the only technology that it supports is UI Automation. So recording looks like this:
Note that all actions are recorded as low level. The test recorded this way is very low quality:
See ‘ParamValue’ column: it contains mouse click coordinates. So it is hard to make this test really data driven.
Now let’s record the same in 32 bit mode. We used the ‘corflags.exe’ utility and now trying. Now recording looks reasonable:
The test may now be data-driven:
The only action that were not properly recorded is date selection. But good new is that the control is recognized and required action may be selected using DoSelectDate:
The Takeaways
Let’s summarize. we see there are two ways to do automation.
The slow way seems to be simplest, but it is a trap. Fast way has more steps, but it is quicker in most cases.
Slow and hard way:
1 .Try recording test. Solve automation problems while polishing a particular tests.
2. Add more test cases on-by-one and solve problems when they appear.
Fast way:
- Analyze app in advance, try different technologies.
2. Identify key controls. Don’t waste time on implementing big test.
3. Check which controls are working.
4. For others send info to Rapise team or add support or find workarounds. If possible find a showcase. Or find a demo app. Or create a mini AUT.
Test Automation Demystified Series
Part 1: From Manual to Automated Software Testing
Part 2: Is Application Ready for Test Automation?
Part 3: Choosing a Test Automation Tool: 8 Features That Matter
Part 4: Friends and Foes of Software Test Automation
Part 5: Codeless Test Automation
