What the autonomous robotic vacuum cleaner teaches us: analogy with software testing

We are all a bit lazybones, or at least looking for the most effective way to do our daily tasks. Looking at my Roomba robotic vacuum cleaner, I have been thinking today that in terms of domestic cleaning, automation has just as many advantages than disadvantages in the field of software testing, and that it quickly shows its limitations.

Several techniques can be considered when it comes to soil suction — washing put aside:

• The first is 100% manual. It involves bending over the floor and picking up the dirt that sits to put it in a garbage can (or out, away from your sight)
• Then you may want to use a tool. For example, with a brush and a shovel, it is also necessary to lean, but this allows to pick up a little more dirt at one time in a limited area, without getting dirty hands, also to put everything in a trash can (or outside)
• A more top-level tool is the broom: it can tackle even larger surfaces, making small intermediate piles. But it is not enough to itself, it only moves the problem and we will still need to use the shovel and the brush to definitely get rid of the dirt (unless you have the opportunity to swing everything out in a last ample gesture of the broom)
• Next comes the famous electric vacuum cleaner. There are several models, with bag or bagless, sledding ones or that can be worn on the back, but it does not matter. Unlike the mechanical broom (manual), the vacuum cleaner (which also has its small manual side) has better suction performance, some of the steps are automated (the dust goes directly into a collecting tank) but its use remains manual:

=> It does not move on its own and is the path given by its user,

=> The dust collector must be emptied regularly,

=> Some models with bags require more maintenance, because it is necessary to buy and change the bag regularly,

• Some houses are equipped with central aspiration. The system is located in a garage in general, the house is equipped with pipes and it is possible to connect a hose from each room without having other device to drag behind.
• The robot vacuum cleaner (like the Roomba) is for its part automated with random trajectory while still having intelligence allowing avoidance movements and several calculations of trajectory (there are also robots with mapping feature, which in the end does not seem so much more effective according to a quick search). We press a button (or we program the time when we want it to be triggered) and it aspires by itself following a path determined by its algorithm (random in the case of the Roomba, so it will pass several times same place, and sometimes not at all). The only manual actions to be taken are:
• Retrieve the device to store it if it has not found its base by itself to recharge
• Empty the collector and clean the brushes

If one reflects on all these methods, one sees that each one has its utility and that not to profit from each one of them seems a mistake. If you think that your Roomba vacuum cleaner is the solution to no longer have to use the others then I answer that I can see a whole series of disadvantages:

• Your robot does not move the furniture. It is round, with small brushes to try to go into the corners, but it will always remain full of dirt in corners and areas not vacuumed
• It happens to climb on a carpet, under a low furniture, or on a foot of lamppost and to be stucked
• It does not like the wires that drag, he can brush them and be blocked

And Software Testing?

What about your test automation? Is this a panacea? You guessed that after this introduction to vacuum cleaners, the analogy is quite easy to do.

Your automatic tests are a real plus, but just like the robot vacuum they can not be the only test method:

• They may be executed on your own continuous integration infrastructure, but in case of an error it will be necessary to check the results which will sometimes tend to be a false positive. Your tests do exactly what you tell them to do at a certain time, so they need to evolve, and they have a lot of trouble getting into the corners. The Pareto law applies once again, it will be fairly easy and fast to write the 80% of automated test cases (the simplest, which may not be the highest priority in terms of risk management) but the remaining 20% ​​will require such an effort that they will cost significantly more than the former. Be careful, it’s an image I use there with the 80 and 20%, because I really don’t believe that one can automate 100% of the tests.
• Although you may have planned everything, it will sometimes happen that the automated tests do not give you the expected results (because a repository is down, because the datacenter has worries, because there is maintenance on the network slowing down everything, because the server has been updated and your libraries are out of date, because a developer had the good idea to change the name of an object…). In this case, if you absolutely want a quick result on a particular feature, then it will probably be more efficient to go back to a manual method.

Manual tests you say?

On the other hand, doing everything manually will quickly become a challenge. But just as with the broom, the brush and the shovel to pick up the dust, manually testing means you will often use tools and not only your 10 little fingers.

First, we use our brain, not just to write, click and move the mouse, but also to decide where to explore. Personally, I often have a hard time following a test case, there are always thousands of little things beside the description to try, and it is often there that you will find issues. We also use all sorts of other tools, such as plugins to fill fields in unknown encoding, or development tools to look at logs or simulate a slowdown of the network, or a proxy to watch the queries pass, etc…

You must mix, always mix

The best solution is probably to find a mix in the implementation of several of these test methods with a strategy adapted to your context and environment. With automation of the most critical and repetitive test cases (at the lowest level: unit, integration, API if possible), some automation of the graphical interface to verify that the paths most used by clients are still operational on all browsers, manual non-regression testing for functional modifications, regular exploratory testing, user testing for all cases not internally thought, crowd testing to extend coverage environment, performance testing, security testing, load testing, etc… and your application will probably be always in a good shape.

Just like at home, it may be good to have a robot vacuum cleaner that turns regularly when you are away (every day, from room to room), a little broom or shovel/brush when an area is dirtier than usual, a passage of the vacuum cleaner everywhere by moving the furniture to a weekly frequency, but also why not leaning to pick up the 2 crumbs which are lying on the floor and have fortunately not been crushed yet? Your home will always look clean.

And you, how do you see automation in your daily life?

Originally published at www.lyontesting.fr.