How to be Sherlock Holmes of testing?

‘My name is Sherlock Holmes. It is my business to know what other people don’t know.’*

Once in a while we all, as a professional testers, have a moment when we are stuck with our projects not knowing how to handle this one, special, arduous bug.

You know very well what I’m talking about — it’s all about this issue — reported by our client — that we do not know what seems to be triggering.

We can’t reproduce this defect in our lab environment and no matter how hard we try and what type of approach we make it seems like this bug just has to be a feature.

We’ve tried every possible approach, we’ve double checked blueprints, we’ve prepared the exact same devices and environment as our clients and we still can’t give our developers team 100% reproducible pattern.

We just don’t know why this issue exists in the first place.

Now let’s face it — this is really frustrating. We all feel bad about the whole situation and we all want to solve this problem as fast as possible but the question remains — what can we do? Sending a message „It works for me” is not a valid option.

So how can we handle properly this sort of situation? — one might ask. Let’s think outside the box! — he would probably hear as an answer. The only problem here lays in the fact that being creative is not something you might easily trigger. It’s not like a lack of an energy when you can drink a coffee or eat some sweets and all of a sudden you are triggered into the action.

Creativity and thinking outside the box both require from a tester a special mentality. Some specialists suggest it can be learnt by many years of hard work. Other will say that you need to meditate so the creative stream can hit you.

I on the other hand like to do something different.

Whenever unsolvable issue appears I begin to think less about my job as a mundane IT position and more as a detective’s investigation in which I need to find evidence, match the clues, “interrogate” people and report my progress to higher authority.

Even if the role playing method is nothing new you would be surprised how helpful it might be not to mention the clear, multiple resemblances between those two job positions.

‘I am the last and highest court of appeal in detection.’

First step you should do (after making sure that you read the case files ) is collecting and organizing evidence.

We usually get some sort of feedback from user or client but you must be very pedantic with this step and remember to properly label each minor information. Every little detail might (and will) have a huge meaning later on. Just imagine that you are performing a simple payment gateway testing for your client.

You will need to: acquire proper test data for the dummy credit card number, the gateway information like Google wallet, Paypal and, also, a payment gateway document with error codes, not to mention all of the parameters that are involved in such process (query strings, variables, gateway language, currency format and so on) and we are still talking here about prefatory preparation.

Now you understand why Holmes had Watson to do the dirty work :)

The importance of the correct labeling is also very important just for the order sake — you will be able to find info faster not to mention that after a few months you won’t even remember what was the case you were working on. Spending a few extra minutes now can save a lot of hours later.

BTW if you suspect that something might be important for the case — write it down. Something as banal as f.e. type of phone technology (CDMA or GSM) can drastically change the reproduction rate. Keep in mind, however, as a rule of thumb, the Occam’s razor law which says that “Among competing hypotheses, the one with the fewest assumptions should be selected”.

‘I never guess. It is a shocking habit, — destructive to the logical faculty.’

Once you are satisfied with your evidence start to interview suspects, witnesses and victims just to be double sure where to go with your investigation.

One good example how critical this step might be is a story in which my client gave a lot of detailed information about the problem with web browsers in their office.

The portal we created was not working for their employees. We had all the information about browsers the company was supporting and providing for workers and on all of the mentioned configurations the portal was working for us smoothly. Our client was very paranoid about the security of their company so we were able to speak about this problem only with managers and product owners.

After all, unofficially, we decided to contact some of their wage earners pretending we were doing survey asking them what web browsers are they using currently.

The result? 60% of them were an older employees with habits they couldn’t change so it seemed they all uninstalled modern web browsers and changed them to Internet Explorer 6 (in the year 2013!). IE6 was not supporting some parts of the code (this changed a bit with IE7 and higher) and that alone was the reason why the portal was not working for them.

Two hours and one small patch later both company and employees were happy cause fix was working fine yet this just proves how important such step is.

Remember — assumption is the worst thing possible because when something goes wrong it will make “ass” out of “u” and “me”.

‘You see, but you do not observe. The distinction is clear.’

While doing your job remember to work alongside criminalists to process the crime scene.

Be in touch with the test lead, product owner, developers and your whole team and ask them for any sort of advice. They will provide you with an extra knowledge and explain faster how the product is (or should be) working not to mention the fact that people love to show their experience and skills.

You can’t be alpha and omega with every aspect of the product and nobody is expecting from you that you will know how, let’s say, multipoint feature is working (speaking from the logic point of view) but it’s good to acquire as much know-how as possible.

For example go to the Bluetooth specialist and perform with him a Wireshark package controlling test. This way you might even suggest some solutions like “is it possible that RFCOMM (radio frequency communication) is connecting to the wrong channel?” or “why BRSF (bluetooth retrieve supported feature) is showing something we are not supporting?”.

Sometimes people, especially skilled in narrow, difficult area, take stuff for granted (“I know how it works and it has to work this way”). This is an extra opportunity for you to double check them and assure the problem is not laying on our side not to mention this can be a solid knowledge foundation for other members of your team.

‘The emotional qualities are atagonistic to clear reasoning.’

Whenever it is possible try to attend autopsies as often as possible.

This step is probably one of the most difficult from all of the mentioned so far because it requires from you a deeper hardware knowledge not to mention you have to have the exact, same device which had our client.

Emulators or imitating the environment are correct strategies but sometimes they are not enough. Go deeper. Ask for the problematic device and even attend disassemble process so you can find the root of the whole evil.

This is how my team discovered that smartbands we were working on had a factory chip problem. The problem was only seen in Asian market because the manufacturer used cheaper elements as compared with rest of the world. They were obsolete after one year (instead of five) so it wasn’t a big surprise there weren’t working at all after 12 months since purchase.

We would never discover that if it wasn’t for the package send to us directly from China which contained some of such smartbands bought on their local market.

‘It is my belief, Watson, founded upon my experience, that the lowest and vilest alleys in London do not present a more dreadful record of sin than does the smiling and beautiful countryside.’

Doing your job means also that you need to perform surveillance work and monitor the suspects.

Many testers forget about necessity of checking memory leakage, CPU usage, battery life and so on. Saturation testing (checking the behavior of a software product when specific resource is overloaded) in this day and age, when everything is connected with anything, is a must.

This is especially visible with mobile devices industry. With each next iteration of a software release user can see drastic performance downgrade of his device.

FOTA (firmware over-the-air) process requires from you to check that device won’t act fishy after the update so you need to focus your energy on covering a huge spectrum of devices and systems: Android, iOS; Samsung Galaxy, Xperia, iPhone; Kitkat, Lollipop, Marshmallow, Nougat, Oreo; product with factory firmware, up-to-date, with previous release — as you can see there is A LOT to think about when you want to be sure that your “suspect” is doing well (or, perhaps, when you want to explore the issue).

Of course, as a good “detective” you sometimes have to be a little wary with your suspect. A good example of that is case in which I’ve discovered that Bluetooth headset was losing connection after one hour of an ongoing call. We fixed the problem but how can you verify that the device is now acting correctly?

If your idea was to make a “live” call for over one hour then tell me did you consider that: some phones use HD voice and other SD voice (quality of the conversation), microphone can sometimes drain more battery depending on the sound source (so using music player is not an option), leaving headset just as it is, by itself, in loud environment can end with static noise or crackling, some platforms are using a call improvement (better quality of the conversation for some extra battery life).

This list can go on and on so I’ve decided to use some clever approach — an audiobook and BT headset connected with the most popular devices on the market. The advantage of this approach is the fact that lector is speaking on the same level of loudness all the time and audiobooks last for a couple of hours so tester doesn’t have to be focused and involved with this one particular test all the time. Not to mention this is a quite good imitation of the real scenario.

“The world is full of obvious things which nobody by any chance ever observes.”

Keep in mind that you will also have to testify in court as an expert witness.

Always try to be the first one to ask the most of the questions. Try to be as useful as possible both for your team and your Product Owner or Test Leader.

Being a professional means being active so don’t sit on your seat all day long but rather than that share your thoughts and use some neat tricks to get better results.

How often do you check IMEI/USSD codes? Do you use testing dashboard? Are you using shell monkey to better handle the randomness nature of the user? Are you saving time by using testing mind maps? Those are just a bunch of simple questions you should ask yourself right now.

If in any case you said “no” then maybe it is the best time to sit down, google some stuff and become even more proficient in your job?

“To a great mind, nothing is little”.

Being testing detective might not be an easy thing to do but once you get used to the role and start pretending that your job is something more than just an IT position you will see drastic improvement in your performance.

This might not be perfect solution for everybody but it is worth at least a shoot.

Overall remember that the world is already a big, scary place full of many bugs, issues and defects just waiting to be discovered so it is more than ok for you to at least have as much fun with that as possible.

“Excellent!” I cried. “Elementary,” said he.

*Quotes come from Arthur Conan Doyles series of Sherlock Holmes books.


Marcin Sikorski — IoT Freak. Tester. Writer. Public Speaker. Trainer. Owner of http://smartrzeczy.pl Also — big enthusiast of China.