Why On Earth Do We Need UAT?
Have you ever used an app then the page takes a long time to load? Or you get stuck on one screen and have to kill the app? If so, then let me tell you a bit about a group of people who get paid to “avoid” consumer complaints before an app gets released to Playstore or Appstore.
About two years ago whenever I heard the word IT, the first thing that came to my mind was always the magnificent programmer. This started to shift about a year ago when I began to work as User Acceptance Testing aka UAT in one of the famous ride sharing companies in Southeast Asia. Day by day I started to learn the different functions within the IT industry as if I ran a light into a prism.
As someone who was very new to the position and had zero knowledge about testing vocabulary or mobile apps, it felt like jumping into a roller coaster. Whenever a coworker said, “Core” or “Force Close” I’d be the first one to say “Huh?” or “What the heck is that?” My job description clearly mentioned to find any issue on the app and my very first issue that I reported was about wrong content like a condom on baby section, and that kind of issue ended up with Invalid marks from the QA. Slowly, I learned more about what made a ticket to get recognized as a bug.
Like a curious toddler, my index and thumbs started to explore each button, scrolled up and down on the page, and typed whatever came to my head. One day, I happened to catch a force close on Android 4.0 just by scrolling up and down on the page. It made me feel so happy to finally be getting into it. And as the time passed by, I finally gained the sense of where it’s going to break.
After one and half year as UAT, I then end up in a different company known for its online marketplace called Bukalapak and they just started their own UAT team a couple months before I joined in. My probation task from the Head of QA -since I report to her directly- is to shape this team. So my responsibilities are not only getting them understand the definition of bugs or getting to know the device they use or reporting bug on Jira or run the execution test on Xray. Beyond those mentioned, I to have put Nike’s “Just Do It” on their mind when they can edit the test steps without waiting for review and taking no hard feelings when their test case gets changed, and most importantly creating a psychologically safety environment where UAT peeps can openly share their disagreement when something is wrong.
At first I think it would be easy just like the previous company. At Bukalapak, there are more surprises. First of all, the team; non IT background and one of them is a former nurse. Secondly Bukalapak has lots of features that are belong to thirty two squads. Therefore, it needs different approach to perform the UAT check. Our first approach — when there are only 5 UAT peeps — is to test based on the squad, but this gets very ineffective because with that limited resource, it does not cover the whole squads to be checked.
After a couple of releases and having two more people, we reflect again on the idea of UAT has to test as the end user. Therefore we change the approach to test based on actors’ journey and then I start to split these actors into five big categories of Bukalapak users which are First Time Buyer, Returning Buyer, Seller, Quick Buyer, and Agent. How big the area of coverage and explorative can the test for an actor be? First Time Buyer for instance, covers at least four squads and the journey including signing up, verifying account, buying an item, complaint to customer service, and logout.
If UAT peeps test as an end user and following the journey, do they still use test cases? Journey is defined based on user’s necessity when accessing the app, and when going through it, test cases help to record the steps, so the test is accountable. To be more clear, let’s see a couple of images below.
Let say a user happens to check a Flash Deal item on Bukalapak App and he or she wants to buy it but does not have the account yet. So the user has to create an account first before making the payment, and the common steps for the user to sign up can be seen below.
Why do we use three different type of buyers for testing and what’s the difference among First Time, Returning, and Quick Buyer? Just because we have three actors focusing on the buyer, it does not mean we do not care about the seller. If you are familiar with Bukalapak, then you may know that you can buy without logging in which we called it Quick Buyer. If First Time Buyer has to sign up, then Returning one only needs to login. What about Seller and Agent, are they different? Both are in seller category, however the Seller focuses more on the market place while Agent can sell virtual products and do sell or buy on the market place.
How do we decide which platform or OS version to test or what payment method should we use for UAT testing?
On my second week of joining, I started to hunt the data about Bukalapak user’s behavior, from favorite payment method to average number of item replacement per week to average time to add an item to be sold. Basically, lots of data. A couple of interesting facts that I learn during gathering data: courage to ask can make you get things faster and people in high level positions at Bukalapak are approachable. Long story short, I reach out to the Head of Data Science, and voila.
Back to why we need UAT. If an app is made to spoil the user, then QA testing only is not enough, it needs to add the user perspective to see whether the user really gets a good experience when using the app. For instance, a new feature is developed and it is about a bike that can be used to go around after the user scan the barcode to unlock. QAs probably check based on its requirements and if all the functions work well. But as a user, UAT will go around with that bike without knowing the active area coverage. So when the bike is approaching outside area coverage, UAT will check whether the system already has the warning pop up for it. UAT will definitely ask other things other than its functionality such as an emergency number to call if let say the bike gets locked in the middle of the road.
Or let say an app is redesigning its home screen that looks completely different than the previous one. Again, since UAT test as an end user, they will explore the experience as well as the UI and functionality of all the buttons. Ensuring there is no slanted or weird display, all buttons are redirecting user to the right screen, and the new design is not confusing the user. And when there are things that do not “spoil” the users, that’s when UAT has to stand their ground to make sure the app gives the best user experience because UAT are hired for their attention to details, analytical thinking, and the ability to communicate with diplomacy as they interact not only with QA but other stakeholders like PM and if necessary VP of Product.
Currently, Bukalapak has twelve people as UAT and — since the squads are growing — we look forward to have eight more people by the end of this year.