Adapting to Complexity: Autonomy and Strategy in Software Testing for Public Service Platform

Writers: Petrisia Meiga Natalia and Toyib Abdullah

Testing public service applications over the past four years has been an exciting journey for the Software Quality Analyst (SQA) team at GovTech Edu, with each product serving a unique user group. As a result, each tribe (team) naturally adopts its own distinct testing approach. In our pursuit of best practices that could be universally applied across all tribes, we discovered that no single best approach perfectly addresses the needs of every product.

Instead of mandating uniform best practices for our day-to-day testing, we give teams the autonomy to determine the best testing approaches for the specific products they handle. This approach calls for a testing strategy that can effectively guide the SQA team in performing purposeful and targeted testing.

Overview of our test strategy thought process

The framework outlined above represents the approach we use to structure our testing strategy thought process. This flow serves as a heuristic — flexible and adaptable to the needs of each team. Not every node must be completed; instead, they act as a guide or checklist to assess relevance.

This framework was developed through extensive observation of testing processes across various products that have been developed by GovTech Edu, combined with insights from books such as *Lessons Learned in Software Testing* and the *Rapid Software Testing* curriculum.

The key focus of the framework is the objective or mission of the testing to be conducted. This is crucial because different objectives or missions lead to varying quality criteria and testing techniques. Below are examples of objectives we’ve encountered in our work.

Technical Mission

• Migration : To transition data, systems, or applications from one environment to another (e.g., from an on-premise system to the cloud) without losing functionality or data integrity. The focus is on ensuring smooth continuity during and after the migration process.
  Precedent cases : Sistem Informasi Sumberdaya Terintegrasi (SISTER)
• Architecture change : To update or modify the system’s architecture to improve scalability, performance, or maintainability. This can involve adopting new technologies, reconfiguring infrastructure, or shifting to a more modern architecture to meet future demands.
  Precedent cases : Asesmen Nasional Berbasis Komputer (ANBK)
• System recovery : To ensure that systems can recover from failures, disasters, or unexpected interruptions swiftly and efficiently.
  Precedent cases : Kartu Indonesia Pintar — Kuliah (KIPK)

Non-Technical Mission

These categories can be

• User-driven: To develop products based on user needs and feedback. This approach prioritizes enhancing user experience (UX), functionality, and usability to meet the specific demands and preferences of the end users. Precedent cases: Almost all PMM features
• Policy-driven: To ensure the product adheres to organizational, industry, or legal policies and regulations (e.g., security standards, data privacy laws). Policy-driven development aligns the product with compliance requirements and governance standards. Precedent cases: data matching SISTER, ARKAS
• Branding driven: To align the product with the company’s branding strategy, which may involve rebranding efforts and changing product names, logos, or designs. The goal is to maintain brand consistency across platforms. Precedent cases: INA Digital
• Content/Data-driven: To focus on the content that supports the product, such as public reports, glossaries, or FAQs. This development ensures that the product’s informational resources are accurate, comprehensive, and accessible to users, enhancing overall usability. Precedent cases: Rapor Pendidikan, Portal Data

As an example of our testing experiences, we will share two cases in more details: how we streamlined the SIPLah testing process to uphold partner quality, and our involvement in data recovery for KIPK.

SIPLah

The Ministry of Primary and Secondary Education has again initiated the selection process for SIPLah Pengelola Penyelenggara Perdagangan Melalui Sistem Elektronik (PPMSE) for 2021–2023. This process aims to provide opportunities for potential SIPLah PPMSE, including both online marketplace system providers and potential suppliers of goods and services.

In this regard, GovTech Edu plays a key role in developing the SIPLah priority platform in collaboration with the Ministry, by issuing the Kerangka Acuan Kerja (KAK) for the competition or selection process of SIPLah PPMSEs for the next period. The KAK is a document containing guidelines and instructions for implementing a project or activity, covering objectives, scope, schedule, and expected outcomes.

After the Product and Policy & Transformation teams create the KAK, GovTech Edu collaborates with Pusdatin, the Biro Umum (General Bureau), and the BLPT of MoECRT to conduct a briefing for potential PPMSE, explaining the business process, system architecture, scope, and development work plan for SIPLah.

Prospective SIPLah PPMSEs are required to develop SIPLah following the KAK and pass the Technical Development Testing based on the test scenarios created by the GovTech Edu team.

Testing Mission

With this, our mission becomes clear: to make the User Acceptance Testing (UAT) process more effective, efficient, and scalable. The goal is to ensure that it can be easily implemented across multiple PPMSEs without compromising product quality, while also ensuring that each PPMSE adheres to the standards outlined in the KAK and existing regulations.

This means designing a UAT framework that maintains rigorous quality assurance and streamlines processes to reduce time and effort for all parties involved. By doing so, we aim to foster collaboration with PPMSEs while consistently delivering products that meet both technical and business expectations.

Testing Technique

UAT (User Acceptance Testing)

SIPLah PPMSEs are required to develop SIPLah in accordance with the Terms of Reference (KAK) and pass technical testing based on the test scenarios created by the GovTech Edu team, which is conducted through a UAT session.

UAT is conducted comprehensively and attended by Pusdatin, BLPT, GovTech Edu, and the PPMSE development team. During this session, Pusdatin’s role is to verify that all system features and functions operate properly according to the established test scenarios and business processes.

Initially, much of the time in SIPLah development was spent on UAT sessions, such as setting up environments, test accounts, and test data. A testing document template was created to streamline this process, containing information and requirements for PPMSEs to meet, making the UAT process smoother.

Example of test data prepared by PPMSE

Example of SIPLah test cases

Testability

1. Information Availability

The testing document includes general test scenarios designed for all PPMSEs while still following the guidelines of the KAK, Change Requests (CR), and existing regulations. The language used is simple Indonesian to ensure clarity for all parties involved and minimize misunderstandings, including those who are not part of the QA team but are involved in SIPLah development. This is important as each PPMSE has a different User Experience (UX) in the purchasing process or with additional features.

For example, regarding the standardization of NPWP in the KAK, it states, “PPMSE is encouraged to require a standard NPWP that can be used by providers in the system, which is the NPWP issued nationally by the DJP, not a regional NPWP.” Therefore, the test scenario was created to allow for the input of a 15–16 digit numeric NPWP, considering DJP rules that require a 16-digit NPWP and the fact that some corporate NPWPs still have 15 digits.

Another example concerns the standardization of tax and price calculation formulas. The KAK states, “When a provider enters a price, the provider can see a breakdown and explanation of the entered price, including DPP, VAT, income tax (PPh), and the final amount the provider will receive, calculated automatically by the system.” Not all PPMSEs have a price breakdown page when entering items, so this scenario is optional.

All these documents were compiled into a Google Spreadsheet for easy access by PPMSEs, Pusdatin, BLPT, and other stakeholders. A dashboard sheet was also created to monitor development progress and the readiness of UAT for each PPMSE.

Dashboard to track PPMSE development progress

2. Testing environment and process

Starting in 2021, the development environments were separated into staging and production environments. This separation was implemented with the expectation that User Acceptance Testing (UAT) could be conducted more reliably. Additionally, it allows greater flexibility in using test data without affecting the production environment.

With this strategy, we can cut down the UAT process from around 1 day to 2–3 hours per PPMSE by eliminating test data preparation and test environment setup beforehand.

Additionally, the UAT process was enhanced by introducing a division between Pre-UAT and UAT. Pre-UAT serves as a preliminary testing phase where a thorough review is conducted to ensure that the PPMSE has completed the features according to KAK. This improvement significantly optimized efficiency, as running all test cases during UAT could previously take up to one week. With the implementation of Pre-UAT, the testing time was reduced to just 4–5 hours per PPMSE, making the process far more effective.

KIP Kuliah

KIP Kuliah (Kartu Indonesia Pintar Kuliah) is a government initiative managed by the Ministry of Higher Education, Research, and Technology. Its purpose is to expand access to higher education (bachelor’s and diploma programs) for academically high-achieving students from low-income or vulnerable families across Indonesia. The number of applicants yearly exceeds 835,000, with a total of 850,000 recipients since 2020.

In June 2024, Temporary National Data Center (PDNS) system experienced a ransomware attack, encrypting all stored data, including the KIP Kuliah database. As a result, students were unable to apply for KIP Kuliah, existing recipients could not view their disbursement data, and universities were unable to process benefits and distribute funds.

The GovTech Edu Edu team was mobilized to recover the KIP Kuliah data from available backups and migrate the system to the Pusdatin public cloud. With a tight deadline of one month, the management team formed a cross-functional task force, including the QA team.

Testing Mission

The primary mission of the KIPK project was to restore the data lost in the PDNS incident within a month. Additionally, the project aimed to enhance the security of the information system. Therefore, the data was successfully recovered, and the system was also made more resilient to cyberattacks.

Quality Criteria

Our primary quality criteria is to ensure that there is no broken workflow resulting from the newly constructed functions or migrated data. We focus on testing the core functionalities to guarantee a seamless and efficient user experience with the system. Additionally, we ensure that users receive informative error messages if any errors occur due to missing or unretrievable data.

In addition to functionality, system performance is another critical priority. Given that the application has been in maintenance mode for a month, we anticipate a spike in user activity during the release phase as many users log in to access their accounts. To prepare for this, we have implemented performance testing, to ensure the system can handle the expected traffic surge. Our goal is to maintain fast response times and system stability, even under heavy usage, ensuring a reliable and smooth experience for users.

Testing Technique

1. Exploratory testing

Exploratory testing is conducted by freely and creatively exploring the system, allowing testers to better understand its behavior. This approach often uncovers unexpected bugs that might not be identified through standard testing methods.

Through exploratory testing, our team gained a broader knowledge of KIP Kuliah system’s key features. This deeper understanding has been instrumental in shaping our next testing strategies and ensuring a more thorough approach.

During the exploration, we documented several points that needed clarification with the Puslapdik team. These questions were addressed during daily meetings to ensure alignment. Additionally, we actively validated our understanding of the system’s workflows to ensure accuracy and consistency.

2. Experienced-based testing (EBT) approach

Experienced-Based Testing (EBT) is a testing method that relies on the tester’s knowledge, experience, and intuition. This approach enables testers to design test scenarios that are more realistic and address complex potential issues effectively.

An example of EBT implementation can be seen when testing a form requiring the input of a mother’s name. Based on their experience and intuition, testers recognized the wide variations in how a mother’s name might be written. Some of the test scenarios include:

• Inputting names with academic titles
• Abbreviations for family names or given names
• Minor typos
• Other common variations

The SQA team tested variations of the mother’s name input using several similarity score thresholds, such as 95%, 90%, 85%, and 80%, with test data like “Maria Br. Sinaga” compared to “Maria Sinaga.” The results recorded whether the system could recognize the input name. To improve efficiency, the team utilized a simple script to log the test results automatically and compiled the data into a table for easier understanding and analysis. This process raised a critical question: “How tolerant is the system toward variations in the mother’s name input?” The question prompted in-depth discussions within the team, helping to refine the testing approach and ensure the system’s ability to handle diverse inputs effectively.

Example of exercise of test data to search for optimum similarity using self custom PHP program created by Petris

3. Performance Testing

Performance testing was conducted to anticipate the high enthusiasm of students registering for the scholarship program, especially as the website had been in maintenance mode during the recovery period. This step was taken to ensure the system could handle traffic surges effectively.

During the planning phase, the acceptance criteria were set as follows:

• Error rate: 0% across all endpoints tested during the load test.
• Total throughput: approximately 1600 requests per second (rps).

However, during the testing phase, the error rate reached 7.86%, potentially impacting around 10% of users.

The test results and potential risks were communicated to stakeholders. Despite the relatively significant error rate, stakeholders deemed the level of risk acceptable. To mitigate potential issues, the following actions were prepared:

• Displaying a more informative error page for users.
• Adding virtual machines (VMs) as needed to increase system capacity.

Example of performance test report that needed to be analysed further

Testability

Testability refers to the extent to which software can be effectively tested. The easier an application is to test, the higher its testability. High testability directly contributes to the quality of the application as it enables comprehensive testing. Furthermore, easily testable applications enhance developer productivity by streamlining the testing process. After thoroughly exploring KIP Kuliah, our QA team identified several features that could be modified to improve their testability.

1. External Service and Test Data Challenges. One example of this feature is in the registration process, where data is validated against other services and this data is actual. Using personal data is not feasible as our data may not meet the registration criteria. We brought this discussion to the team and eventually found a simple yet effective solution to facilitate testing of the registration feature: data team would hold back a portion of the data during the data flow to the staging environment.. By taking this approach, the registration process is not hindered, and the flow of checking registration data against other services can still be simulated accurately.
2. Addressing Limitations of Non-Repeatable Functions. Before conducting performance tests, we identified several potential roadblocks. We expressed our concerns to the team and suggested alternative solutions such as disabling the captcha feature and sending the grid service. By disabling the captcha, our QA team could create performance test scripts more efficiently without the need for additional workarounds. Temporarily deactivating the send grid service prevented email sending during performance tests, significantly reducing development costs and avoiding sending emails to real users.
3. Limited Test Data and managing account state. During KIP Kuliah performance testing, we needed to execute multiple iterations for various endpoints. Each test run required 300,000 accounts. However, when calling certain endpoints, user states would transition, hindering us from achieving our performance test goals. This posed a significant challenge in reusing these affected accounts. SQA team devised a solution to restore these accounts to their original state by directly modifying the database using SQL queries. Fortunately, we conducted the load tests in a staging environment, allowing for safer execution. While the staging environment was configured to resemble production closely, it still provided a safety net. Additionally, the SQL queries created for this purpose could be adapted for functional testing, empowering the QA team to become more self-sufficient and reduce reliance on the data/infrastructure team.

Post release KIP Kuliah

Our team established a war room to facilitate internal coordination to ensure a smooth release. Prior to deployment, we conducted a final smoke test on the features using an environment similar to production, complementing earlier testing done exclusively on the staging environment. Post-deployment, we performed sanity testing to verify that the features functioned properly in the production environment.

We also implemented intensive monitoring through internal tools, social media platforms, and Telegram groups to gather real-time user feedback. Internal monitoring revealed that traffic levels were close to our load test targets, yet the error rate remained below 1% — significantly lower than the 7.86% observed during load testing. Additionally, observations from social media, Telegram groups, and the customer operations dashboard indicated minimal user complaints about functional disruptions.

Observation on the launch day

Closure

In a fast-paced environment with layered bureaucracy, high complexity, and ambiguous requirements, having a precise and well-thought-out software testing strategy is crucial . Striking the right balance with acceptable trade-offs and effectively communicating potential risks to stakeholders are equally important.

This approach highlights the critical role of the Software Quality Analyst in the product development lifecycle. By providing stakeholders with meaningful insights and data, the SQA team empowers them to make informed decisions about whether a product is ready for release. In this way, SQA serves as an enabler and supporter of a collaborative environment where risk management and product quality work hand in hand.