Catalyzing Indonesian Lecturer Career Management through System Centralization

Writers: Muhammad Haris, M. Burhanudin Yusuf, Muhammad Imaduddin, Eka Mukti Arifah, Andreas Dwi Nugroho, Rachmad Akbar, Franova Herdiyanto, Fransiska Putri Wina Hadiwidjana, Muhammad Ghifary and all SISTER Engineers

TLDR

Through system centralization, we successfully catalyzed career management for lecturers in Indonesia, ensuring nationwide accessibility, seamless and accurate data integration, and enhancing scalability with a 67% cost reduction. This initiative created a scalable, reliable, and secure platform for managing academic careers across all universities in Indonesia.

Background

Lecturers, as academics, are essential figures in the higher education system. In Indonesia, the role of lecturer is two-fold: i) an educator and ii) a researcher, who has the duty to develop and deploy science, technology, and art to the community through the Three Pillars of Higher Education (Tridharma Pendidikan Tinggi) activities: education, research, and community services.

Each academic activity undertaken by a lecturer can contribute to advancing their career, with the ultimate goal of attaining a professorship. However, the diverse nature of these activities adds complexity to the reporting and justification process. Due to limited interoperability between systems, lecturers often need help with preparing and submitting redundant documents across multiple platforms.

Sistem Informasi Sumberdaya Terintegrasi (SISTER), owned by Direktorat Jenderal Pendidikan Tinggi, Riset dan Teknologi, was developed to address these challenges. It serves as a career management platform that consolidates all administrative documents related to lecturers and seamlessly integrates with the PDDIKTI ecosystem.

SISTER consisted of two platforms: SISTER PT and SISTER Pusat. SISTER Pusat, managed by the Ministry, acted as a central platform and served as the main hub for data synchronization from SISTER PT. On the other hand, SISTER PT was deployed on the local infrastructure owned by individual universities or clusters of universities and operated as a standalone application.

However, the decentralized architecture of SISTER presented several significant challenges:

1. Scalability and Reliability: The platform’s design was not optimized to handle high-volume requests, resulting in several bottlenecks, including unoptimized queries and infrastructure limitations. During program (Layanan) deadlines, users often had to access the platform at off-peak hours, like midnight, to avoid congestion.
2. Data Consistency: Data synchronization between SISTER PT and SISTER Pusat frequently caused issues, leading to inconsistencies between the databases of the two platforms. These inconsistencies could result in incorrect decisions for Layanan like Sertifikasi Dosen and Kinerja Dosen, affecting lecturers’ eligibility for Sertifikasi Dosen and their allowances (tunjangan fungsional).
3. Disparity in Access: Many universities lacked the resources to host SISTER on their internal systems, creating disparities in access, especially for smaller institutions. This limitation prevented SISTER from serving as the primary platform for managing lecturer activities across all universities. In some cases, universities hosted SISTER on laptops, operating them only when necessary, while others pooled resources to create mini-servers for deployment.
4. Security: This aspect spanned multiple layers, from insecure code implementations to vulnerable infrastructure. The separated deployment of SISTER PT at each university, without robust security measures, heightened the risk of data breaches and compromised the protection of sensitive information.

In collaboration with the PDDIKTI team, we aim to centralize the platform into a one-stop service for lecturers, addressing all the previously mentioned issues and delivering a solution that is highly reliable, secure, and fully integrated.

Topic 1: Migrating all the data into cloud

The previous architecture of SISTER was as follows:

SISTER PT was deployed and managed independently by each university, while SISTER Pusat operated as a centralized platform for the Ministry, serving as the central hub for synchronizing data from the SISTER PT systems of individual universities.

Our top priority was to migrate all data from both SISTER Pusat and SISTER PT to the cloud. SISTER Pusat housed a centralized database with extensive data accumulated since the platform’s inception in 2018, while SISTER PT involved around 3,000 individual databases deployed across various universities. Each system presented its own unique complexities. Most critically, this entire migration process must be completed with minimal downtime from the existing services.

We began by setting up a replication job to transfer the SISTER Pusat database to the cloud. This process captures real-time data changes in SISTER Pusat using SQL Server’s change tracking features. This step is essential to keep SISTER Pusat synced with the cloud environment, minimizing data gaps at the time of release.

Even with successful replication of all data from SISTER Pusat to the cloud, some data was expected to remain exclusively within the SISTER PT systems. This raised a challenge: how could we effectively replicate and merge all data from approximately 3,000 universities?

To address this, we developed two new endpoints for SISTER PT. The first endpoint temporarily disables login activity in SISTER PT, ensuring no data changes occur during the replication process. The second is a sync endpoint in SISTER PT, allowing direct data synchronization to the cloud. This endpoint can be manually triggered by the university’s admin or by our engineers. Using this approach, we successfully replicated the majority of SISTER PT data within three days, even handling various edge cases that required remote access to each SISTER PT server.

In total, we gathered 13 TB data.

Topic 2: Centralizing the system into a single hub

The decision to create a centralized platform also required merging SISTER PT and SISTER Pusat. Maintaining two platforms without distinct boundaries could lead to user confusion and negatively impact the user experience.

To address this, we centralized the authentication and authorization processes across SISTER PT and SISTER Pusat, enabling users to transition seamlessly between the two. Users can switch platforms simply by selecting their role, allowing for a smoother, more intuitive experience.

With the centralized platform in place, integration and synchronization between SISTER and the PDDIKTI ecosystem is now achievable, enhancing SISTER’s interoperability.

Using the same replication job developed for data migration, we implemented automatic replication between SISTER and the PDDIKTI ecosystem. This automation eliminates the need for manual syncing, allowing lecturer portfolios and personal data to be updated based on entries from other platforms like Neofeeder and Admin PDDIKTI. SISTER now performs near real-time syncs, significantly reducing manual tasks for importing data into lecturers’ portfolios.

The current centralized architecture is as follows.

Topic 3: Optimizing key processes

In terms of system reliability and cost efficiency, optimizing the platform is also a mandatory requirement. There are two main highlights of this process.

BLOB migration: a story of reducing 13 TB to 200 GB

SISTER functions as a career management platform that handles large volumes of unstructured data, including images and PDFs. Previously, these documents were stored as Binary Large Objects (BLOBs) within the database, which led to several issues, such as excessive database storage and memory usage, making cloud deployment cost-inefficient.

To resolve these challenges, we migrated all documents to Google Cloud Storage (GCS). The migration process for the BLOBs was carried out in three steps.

Adjusting SISTER Application

The migration involved two main processes: writing and reading. We also added extra metadata in the database to support this migration.

For the writing process, the application now saves new BLOB data to GCS, storing only the object name in the database for easy retrieval later.

For the reading process, the application first locates the object name in the database and then retrieves the corresponding document from GCS. To ensure a smooth transition, we implemented logic to handle both data sources: reading BLOBs from the database and from GCS. This allows existing BLOB data to remain accessible throughout the migration process.

The challenge lies in updating every line of code that writes to or reads from BLOB data. Our approach is to make minimal changes while still achieving our goal. To accomplish this, we leverage events in the Eloquent model, adding a hook in the saving function for each model, as shown below.

These adjustments check whether the BLOB is stored in the database; if it exists, the document is uploaded to GCS instead. A similar modification was applied to the reading process. This approach enables us to handle the migration seamlessly without the need to modify every line of code related to BLOB writing and reading.

Migrating the existing blob from the database to GCS

At this stage, we developed a script to copy all existing BLOBs from the database, convert them into document files, and store them in GCS. We ran this process multiple times until all BLOB data in the database were successfully saved to GCS, ensuring that no write operations were missed during the migration process.

Reduce database storage

In this final migration step, we began by validating that all documents saved in GCS matched the BLOB data in the database. Once consistency was confirmed, we deleted the BLOBs from the database, repeating this process for all records. This optimization successfully reduced database storage from 13TB to 200GB, lowering costs by approximately 67%.

Query and system optimization

SISTER PT was initially designed to manage requests for a single university per deployment, so its architecture and development were not optimized for high-volume, nationwide usage. With the centralized SISTER platform, we anticipate daily requests to range between 50–200 requests per second (rps). However, the initial load test achieved only 1 rps, highlighting the need for essential optimizations to effectively handle the expected increase in demand.

Core platform optimization

This phase encompasses a series of initiatives designed to elevate both the performance and security of our platform. We began by reengineering the caching mechanism, transitioning from a traditional file-based approach to a more robust, high-speed, in-memory storage solution using Redis. This shift not only accelerates data retrieval but also significantly improves overall system responsiveness under heavy load.

To bolster security, we replaced Apache with Nginx, an alternative web server renowned for its superior security capabilities and efficient handling of concurrent connections. This change not only hardens our defences but also optimizes resource utilization, leading to faster, more stable performance.

In tandem, we upgraded the PHP version, addressing critical performance and security issues. Alongside this upgrade, we introduced PHP-FPM (FastCGI Process Manager), a specialized PHP handler designed for improved scalability and enhanced handling of concurrent requests. This addition enables the platform to maintain high performance even as traffic and demand surge.

The final and most intricate step involved an extensive series of load tests. Through iterative testing and meticulous tuning of the web server configuration, we calibrated the platform to achieve optimal scalability and reliability. This thorough approach ensures that the system not only meets but exceeds the demands of our user base, providing a resilient, high-performance experience.

Optimized heavy query

Tracing every heavy query within the SISTER codebase is impractical. Therefore, our approach focuses on reactive query optimization. We have set up alerts for any SISTER page with high latency, enabling the on-call engineer to investigate and optimize the query as soon as an alert is triggered. Additionally, we monitor queries using Query Insights, allowing the on-call engineer to identify and refine resource-intensive queries proactively.

Our core principles for query optimization remain fundamental: (1) always use indexing, and (2) avoid nested queries, opting for Common Table Expressions (CTEs) or joins whenever possible.

Sertifikasi dosen flow optimization

Sertifikasi Dosen (Serdos) is a certification process ensuring university lecturers meet certain professional and academic standards. Lecturers who pass the certification process receive a certificate that validates their academic competence and quality as educators. Furthermore, certified lecturers become eligible foran additional monthly allowance, which offers financial incentives to upload high teaching standards and contribute to their academic disciplines.

Through this centralization effort, Layanan Serdos has become one of the most enhanced features within SISTER. Previously, maintaining data consistency and managing cut-off times were significant challenges due to inconsistent data synchronization between SISTER PT and SISTER PUSAT. With the centralized platform, real-time data cut-offs and comprehensive end-to-end monitoring are now possible, ensuring seamless data integrity and flow.

A major breakthrough has also been achieved in streamlining the text similarity check process for applicants’ short essays, dramatically improving its efficiency. Previously, this process was time-intensive, taking nearly a full week to complete. Through a re-engineered approach, we developed a hybrid method that combines the strengths of the Rabin-Karp and Longest Common Subsequence (LCS) algorithms. This tailored combination has allowed us to reduce the computational workload by an astonishing 99%. With this optimization, we’ve cut the processing time down to just 3 hours, achieving an extraordinary 40-fold improvement in performance over the original system. This improvement accelerates the application review process and enhances the overall user experience.

Impact

The centralized SISTER platform, released on August 10th 2023, has transformed higher education administration across the nation, bringing significant benefits:

Nationwide Accessibility: By removing infrastructure limitations, SISTER is now accessible to every university and lecturer in Indonesia. It ensures that even smaller institutions with limited resources can leverage SISTER to manage lecturer’s academic careers and portfolios.

Seamless Data Exchange: The integration between SISTER PT, SISTER Pusat, and the PDDIKTI ecosystem has streamlined data synchronization. This seamless data flow minimizes manual input, reduces redundancies, and ensures data is accurate and up-to-date across platforms.

Enhanced Scalability and Reliability: With optimized architecture, the platform now supports up to 200 requests per second, allowing it to handle a high volume of simultaneous users. This scalability improvement guarantees a reliable user experience, even under peak demand.

Cost Efficiency: By optimizing data storage, migrating to Google Cloud Storage and other optimizations, SISTER achieved a 67% reduction in monthly operational costs.

These improvements mark a milestone for SISTER, establishing it as a robust, efficient, and accessible solution for academic career management across Indonesia’s higher education landscape. Beyond being a mere technological upgrade, this transformation alleviates the administrative burden for university lecturers nationwide, enabling them to focus on what truly matters: teaching, research, and community service.

Acknowledgment

Special thanks to all GovTech Edu’s Higher Education team, Cloud Infra and Security team, as well as PDDIKTI Kemendikbudristek and Pusdatin Kemendikbudristek for their substantial technical contribution and feedback.