Enabling Machine Learning Operations (MLOps) for WOG with DSAID DE
My Software Engineering Internship with Analytics.gov
Hi! I am Seeu Sim, an undergraduate in my penultimate year majoring in Computer Science at the National University of Singapore. As an aspiring software developer in the web and machine learning space, the Analytics.gov project under GovTech’s Data Science and Artificial Intelligence Division (DSAID) greatly appealed to me, and that’s where I decided to pursue an internship as a Software Engineer .
GovTech’s Agile, Bold and Collaborative culture stood out to me, among other firms and this also motivated me to choose GovTech to experience this. It’s been a great experience throughout the past six months, I’ve gained tons of valuable experience working on this project and within the DSAID Data Engineering (DE) team, and I’d love to share more about it here.
Overview of Analytics.gov
Analytics.gov is a data exploitation platform for the Whole-Of-Government (WOG). It is a secured platform that enables public agencies to analyse data and develop insights through an easy-to-use and readily available environment accessible from their government-issued machines.
Analytics.gov offers a wide range of tools and features that can be customised to meet the needs of individual agencies, including a virtual environment for data analysis, a wide range of analytical tools, Integrated Development Environments (IDEs), and Machine Learning Operations (MLOps) capabilities.
Here are some of the key features that Analytics.gov offers:
● A diverse range of analytical tools, including visualisations, statistical analysis, and machine learning capabilities.
● Ample computing power for handling computationally-intensive analytics projects.
● Access to readily available and up-to-date code libraries and packages for efficient data analysis.
● Seamless code sharing and collaboration among users within and across agencies.
● Machine Learning Operations environments and compute with AWS SageMaker
Expanding my technical horizons with what I built
Throughout my internship as software engineer, I was assigned many tasks to incorporate, improve or add new functionalities to the teams’ product. I’ll elaborate more about them in the following sections.
1. Web Portal
To make interactions with the MLOps environment and resources on Analytics.gov more accessible, I was tasked with extending the product’s landing page to include a custom Web Portal. The objective of this task is to make it easier for the users to view, manage and launch resources unique to their product teams securely, and in an interactive manner .
The task started with the need to gather the necessary product requirements to build each component of this full-featured portal. For that, I was entrusted by the product team to drive the discussion with my end users. Under the guidance from my product manager and technical lead, I immersed myself in a cycle of technical development and feedback gathering, in order to develop iteratively and optimise the existing workflows of the portal’s end-users. Through this experience, not only I had gained valuable insights into the user workflows, I could also better appreciate the motivation behind our product’s vision and goals.
The development work I undertook culminated in the web portal, as displayed above. This may seem like a ton of User Interface (UI) and frontend work, and it is. However, I was blessed to have the opportunity to set up various workflows within the project to develop UI in a scalable manner, such that the many views that you see above are simply the composition of various custom UI components within a layout.
I also gained the opportunity to experiment with shadcn-ui, Radix UI, TailwindCSS, React and NextJS, to create reusable data layers, components and layouts that are custom to our product and brand. This allows for consistent styling and theming (including light and dark mode), while still maintaining full compatibility with browser APIs for accessibility and enhanced User Experience (UX).
2. Microservices, Infrastructure and CI/CD
To power the web portal which I built, I was also tasked with creating the backend integrations that powered the views and actions within the Web Portal. This was necessary to provide accurate, insightful data to the users in a secure manner, and also ensure that our service could scale efficiently to support thousands of users.
As the portal serves as an interface between resources on Amazon Web Services (AWS) SageMaker and the users, it requires many integrations with AWS’ Software Development Kit (SDK), as well as the creation, updating and deletion of various resources on AWS in a repeatable and scalable manner.
Collaborating with the engineers on my team, I helped develop multiple microservices on AWS Lambda and Step Functions, as well as custom access indexes for AWS DynamoDB. This provided speed, reliability and scalability for our application. I also automated and optimised the Continuous Integration/Continuous Deployment (CI/CD) of our services and underlying infrastructure with respect to code changes and applied them to our AWS environment with Infrastructure-As-Code templates for replicable deployments.
These were eye-openers for me, as I had little prior experience working with cloud providers such as AWS and their services. I also gained valuable experience in working extensively with cloud services as well as the development lifecycle and CI/CD of services and infrastructure on the cloud. I’m grateful to the team for giving me the chance to work with cloud services on such a large scale, and this experience will greatly help me as a developer in today’s Cloud-Centric landscape.
3. Security in the Cloud
As our application primarily serves government officials who work with data, security is of paramount importance in designing our application. I was tasked with a few workflows to secure access to our application and yet maintain the same intuitive User Experience for our users.
I was tasked with integrating the same, secure Single Sign On (SSO) experience that government officials enjoy for other WOG applications into the application, securing our application’s access to only the users which are authorised to access them. I also integrated the AWS Key Management Service (KMS) and other measures into the frontend portal, providing encryption-in-transit as well as allowing users to secure their access to their resources on our AWS environment.
I also created the required Virtual Private Cloud (VPC) endpoints to allow access to native AWS APIs and actions in an air-gapped environment without public internet connection. This further secured the users’ data access to within our application’s private network, and also protected the application and users’ data against malicious access with various access policies.
Through this, I gained valuable experience on security in modern applications, and learned more about best design principles in securing application access while still providing the same interactive experiences to our users.
An all-rounded internship experience — Engaging key stakeholders and end users
While development work is key in actualising capabilities for end-users, I also learnt during my internship stint that product outreach and engagement efforts are equally critical to ensure user growth, through educating potential users about any newer capabilities of our platform and how it can help accelerate their agencies’ data exploitation and MLOps adoption.
In this regard, the product team organised a couple of product sharing webinars and technical workshops to engage key stakeholders and end users across WOG who are interested to onboard to AG, and I was privileged to have been a part of them.
One notable experience for me would be the hands-on workshop which we jointly organised with AWS where prospective agency users registered for the session to try out Machine Learning and Generative AI features through our platform. I conducted a segment of the workshop where I gave a live demonstration on the features that our web page provides, as well as guided the end users accordingly on how to navigate around the portal throughout the workshop.
As an intern, this was a rare opportunity to take charge for the development of a feature and showcase it to external stakeholders, and this was an experience I greatly treasured. It also allowed me to get valuable feedback from our users apart from that of our alpha users, and greatly accelerated our web app feature development.
Life outside of work
While the work we do in DE is interesting, what sets DE apart from other organisations is certainly the efforts invested into cohesion and team-bonding, and I’m glad to have benefited from that.
During my internship stint, I could always look forward to exciting activities such as DE’s and even DSAID’s cohesion activities, such as bowling, potlucks, and board games, to name a few. Through these events, I got to interact with other colleagues that were external to my work unit, and bond with them through team activities.
Key Takeaways — Building the ‘right’ product vs building the product right.
This internship has given me the avenue to explore and learn about various new aspects of software engineering, such as UI/UX, Cloud, ML and DevOps. Through the features I helped implement, I have certainly gained valuable technical expertise. In addition, the interactions I had with my colleagues have also honed my communication skills and professional etiquette.
This internship has also taught me the importance of concurrently engaging my external stakeholders and end users while I was developing, as it also helped me to iterate my work more effectively, and gave me valuable insights on exactly how the developments would maximise the value and the user experience of AG’s end-users.
This set of exposures differed greatly from my past internships — as it not only got me to focus on the ‘how’ but also the ‘who’ and ‘why’ I was doing what I did during my stint. This played a key role in my understanding that actual product success is never about just sticking to a static product plan and getting tasks done on time.
In this dynamic landscape, user requirements evolve rapidly, and as such building the ‘right’ product today might not be the ‘right’ one tomorrow. Working with Analytics.gov also taught me to embrace an agile and user-centric development approach so as to deliver a product that constantly maximises its value to users and actively does what matters for them. That way, we can always be sure that we are building the product ‘right’.
Parting Words
In short, my internship with GovTech has certainly given me the space to develop interesting solutions to tackle challenging problems, and I can’t have been more grateful for such an opportunity.
If you are looking for exciting problems that require innovative solutions, GovTech is certainly the place for you. You can find out more about GovTech’s internship programme here: go.gov.sg/GovTechInternship.
