What it’s like to work as a Software Engineer (Infrastructure) at CSIT

Software engineering is getting more complex. Engineers have access to a huge variety of tools and services to build their products. To continuously deliver value and ensure an excellent developer experience, companies use managed services from commercial cloud providers (e.g. AWS, Azure, GCP) and SaaS companies (e.g. Auth0, DataDog, Slack). What would happen if a company could not use any services on the Internet?

CSIT develops products to advance the national security interests of Singapore and our work is highly classified. The consequence is an air-gapped infrastructure that imposes a large cost on developer productivity. To address this, the Cloud Infrastructure and Services department develops and maintains the core infrastructure and services that enable engineering teams to efficiently develop and deploy their products.

What does a Software Engineer (Infrastructure) do?

The exact responsibilities of a Software Engineer (Infrastructure) vary between teams. My team focuses on developer platforms and our key areas of work are:

API management platform: An API management platform centralises the access and monitoring of an organisation’s internal APIs. It allows teams to add authentication, authorisation, request logging and more to their APIs in a consistent manner. Our API management platform is built by integrating open-source software (Kong, Keycloak, OpenSearch).

Container platform: Containerisation is one of the biggest trends in software development and the industry has settled on Kubernetes for container deployments. Unfortunately, Kubernetes has an infamous learning curve. We provide a managed Kubernetes experience using Rancher to allow teams to focus their efforts on feature development.

Productivity tools: This refers to tools that developers use when developing their products. For example: collaboration, knowledge management, source control, CI/CD pipelines, project templates, software repositories and so on. We deploy commercial products (Artifactory, Atlassian Suite, Gitlab, Mattermost) and focus on integrating these products to provide a seamless developer experience.

What I’ve worked on

Over the past two years, I’ve had the opportunity to work on a wide variety of products and technologies. Some of the highlights are:

• Software repositories: We deployed Artifactory as a single platform to consolidate and reliably sync most software packages. This allowed many teams to deprecate their self-hosted software repositories, saving hundreds of engineering hours. Additionally, we automate around Artifactory to continuously sync packages from the Internet.
• Storage on Kubernetes: Deployment of block (Rook-Ceph) and object storage (Minio) on Kubernetes. This included testing (e.g. stability, performance, failure handing), upgrades, monitoring and capacity management.
• Certificate management: Our previous process for acquiring SSL certificates was tedious and required multiple approvals. We used cert-manager and Vault to automate certificate signing for all services on Kubernetes.

Thoughts on my current role

Coming from a background in building user-facing products, there are notable differences between the roles:

• Learning: Infrastructure work requires breadth and depth across many technologies and fields. There is always something new to learn or improve upon and this is one of the most satisfying aspects of the role.
• Freedom to build: Our users are engineering teams and we can build independently of business requirements. This comes with the added responsibility to prioritise the best set of features for engineering teams.
• Operations: All products depend on the platforms run by infrastructure teams. As a result, infrastructure teams place great emphasis on monitoring and improving the reliability of their platforms. Detecting the root cause of issues may take weeks, especially for issues with distributed systems that are difficult to reproduce. While some engineers relish the challenge (akin to solving a shifting puzzle), others may find feature development more appealing.
• Indirect impact: The impact of well-run infrastructure is indirect compared to a well-made product. Infrastructure is expected to “just work” and is often only noticed when something goes wrong (e.g. ISP error making Google unreachable).

I enjoy the technical nature of my work. I’ve gained a deeper appreciation of how the Internet works as well as the incredible engineering required to develop a cloud platform. Much of this is due to CSIT running its own infrastructure. Instead of calling an API or using a managed service (like on the Internet), we deploy and run the services for engineers. If you’re interested in a deep technical role that contributes to the national security of Singapore, CSIT is the place for you. We are hiring!