Remote Development at Hiver

For years, Hiver engineers have been writing code on their local laptops or AWS Cloud9. We grappled with many moving parts in our development environment that impede our workflows. In Particular, we have a large system (written in PHP and Python), around 20 microservices (written in Python), each with its own set of DBs. Our shared databases further complicate the setup.

Each microservice and its associated systems use a shared docker-compose.yml to define the stack. Developers create a docker network to enable services to interact with one another, as well as deploy a local stack to mimic AWS services in development (SNS, SQS and others).

As the team size increased we faced these challenges:

• We performed manual installations that might take days before we could work on the systems.
• Docker Compose deployment was resource-heavy
• Running more than a couple of microservices concurrently was almost impossible.
• Vague service dependencies.

These issues reduce productivity. Moreover, we found it difficult to maintain and keep the system up-to-date as new microservices were added.

Therefore, we seek a solution to resolve existing issues and allow us to deploy microservices at scale. We needed a new solution that fulfills these requirements

• Easy to set up and replicable
• Emulate our production environment’s structure and technologies,
• have a fast inner-loop workflow

It was also important to maintain a development experience that closely resembles the local environment. For example, hot reloading and debugging should work seamlessly.

Last but not least, the new system must run smoothly on resource-efficient laptops 🙂

Enabling Remote development

Hiver’s web application and email sync engine contribute to more than 50% of the entire engineering team’s code changes. The number of steps to make the system run smoothly is enormous. To enable remote development, we had to automate the entire setup process and orchestrate application deployment on Kubernetes.

Our principle is to keep the remote setup simple. Hence, we built a CLI tool that allows us to create and dispose of development sandbox environments in one step.

Responsibilities of hiver-cli tool.

• Create an area with default services running, reducing the setup time from days to minutes.
• Change services running mode to dev mode
• Sync files between developer machines and remote servers every time a file is saved.
• Creating an SSH terminal with remote machine to run terminal commands if required
• Enable port forwarding to access remote services and add breakpoints.

With `hiver-cli create <area-name>` developers can reserve an area for them in the Kubernetes cluster.

hiver-cli create <area-name>

With `hiver-cli start-dev <service-name>` developers can start working on their service. All the processes on the cloud, where the developer has full control over what services to run and deploy.

hiver-cli start-dev <service-name>

How we built hiver-cli

Hiver-CLI is architected with open-source softwares. We assembled appropriate software and packaged them into a single applicable tool for Hiver’s engineering team.

1. Kubernetes: for deployment and management of containerized applications.
2. Devspace: DevSpace is an open-source developer tool for Kubernetes that lets you develop and deploy cloud-native software faster.
3. Helm Charts: A Helm chart is a collection of files that describe a related set of Kubernetes resources. As an example, deploying images, creating volumes, networking, and everything related to running containerized services.

We chose DevSpace because:

• It helped us sync code from local to remote pod
• We can abstract all networking like port forwarding
• It’s helpful for installing helm charts

We went remote with remarkable results

We managed to establish a remote environment by standardizing development requirements for all Hiver engineers. Our efforts removed substantial overhead and reduced the cost of context switching. Engineers can request an isolated environment for each service they work on, improving productivity, flexibility, and scalability.

As a result:

• New hires can start developing without a steep learning curve.
• Much of the prerequisites for development were eliminated, bringing down the setup time from days to mere minutes.
• Developers were able to selectively choose services to run and test.

What We’re Working On

We are currently implementing auto-sleep, which will put the development environment to sleep when not in use. This will help to better manage infrastructural costs..

Closing Notes

An efficient, reliable, and scalable development workflow is critical to the success of fast-paced technology companies like Hiver. OurDev Infrastructure team is continuously making the development experience seamless for all engineers by leveraging cloud resources — Join the Hive to experience and shape the future of software development.