Use IDE of your choice: Setting up Mutagen for Local-Cloud File Synchronisation

In our company, we’ve implemented a custom Cloud IDE infrastructure designed to optimise development workflows. Each team member is allocated a dedicated space, providing an individualised environment for cloning and managing Git repositories. Within this environment, a Node.js runtime is integrated, enabling the execution of code in alignment with our application ecosystem. Moreover, we’ve incorporated a VSCode server that empowers you to access and utilise the full capabilities of IDE directly within your web browser, facilitating a seamless and efficient coding experience.

Cloud IDE allows greater collaboration and accessibility. Developers can work on the same project from different locations and devices, without having to worry about installing and updating software, syncing files, or resolving compatibility issues. A cloud-based IDE also provides a consistent and secure environment for development, testing, and deployment, as well as features such as version control, debugging, and code analysis. Additionally, it can reduce the cost and maintenance of hardware and software resources. Scalability is another key benefit, as cloud-based IDEs can easily accommodate changing project demands by provisioning additional resources on-demand. Lastly, they promote consistency and streamline onboarding processes, as team members can work within the same development environment regardless of their physical location, ensuring uniformity in toolsets and configurations.

While there are undoubtedly numerous advantages to using a Cloud IDE, it’s essential to provide a balanced perspective. Let’s shift our focus from singing its praises and address some of the concerns that some of us had.

In our development setup, we employ a mono-repository for our core product, housing a vast number of files. While using VSCode in a web browser or even having it installed locally but connected to the cloud offers advantages, it does come with some drawbacks. The performance can be notably slower than working with files stored locally due to the inherent latency associated with cloud-based access. This dependence on a stable internet connection adds an extra layer of vulnerability, as any disruptions can impede productivity and create challenges for developers. Additionally, the necessity of connecting to a VPN to access our Cloud IDE introduces further latency and the potential for connectivity issues, which can impact the stability of our development environment.

Another consideration for some of us was the choice of IDE, specifically VSCode. Let me clarify, VSCode is undoubtedly a robust IDE, but we have been long-time users of JetBrains IDEs, which, in our opinion, offer a superior experience. After relying on JetBrains IDEs for several years, the familiarity and extensive feature set made it our preferred choice.

Initially, we experimented with the JetBrains Gateway solution. This approach involved installing an agent on the Cloud IDE server and then utilising the IDE on my local machine to establish a connection. However, we encountered performance issues, and it proved to be equally slow or perhaps even slower than using VSCode. Consequently, it became clear that this was not a viable option for me.

The logical step forward was to explore a solution that allowed us to keep our mono-repository on a local machine synchronised with our Cloud IDE, ensuring that any changes we made were seamlessly reflected. Several tools cater to this requirement, but Mutagen emerged as the perfect fit for our needs, and the best part is that it’s free. It provides real-time file synchronisation and flexible network forwarding for developers, extending the reach of local development tools to cloud-based containers and infrastructure.

Now, let’s dive into the technical aspects. In the following section, we’ll provide a straightforward walkthrough of our Mutagen setup. Keep in mind that since we primarily use Macs, if you’re working on a different platform, you’ll need to refer to Mutagen’s platform-specific documentation for guidance.

To begin, the first step is to install Mutagen on your system.

brew install mutagen-io/mutagen/mutagen

Once you’ve completed the installation process, consider creating a global configuration file. In our scenario, where we’re synchronising a single repository, centralising our configuration within a global file proves to be a logical choice. This approach simplifies synchronisation sessions command also.

Here is a config (YAML) that we currently use:

sync:
  defaults:
    mode: "one-way-safe"
    ignore:
      vcs: true
      paths:
        - "node_modules"
        - ".idea"
        - ".vscode"
        - ".husky"
        - ".DS_Store"
        - "build"
        - "coverage"
        - "junit.xml"
        - ".eslintcache"
        - "cdk.out"

Our objective was to synchronise changes exclusively from our local machine to the Cloud IDE, without the need for two-way synchronisation. To achieve this, we opted for the one-way-safe mode.

Here is a break down of modes available:

• two-way-safe: In this bidirectional synchronisation mode, both endpoints are treated with equal precedence, and conflicts are only automatically resolved if they don’t result in data loss.
• two-way-resolved: This is the same as two-way-safe, except that the alpha endpoint automatically wins all conflicts, including cases where alpha’s deletions would overwrite beta’s modifications or creations.
• one-way-safe: In this unidirectional synchronisation mode, changes are only allowed to propagate from alpha to beta. Deletions on beta are overwritten by content from alpha, but modifications and creations on beta can’t be overwritten by alpha (unless both endpoints have made the same modifications or created the same content). Conflicting contents on beta that can’t be overwritten will be recorded to the session state. Extra content on beta that doesn’t conflict with contents on alpha is simply ignored.
• one-way-replica: In this unidirectional synchronisation mode, beta becomes an exact replica of alpha. Any modifications or additional content on beta are instantly overwritten or removed, respectively. No conflicts can occur in this synchronisation mode

Additionally, we configured the synchronisation process to exclude certain directories such as version control repositories and node modules, among others.

However, it’s important to note that some tasks still require direct access to the Cloud IDE console. For instance, installing node modules and running applications must be done there. It’s crucial to exercise caution when working with files that are set to sync from your local machine (and are not on the ignore list), as any modifications made on the Cloud IDE side can lead to conflicts. To prevent such conflicts, any files generated as a result of running or building applications within the Cloud IDE should be added to the ignore list.

The final step in this setup process is initiating the synchronisation session, which can be accomplished with a straightforward command.

mutagen sync create --name=yourProjectName ~/sourceRepoPath cloudide:~/destinationRepoPath

cloudide that is in the command above is something that we have configured in ~/.ssh/config . If your setup is different you would have to refer to Mutagen SSH documentation. Mutagen also supports different transport methods and forwarding.

Once you’ve executed this command, you’re all set to begin working on your local machine without concerns about editing files on the Cloud IDE. Notably, there’s no need to interrupt or pause the synchronisation session. In fact, we’ve been running it seamlessly for weeks without any interruptions. The sync session automatically reestablishes connection to the Cloud IDE whenever you wake your machine or connect to a VPN. To keep an eye on the status of your sessions, you can utilise their mutagen sync monitor command, a tool we also run constantly.

For other commands you can check getting started page.

We decided to put the speed of synchronisation to the test by switching to some very old branches, and we were truly impressed by how quickly the files were synced to the Cloud IDE.

In conclusion, we couldn’t be happier with our new setup. Mutagen’s fast file synchronisation and the freedom to work with our preferred IDE have revolutionised our development process. This winning combination has not only saved us time but also boosted our overall productivity. We’re thrilled with the results and eager to explore further possibilities with this efficient synchronisation solution.