Mount Google Cloud Storage as File system on Windows and Non-Windows Systems

Google Cloud Storage

Google Cloud Storage is a managed service to store objects in cloud storage. Objects can be any blob. Objects in containers called buckets. Google Cloud Storage offers four storage classes, identical in throughput, latency and durability. The four classes, Multi-Regional Storage, Regional Storage, Nearline Storage, and Coldline Storage, differ in their pricing, minimum storage durations, and availability.

Use Case

Mount Cloud Storage buckets as file systems on Windows, Linux, macOS systems, so applications can upload and download objects using standard file systems. Cloud storage mounted file systems can make use of storage lifecycle.

Existing Solution

Cloud Storage FUSE is an open source FUSE adapter that allows you to mount Cloud Storage buckets as file systems on Linux or macOS systems, so applications can upload and download objects using standard file system semantics without requiring you to rewrite I/O code. Cloud Storage FUSE is ideal for use cases where Cloud Storage has the right performance and scalability characteristics for an application that requires file system semantics.

Cloud Storage FUSE can be run anywhere with connectivity to Cloud Storage, including Compute Engine VMs or on-premises systems. Cloud Storage FUSE is supported in Linux kernel versions 3.10 and newer.

Limitation with existing solution

Cloud storage FUSE supports Linux or macOs systems however it does not support Windows systems

Solution for Windows systems

Cloud solution will use a combination of linux compute instance and Gcsfuse (similar to AWS storage gateway). The files will be directly stored in Cloud storage and the linux machine will act only as a gateway to mount the bucket to Windows VM. The Lifecycle policy can be set on the bucket directly.

Solution Architecture

Solution Steps

1. Create a Cloud storage bucket : Create buckets | Cloud Storage.

On Linux VM:

2. Create a Linux VM — ( CentOS 7 and n1-standard-8 size ) in a shared VPC subnet and associate a service account with permission to access Cloud storage( roles/storage.admin ). To create shared VPC subnet please follow the link here Provision Shared VPC | Google Cloud. This will be the SAMBA-Server where we will mount the cloud storage bucket created in step 1 using gcsfuse and share it across the network using smb.

3. SSH to the Linux Machine ( Connect to Linux VMs | Compute Engine Documentation | Google Cloud ), switch to root using below command:

sudo su -
export BUCKETNAME=Name of cloud Storage bucket

4. Run the below shell script on the VM

#!/bin/bash
set -e
yum update -y
sudo yum install samba samba-client -y
sudo systemctl start nmb.service
sudo systemctl start nmb.service
sudo systemctl enable smb.service
sudo systemctl enable nmb.service
sudo mkdir -p /samba
sudo groupadd sambashare
sudo chgrp sambashare /samba
sudo useradd -M -d /samba/smbuser -s /usr/sbin/nologin -G sambashare smbuser
sudo mkdir /samba/smbuser
sudo chown smbuser:sambashare /samba/smbuser
sudo chmod 2770 /samba/smbuser
pass=admin123
echo -e "$pass\n$pass" | smbpasswd -a -s smbuser
sudo smbpasswd -e smbuser


sudo tee /etc/yum.repos.d/gcsfuse.repo > /dev/null <<EOF
[gcsfuse]
name=gcsfuse (packages.cloud.google.com)
baseurl=https://packages.cloud.google.com/yum/repos/gcsfuse-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=0
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
      https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
sudo yum install gcsfuse -y
mkdir /gcsmount
gcsfuse -o rw -o allow_other --file-mode 0775 --dir-mode 0775 --uid=99 --gid=99 --implicit-dirs $BUCKETNAME /gcsmount
setenforce 0


mv /etc/samba/smb.conf /etc/samba/smb.conf_orignal
sudo tee /etc/samba/smb.conf <<EOF
[global]
   workgroup = WORKGROUP
   security = user
[gcs]
   comment = Anonymous File Server Share
   path = /gcsmount
   browsable =yes
   writable = yes
   guest ok = yes
   read only = no
   force user = nobody
EOF
service smb restart
service nmb restart

On Windows VM:

Open Run give “\\VM_IP_ADDRESS\gcs’” and mount the drive as Network Drive, using username smbuser and password admin123.

Firewall:

1. Open port 445 and 139 on Windows and Linux server to allow smb.

Advantages

1. No need to use any 3rd party service/tool for e.g. rclone, nasuni etc
2. Simple VM configuration to support both windows and non-windows systems
3. Gcsfuse is open source so no licensing required
4. Low Cost solution
5. This solution can be used with a mix of windows and linux clients.

Disadvantages

Since there is a hop (a VM); an additional latency can be observed while transferring files from Cloud Storage to bucket.

Authors:

1. Kartik Saxena @kartiksaxena_28621
2. Vinay Damle @damlevinay