SFTPGo access to GCS via SFTP
SFTPGo is an SFTP server that is distributed as an open source project that has the built in ability to access Google Cloud Storage (GCS). We can use SFTPGo as a mechanism for providing access to GCS through SFTP. This article takes us through a recipe to get it working and adds some additional considerations on use.
From an overall architecture perspective, SFTPGo can be installed on a Google Cloud Compute Engine. Once installed, it runs and maintains its own stateful local configuration. A browser/web based administrative console is then available for further detailed configuration. We would then create user identities that are known to SFTPGo. Each of these user identities has an associated name and password. When the user is defined, we also name a GCP bucket that will be used as the storage that the user sees through an SFTP client. In order to access the bucket, we would also supply some Service Account credentials that SFTPGo will use to interact with the bucket on behalf of the user. This configuration is incredibly flexible as it permits per user bucket access and Service Account configuration meaning that different users can have different access to different buckets. Once configured, users can then attach SFTP clients (or optionally a web interface) to browse, put and get files stored on Cloud Storage.
In this story, we will start with the idea that we will create an SFTP server hosted on a Compute Engine with a public IP address. We will then allow clients to access this SFTP server with the view that files they upload and download will be mapped to GCS buckets.
We’ll start with a newly created Compute Engine running Debian. Once started, I logged into it using SSH.
We next followed the steps to install it using apt. These are documented here.
curl -sS https://ftp.osuosl.org/pub/sftpgo/apt/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/sftpgo-archive-keyring.gpg
CODENAME=`lsb_release -c -s`
echo "deb [signed-by=/usr/share/keyrings/sftpgo-archive-keyring.gpg] https://ftp.osuosl.org/pub/sftpgo/apt ${CODENAME} main" | sudo tee /etc/apt/sources.list.d/sftpgo.list
sudo apt update
sudo apt install sftpgoOnce the installation is complete, sftpgo will be up and running.
Next, create a firewall rule that allows incoming traffic into your VPC on ports 8080 and 2022.
Open a browser to http://[COMPUTE_ENGINE_IP]:8080
You will see a panel prompting for the SFTPGo admin userid that you wish to create. Choose something secure for your password.
You will now be shown the SFTPGo admin screen.
Now it is time to create users. These will be the users who are authorized to login to the SFTP server. However, before we create a user, we need to prepare for GCS access. The default function of SFTPGo is to be an SFTP server using the local file system as a datastore. For our story, we want to use GCS as that store. To interact with GCS, we need to provide Google Cloud authentication proof that we are a known identity. To achieve this, we create a Service Account and also retrieve the JSON credentials for that account.
Click DONE as we don’t need to set up any other attributes.
Next, we want to create a JSON key
This will be downloaded to your computer. I suggest renaming it to match the name of the service account for which this is the key.
With the Service Account created, we need to give permissions to it to be able to access the bucket. At a minimum, Storage Object Viewer role will allow the account to get files, Storage Object Creator will allow the user to create files while Storage Object Admin will additionally allow everything including the deletion of existing files.
What remains is to define the users that are permitted to access the SFTP server. When an SFTP client attempts to connect to an SFTP server it must authenticate. It can use either a userid/password pair or provide SSH credentials. We can define the users that SFTPGo will recognize through the admin web page.
We start by clicking Users in the selection panel. From there, the “View and manager users” screen is shown. If we then click “+” we get to define our user:
There is quite a bit going on in this panel so let’s break it down. First, we provide a Username and Password. This combination will allow the SFTP client (or web client) to access SFTPGo. There is no relationship between these usernames and passwords and other identity sources such as Google Accounts. Only SFTPGo uses these identities.
The next important item is the Filesystem entry. When the user connects to SFTPGo, it is this definition that defines what files system is presented to the user. The default is “local” meaning what is shown is the local Linux file system of the Compute Engine. In our story, we change this to Google Cloud Storage because we want to access GCS buckets. When we select the Storage = Google Cloud Storage entry, other properties change to reflect Google Cloud specific entries. There are two further settings we must supply. The first is the name of the GCS bucket that we are working against. The second is the GCP Service Account credentials JSON file that we exported earlier. SFTPGo uploads and stores this credentials file.
And that’s it. Now when we connect an SFTP client we will be prompted for a userid and password pair and we can enter the information we created. We will then be able to execute SFTP commands such as “ls” to list files, “put” to put files and “get” to get files. The access to GCS will be performed using the service account referenced by the credentials file. GCS will also only allow actions against it permitted by the IAM roles associated with the service account. For example, if the account only has Storage Object Viewer, then it can’t create new files while if it only has Storage Object Creator it can’t retrieve existing files.
When we create SFTPGo users, we may not want to continually specify GCP credentials and bucket details (but we can). Instead, we can define our file system details in the specification of an SFTPGo group and associate the users we create with those groups. In the following, we show the definition of a new group we have called “read-group”
Now when we create a user we can reference this group and forego specific Filesystem definitions on the user page. The group’s file system definition has precedence over the user specific file system definition.
Not only can we use SFTP clients to connect to SFTPGo but we also have the option to use its built in Web UI that is client specific. If we visit the Compute Engine at port 8080 with a browser, we will be prompted to login as an SFTPGo user. From there we are presented with:
Take a few moments to look at this screen. Its core is a file system view showing files, their size, and when they were last modified. These files are the GCS files we would see from an SFTP client. From here we can perform some important actions including downloading files, uploading files, deleting files and more. A feature I want to point out is the edit capability. If we click the edit icon associated with a file, we immediately get to see the content of that file in a browser based editor. Not only can we see the content but we can also change the content and have it written back as a replacement for the original file. Effectively, we have a GCS file editor!
There is much more that can be said about SFTPGo and I direct you to the GitHub project home page for more details. Since this is an open source project, you can download and use it without charge. The flip side is that there is no formal support. If you find a problem, you can raise an issue but there are no SLAs for resolution. Since the source code is also present, there is the ability for you to fix it yourself but, as we know, that is easier said than done.
Beyond SFTPGo, there are a variety of other SFTP solutions and file browsers for GCS.
Video
A short video illustrating many of the concepts of this articles is available here:
Debugging
In the event you need to perform some debugging, SFTPGo writes SFTPGo log records to the local /var/log/syslog file.
