File transporting service in Go — part 2
In the previous part we made the first iteration of a file transport service called Shuttle. While it was simple and usable for a first iteration, it depended too heavily on the underlying system. It was impossible to support file transfers where closing the file did not mean the transfer was finished. It also made open-sourcing and distributing the tool unnecessarily difficult since we would have to separate out all the interdependent Ansible roles for putting up an SFTP / FTP server and configuring them for Shuttle.
In this part we’re implementing the SFTP and FTP servers as part of Shuttle using github.com/pkg/sftp and github.com/fclairamb/ftpserver, albeit modified to suite our purpose.
Design changes
The architecture of having a Queue of Shuttles, a Route for each user / endpoint combination, which is given to a Shuttle, and workers that handle sending those shuttles to their destined HTTP endpoint works well here and does not need to be modified. However other parts are going through a major overhaul.
We no longer want to have any dependencies to the underlying system other than having the folders placed for us. That’s why configuration is now stored in a separate JSON file instead of .endpoint files since we no longer want to use Unix users for authentication and thus have to specify credentials somewhere else.
Recall that our folder structure in the last part was dictated by how the chroot implementation of the OpenSSH SFTP server requires that the chrooted user has no write access to the root folder of the chroot. Since we are implementing SFTP into the application we will no longer be constrained by the restrictions, and we can simplify the structure to look as below.
/chroot
└── [user]
├── [file 1]
└── [file 2]Introducing Services
The new architecture main point revolves around generic Services. Service is an interface that dictates a handful of public methods that each service must have. Beyond that, each service has freedom in implementing the details.
The main routine simply starts off the services and has a goroutine for listening on each service’s WriteNotification channel, transforming each write notification into a shuttle that gets pushed onto the queue.
The Reload and Stop methods are used for causing minimal disruption to the service as a whole.
The Reload method causes the service to take into account the new routes with as little disruption as possible. The SFTP service for example only considers routes during login to validate credentials, so it can simply set it’s internal copy of routes to the new routes and carry on without any disruption.
The Stop method should shut down the service gracefully, i.e. wait for transfers to complete before closing sockets, cleaning up and returning.
The application listens for SIGINT, SIGTERM and SIGHUP using signal.Notify. Whenever a SIGINT or SIGTERM is caught, the Stop method is called for all services and any shuttles that are enroute to their endpoints are waited on, bringing the application to a graceful stop.
Whenever a SIGHUP is caught, the configuration is re-read and the Reload method is called for all services, reloading the configuration without disruption to connected clients or transfers.
Let’s take a look at how we implemented the services.
SFTP
Go has an SSH implementation in the subrepositories, that is, it’s made by the Go team but not shipped as part of the standard library. This makes implementing SFTP in Go much, much easier. However we don’t need to do even that by ourselves. There is only one major package for Go that implements SFTP, github.com/pkg/sftp, but it was created by Go developer Dave Cheney, and every package under github.com/pkg is of very high quality. Off to a great start!
The package had to be slightly modified to suit our purpose:
- Added support for chrooting users
- Added support for gracefully stopping the SFTP server, as in waiting for all open transfers to be completed
- Whenever a file transfer is complete a notification of the file write is posted on a channel. This allows us to simply convert these notifications to WriteNotifications and push them along
After the modifications, writing the service implementation was quite simple and the entire implementation file clocks in at ~250 lines.
FTP
The FTP service was much hairier. There are a number of FTP server packages, most of which are experimental, abandoned or incomplete. The closest package to being workable was github.com/fclairamb/ftpserver.
The package works by giving it a “driver” that contains all the necessary methods for operation, such as changing directory and opening files. This makes it really easy for us to implement chrooting. However as said, the package isn’t perfect. We had to make the following modifications:
- Ripped out the log15 library it was using for logging in favour of a system where you pass it an io.Writer to write plaintext logs into. The io.Writer can then be io.Discard if needed
- Fixed a critical bug where authentication was not actually required as you could skip sending the USER and PASS commands entirely
- Added a method for fetching the current authenticated user
- Added support for gracefully stopping the server
- Added a new method called NotifyWrite to the driver that is called whenever a transfer finishes
As you can see it required quite a few modifications. After the modifications we had to implement the driver which was luckily trivial due to the good example driver that the package has. The entire service implementation file is roughly the same length as the SFTP service implementation.
Local
Since we implemented an inotify-based solution in the last part, there’s no point in throwing it away. Let’s implement it as a service in case we run into more esoteric protocols without implementations in Go!
The service implementation is close to a copy paste from the code we had in the last part. While the SFTP and FTP services can both use the same directory, we cannot have the local service paired up with any other service, the local service must be a special case. This is because other services do not know of any file transfers going on by services other than themselves. However the local service just monitors for IN_CLOSE_WRITE events, which are triggered regardless of what service or method the file was transferred over.
You probably noticed the local attribute in the configuration file. That specifies whether the route should be used by the local service, or all other services. This means you can have a user that can upload files over SFTP and FTP, or a user that cannot connect to the SFTP or FTP services but whose directory is monitored for created files.
This service implementation was the simplest of them all, not only because we had the code already, but because it clocks in at half the size of the other implementations and required no modifications to the fsnotify package.
Conclusion
Shuttle is now self-contained and much easier to extend if needed. We’re currently working towards open-sourcing it and will announce the release in the coming weeks.
We are hiring!
Do you want to build exciting and interesting apps with modern web technologies? Check out our open positions and join us at Taito United!
Can’t find positions directly suitable for you? Feeling adventurous? Send your resume and an open application to jobs@taitounited.fi — we are more than happy to hear you out and see what you got cooking 🍳
