Modern File Transfer Solution on Azure — SFTP

Are you thinking about a way to modernize your applications that make use of SFTP file transfers?
Classic SFTP integrations leverage on a server to which they send files and a client polls at regular intervals, waiting for a file to became available for processing. This approach introduces time lags between the file being produced and it being consumed.
What about enhancing this approach using event driven file transfers?

With SFTPGo, an Open Source software available on GitHub, you can modernize the way systems exchange data between each other and call target applications in near real time.

The aim of this article is to present an architectural overview for file transfer modernization using Kubernetes and queues.

The presented solution is designed to be Scalable, Highly Available and Secure.
This article focuses on an Azure deployment but the solution itself is cloud agnostic, meaning it can be deployed either on AWS, GCP, OpenShift or on-prem using Kubernetes with some tweaks.
The following components are used:

• Azure Kubernetes
• Azure Service Bus queues for massive processing
• Azure Firewall
• Azure AD Managed Identities
• Azure API Gateway

When the client sends a file, the socket traverse an Azure Firewall that inspects, at Level 4, the source IP and the packets. Then the packets go to the Load Balancer, which is deployed though Azure Kubernetes Service. The LB balances traffic among multiple SFTPGo pods.

Each pod inherits from the AKS ConfigMap and stores its settings to a MySQL PaaS database. Files are then saved to an Azure Blob Storage.

SFTPGo handles client authorization, which can be delegated to an internal SFTPGo database or to Azure AD with a custom authenticator plugin. The authentication between the components of the solution uses Azure Managed Identities to minimize the use of secrets.

Wherever an event occurs, such as a file being uploaded, SFTPGo pods send a message into a ServiceBus queue. A MessageRouter pod listens on that queue waiting for new messages to be routed, evaluated the body of the message, and routes it to the relevant target queues or applications using HTTP or AMQP or MQTT protocols.
In this way, target applications receive a near real time notification about the file being uploaded (or whatever event is configured) and can properly react.

References
SFTPGo on GitHub
MessageRouter on GitHub

Disclaimer: This article is published as an individual and is not related to my current job.