SFTP to Object Storage cloning with Oracle APEX

Content management on a new level

• Synchronize FTP server files with Object Storage
• Upload, Download and Manage files in APEX
• Expand — Add document understanding and other AI services for more advanced scenarios!

Local Environment Setup

Any environment combination will work as long as the machine is able to access internet.

Let’s start by creating a new user that will be specifically used for the FTP Share:

sudo adduser rpi_ftp_user -- Create the user rpi_ftp_user
su rpi_ftp_user -- Change to rpi_ftp_user

Then create a new folder where the files will be stored:

mkdir share

If we now inspect our FTP Share, we will see the following structure:

We now have a share that can be used to store files locally, our target is to replicate those files to OCI, in this particular case we will be moving them to Object Storage using Rclone.

Object Storage Setup

Before we can push our files to Object Storage you will need a bucket, here’s how you can create one.

Take note of your bucket and object storage namespace as those will be needed in the next step when configuring Rclone.

Configuring Rclone

The configuration for Rclone is quite straightforward as it takes you through a wizard and allows you to add all kinds of different data sources, let’s take a closer look at those steps!

You will need the following parameters:

#Bucket Name
#Object Storage Namespace 
#Compartment OCID - Where your Object Storage is located
#OCI_Region - Region of your tenancy 
#OCI_Endpoint - To access your object storage (Example: https://<your_namespace>.compat.objectstorage.<your_region_identifier>.oraclecloud.com)
#OCI_Configuration_File - Obtained when creating an OCI User API Key (Docs: https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm) 

To create a new remote (Data Source) within Rclone execute the following:

rclone config
n #for new entry
my_remote #name of your new remote i.e OSS
37 #For OCI Object Storage - verify against the list
2 #For OCI User/API Key Authentication

#Now enter the Object Storage Details as prompted
Namespace
Compartment_OCID
Region
Endpoint
OCI_Configuration_File

N #For no advanced config

#Verify if all the details are correct and confirm with 
Y 

If Rclone is configured correctly, the command should bring back a list of available remote environments:

rclone listremotes

Let’s look at the available directories(buckets):

rclone lsd oci_oss:

Great success, we can see the Object Storage Buckets available!

Next steps…

Knowing the buckets we can list the files inside them, the command structure is handy as well with:

### ls for list 
### d/f for directory/file
### resulting in lsd or lsf 

rclone lsf oci_oss:FTP_Share

And finally test replication from Local Storage to OCI Object Storage:

rclone -vvv --progress --stats-one-line --max-stats-groups 10 --log-format date,time,UTC,longfile --fast-list --buffer-size 256Mi --oos-no-check-bucket --oos-upload-cutoff 10Mi --multi-thread-cutoff 16Mi --multi-thread-streams 3000 --transfers 3000 --checkers 64 --retries 2 --oos-chunk-size 10Mi --oos-upload-concurrency 10000 --oos-attempt-resume-upload --oos-leave-parts-on-error sync ./artifacts remote:bucket -v

Our files are successfully backed up, lets check if something changed:

rclone sync --interactive ./ oci_oss:FTP_Share

All objects are currently synchronized with the remote Object Storage bucket…

• There is also a ton of options(flags) you can use — click
• In addition Object Storage can also be locally mounted with Rclone

Continuous Sync

Time to set up a cron job that will ensure our files are regularly synchronized (every 5 minutes in this case):

/5**** /$HOME/rpi_ftp_user/scripts/oss_backup.sh

APEXing — CMS

I’ve quickly put together a screen where I can select my buckets and preview, download and delete my files, it’s not pretty but it does its job!

Since there’s a bunch of articles out there describing how you can manage Object Storage files from APEX, I won’t be going through those steps however you can refer to the following:

• Uploading files to Object Storage from APEX
• File Management in Oracle APEX OCI Object Storage
• … and then add extra flair with AI services and other integrations!

Extras

What if we want to replicate into File Storage or look at some alternatives, perhaps using a Site-2-Site VPN or FastConnect to avoid public internet traffic?

Storage Gateway

• Documentation
• Download

Others (fpsync, parcp etc.)

• Documentation

Note: It is now possible to attach a Network File System to an Autonomous Shared instance, read more here.

Conclusion

We now understand how quickly we can synchronize different environments and feed information to our Database/APEX instance, additionally with the native ability to directly load various data( CSV, JSON, XLSX, TXT etc.) from Object Storage into Autonomous Database most of this process can be automated and effortless.

For more information you can find me on LinkedIn, and as always don’t forget to visit the Oracle APEX page for all your latest news.