Mounting your Object Storage Bucket as a File System on your ECS Instance
In this article , i’ll be dealing with Flexible Engine Cloud Provider in order to mount my Object Storage Bucket as a File System on my ECS Instance.
Establishing communication between our ECS instances and the Object Storage Bucket is crucial for facilitating different daily exchanges such as moving our local files and directories directly to the mounted Bucket without having to use the Cloud Provider’s Object Storage Console.
This process can be done in two main ways :
- Writing a Python script using Flexible Engine SDKs.
- Using the s3fs-fuse project
In this tutorial, i’ll focus on the s3fs-fuse project as it is a lot more easier to access Object Storage Bucket content directly using a file system then through scripts. However, if you’re interested on how to deal with Flexible Engine Python SDK to access your Object Storage Bucket, you can checkup my next article.
In order to get started, all what you need is a Flexible Engine account (or any Cloud Provider account that provides an Object Storage API compatible with the s3fs-fuse project), a Linux compute instance and of course an SSH access.
You also need to connect to your Cloud Provider account and setup up an IAM policy for your Object Storage Bucket :
- Allow read and write on your bucket
- Define and get your credentials (access key and secret key)
For more information about how to perform these steps , check out the Flexible Engine Documentation or your Cloud Provider Documentation.
Step 1: Install s3fs-fuse
If you are using a new Ubuntu instance, you first need to update your system.
Next, install the dependencies.
Then,clone the s3fs source code from git.
Now change to source code directory, and compile and install the code with the following commands:
Use below command to check where s3fs command is placed
Step 2: Configure Credentials
Create a new file
/etc/passwd-s3fs and paste your access key and secret key in the below format
Then, change the permission of the file
Step 3: Mount your S3 Bucket as a File System
Now create a directory and mount your Object Storage Bucket in it.
your_bucketname is the name of your Object Storage Bucket that you have created on your Cloud Account.
use_cacheto use a directory for its cache purpose.
allow_otherto allow other users to write to the mount-point.
uid is the uid the owner of the mountpoint(to get your id , enter
id on CLI)
mp_umask to remove other users permission.
multireq_maxis a parameter to send request to your Object Storage Bucket.
/mys3bucketis the mountpoint where the bucket will be mounted .
url for Flexible Engine can be replaced by your
Object Storage Endpointprovided in your Object Storage Bucket Basic Information on your Flexible Engine Account.
In order to mount your S3 Bucket automatically after every reboot you should define the script
/etc/mount.sh then use
crontab -e to automatically run
/etc/mount.sh script to remount your S3 after every reboot.
You can run
sudo systemctl reboot to reboot your ECS instance.
For debugging purposes , you can add
-o dbglevel=info -f -o curldbg in the s3fs mount command.
In order to check if your Object Storage Bucket have been mounted, run the following commands :
You’ll get the following result :
If it shows the mounted file system, you have successfully mounted your OBject Storage Bucket on your ECS Instance. You can also test it by creating a test file
In order to see the above change, Login to your Object Storage Bucket and verify if the test file is present or not. If it is present then Congrats ! your Bucket is successfully mounted to your ECS instance. From now, any files written to /mys3bucket will be replicated to your bucket.
- If you already have some data in your bucket which is not visible, then you have to set permissions in your ACL through the use of the your Cloud Provider management console for that bucket.
- If you get any s3fs error such as
transport end point is not connected, you have to unmount and remount the file-system. You can also do so through a custom script to detect and perform remount automatically.
- To unmount your mounted S3 Bucket use :
sudo umount -l /mys3bucket