Sia S3 Integration: S3FS

What is S3FS?

S3FS is a FUSE (Filesystem in Userspace) based solution that allows you to mount an Amazon S3 bucket as a local filesystem. It translates file operations to S3 API calls, letting you interact with S3 storage as though it were a local disk.

What is Sia?

Sia is a decentralized data storage network that allows users to rent affordable cloud storage. Data uploaded to Sia is encrypted and stored on dozens of hosts around the world. The data is encrypted and under the user’s control. The user must run the renterd software to maintain the data as hosts come and go. This removes the dependency on any one cloud vendor like Amazon or Google and allows the user to optimize the many vendors (hosts) over time based on qualities such as price, speed, and geographic location. This means that with some tuning, over time, a Sia user can provide an affordable storage solution that is fast enough to use for use cases like streaming movies.

Why combine the two?

When you interact with the mounted S3 bucket, S3FS handles the communication between your local filesystem and the remote S3 service. For instance, if you read a file using a standard Linux read system call, S3FS will translate that into an appropriate S3 API call to fetch the data. This enables seamless integration of S3 storage into environments that expect a traditional filesystem, making it easier to use tools or software that weren't initially designed to work with object storage.

⚠️ This guide is for MacOS. Additional Windows and Linux guides will soon be available via the Sia Documentation.

Step 1: setting up Renterd

Basic Setup

Set up renterd for your system. For this guide, I will be installing renterd on MacOS.

S3 Setup

To use Sia with S3FS, we will need to rely on its integration with S3. This API is not enabled by default, but configuring it is straightforward and can be done by editing the renterd.yml file you created during the renterd setup process. This step is the same for all systems

seed: your seed phrase goes here
http:
  password: your_password
autopilot:
  heartbeat: 5m
s3:
  enabled: true
  disableAuth: false
  address: "localhost:9985"
  keypairsV4:
    your_access_key: your_private_key

⚠️ your_access_key must be at least 16-characters long, and your_private_key must be at least 40-characters long.

Verify Your Setup

After setting up renterd and enabling S3, it’s a good idea to test it out quickly. This can be done using either the AWS CLI Client or Minio Client. If you use the Minio client, setting an alias is very useful. Let’s try and list the buckets to see if it works.

mc alias set s3 http://localhost:9985 your_access_key your_private_key &&\\
mc ls s3

ℹ️ It should return the default bucket created automatically when you started renterd for the first time.

Step 2: install S3FS

Install and configure s3fs using the Official Install Guide.

On MacOS 10.12 and newer, this can be done using Homebrew.

Once you have installed Homebrew, you can run the following commands in a terminal to install S3FS.

brew install --cask macfuse
brew install gromgit/fuse/s3fs-mac

Step 3: mount a directory

Authentication

S3FS supports the standard AWS credentials file stored in ${HOME}/.aws/credentials. There are alternative ways to create a custom password file, but we will use the AWS standard:

[default]
aws_access_key_id = your_access_key
aws_secret_access_key = your_private_key

Mountpoint

Create a directory on your local file system. For the purpose of this tutorial, it’s easiest if this directory is empty, but s3 has some custom options that allow using a non-empty directory as well.

mkdir ~/s3-mount &&\\
s3fs default ~/s3-mount -f -o url=http://localhost:9985/

⚠️ Use the following options for debugging:

• -o dbglevel=info
• -o curldbg

Step 4: Test it out

Copy a file into the directory we mounted on our bucket. If you have debug logging enabled, you should see how S3FS translates the system call in S3 API calls that hit the renterd S3 API. Visit the renterd UI and verify the file was uploaded fine.

Conclusion

Using Sia as a storage backend when employing S3FS presents a compelling alternative to traditional cloud storage solutions. Sia’s decentralized architecture offers enhanced security, redundancy, and potentially lower costs. By pairing it with S3FS, you could bring these advantages into environments that are already built around conventional filesystems. This hybrid approach allows organizations to leverage the benefits of decentralized storage without rewriting applications or retraining staff. It creates a bridge between the new wave of decentralized technology and the existing infrastructure, making it a strategically exciting option for those looking to optimize storage costs, security, and resilience.