Parallelizing 4EVERLAND S3 Workloads with s5cmd

S5cmd is a command-line utility that provides a faster and more efficient way to interact with Amazon S3-compatible object storage services. S5cmd is designed to be parallelized, which means it can perform multiple operations simultaneously, making it ideal for processing large amounts of data.

Installation

Install s5cmd on Mac OS X:

$ brew tap peakgames/s5cmd https://github.com/peakgames/s5cmd
$ brew install s5cmd

The tool is written in Go, other platforms can compile and install it using:

$ go get -u github.com/peakgames/s5cmd

Set up credentials just as you would for the awscli tool: Use the ~/.aws/credentials file or environment variables, or a combination of both. (If you’re running on EC2, roles are also supported.)

After the installation, you will need to configure s5cmd to work with your 4EVERLAND S3 account. To do this, you will need to provide your 4EVERLAND S3 access key and secret key.

Find your Access Keys here:

You can do this by setting the following environmental variables:

$ vi ~/.aws/credentials

[default]
AWS_ACCESS_KEY_ID=<your_4EVERLAND_s3_access_key>
AWS_SECRET_ACCESS_KEY=<your_4EVERLAND_s3_secret_key>

Usage

The s5cmd utility follows the “command [command options] argument1 [argument2]” format. In addition, it also provides options that affect all commands run. To view a list of these options, you can use the following command:

$ s5cmd -help

To view a list of available commands, simply run s5cmd without any arguments:

$ s5cmd

This will display a list of available commands that you can use with s5cmd, including put, get, cp, sync, rm, and many others.

Note: the 4EVERLAND — endpoint-url MUST be put before the command in order for the command to run.

List your 4EVERLAND buckets out:

s5cmd --endpoint-url=https://endpoint.4everland.co ls

Use s5cmd to interact with your 4EVERLAND Buckets:

You can use s5cmd to perform a wide range of operations on your 4EVERLAND S3 buckets, including uploading and downloading files, copying files between buckets, and deleting files.

To parallelize your operations, you can use the -m flag followed by the number of threads you want to use. For example, to upload a file to your 4EVERLAND S3 bucket using 4 threads, you can use the following command:

$ s5cmd put -m 4 /path/to/local/file s3://your-bucket-name/

This will upload the file to your bucket using 4 parallel threads, making the operation faster and more efficient.

For more information about s5cmd, please visit github.com/peakgames/s5cmd.

About 4EVERLAND

4EVERLAND is a Web3 infrastructure powered by blockchain technology that provides developers with convenient, efficient, and low-cost storage, network, and computing capabilities. It is committed to helping developers smoothly transition from Web2.0 to Web3.0 and building a Web3.0 cloud computing platform friendly to Web2.0.

Website | Twitter | Telegram | Discord | Reddit | Medium | Email