Alibaba Cloud — Object Storage Service

In this article, I am going to write about a short introduction to Object Storage Service of Alibaba Cloud and why and how we use it in our projects as the Trendyol Seller Core Team. If you are already familiar with Object Storage Service you can skip the introduction part.

Introduction

As the name suggests, Object Storage Service, in shortly OSS, is the storage service that store any amount of data in the cloud and enables you to manage them. Objects mean any type of file. As users, we want some standard features from these services surely such as highly secure, available, reliable, durable cost-effective, etc. Alibaba Cloud provides all of them and their motto for OSS is “Using OSS, you can store and retrieve any type of data at any time, from anywhere on the web.”

I want to mention some important keywords to understand the OSS concept especially for those new to the subject.

Bucket: Buckets are can be thought of as containers where objects are stored in. Before the store file, you should create a bucket firstly. How many and how to create buckets depends on how you categorize your files and how to use them on OSS. You can configure these buckets separately and manage them according to your requirements. For example, you can create two different buckets that the first one for image files and the second one for video files and run them on image processing functions separately.

Object: As I mentioned, objects known as files. Object consists of three items: data, metadata, and key. The Key is the unique name of the file in the bucket.

Region: Alibaba set up its data centers on various locations around the world. Regions are the physical locations of these data centers. You have a chance to select region for your buckets.

Endpoint: Endpoints are domain names of RESTful APIs that provide access to OSS. Each region has its own endpoints.

AccessKey: In shortly AK, provides identity verification and consists of two items: AccessKeyId to identify user and AccessKeySecret to encrypt and verify the signature. OSS verifies a user by using these two items by symmetric encryption method. AccessKey must be kept confidential.

Presigned Url: This term is used usually in storage services and it is the most used feature in our projects. Thanks to presigned url, we are able to give temporarily read or write the right to our specific object for others on condition that provides the policies defined by us.

If you decide to use OSS, you should know three storage class means because pricing depends on these three storage scenarios:

Standard: This supports frequent data access and provides highly available and performance services. If you will read/write data on OSS frequently, standard storage is proper for you.

Infrequent Access: In shortly IA, it is proper for less frequent access and long-lived data. Generally once or twice per month.

Archive: It is suitable for long-term storage, thus the cheapest storage class.

After you apply for OSS and get an account, you can manage your files with a web-based OSS console which is useful for beginners. For instance, you can upload your files by drag-drop or change your configurations by clicking a couple of buttons. On the other hand, you can manage your files with various SDKs (containing various languages such as Java, Python, Go, Node.js, and more), RESTful APIs of OSS, or Aliyun CLI.

Why we use OSS in our projects?

The Process without Object Storage Service

Before the OSS, we manage files completely by our own APIs, and files are stored in our own storage. Files are transferring by FTP or SFTP via our own network. When the client wants to download files, firstly send a request to its backend application and the application gets the file from the storage by FTP as a binary file, then delivers it to the client. If your number of users is huge, this method makes you deal with performance issues. Binary files make your network and applications slower. Although dealing with files is not the main responsibility of your application generally, files cause performance problems and prevent applications to do their main tasks. In summary, manage all of these processes is difficult and costly.

The process with Object Storage Service

With OSS, our application firstly communicates with the RESTful API of OSS after the client sends a request to our API. In other words, our API handshake with OSS API in order to manage files. For example, when the client sends a request for a downloading file, our application sends a request contains a policy to OSS API. OSS API understands which user thanks to accessKey and sends a requested response. This response is presigned url for the client. Our application sends this presigned url as a response to the client. The client has the right to download the file by this presigned url right now and it can get the file directly from OSS. In this way, our application does not deal with the binary file. It just provides communication with client and OSS. Both our application and network are not tired of binary files.

Presigned url is just one of the benefits of OSS. Absolutely, there are more such as defining triggers and function computes in order to apply Serverless architecture.

Java OSS Client Example

In this article, I am going to write about how we implement in Java four main OSS processes that used mostly in our projects. These are:

1. Put Object
2. Get Object
3. Get Presigned Url For Downloading Object
4. Get Presigned Url For Uploading Object

Before creating a project, let's create a bucket. You can easily create a bucket and do some configurations by using OSS Web console. However, I am going to use Aliyun CLI to do these operations. As described on the given link, you can install and configure your CLI. I did it as given below:

$ brew install aliyun-cli
$ aliyun configure
Configuring profile 'default' ...
Aliyun Access Key ID [None]: <Your AccessKey ID>
Aliyun Access Key Secret [None]: <Your AccessKey Secret>
Default Region Id [None]: eu-central-1
Default output format [json]: json
Default Languate [zh]: en

As you see, I configured region as Europe Central, language as English. I have a programmatic user right which has read and write right by programmatically. Then I create a bucket:

$ aliyun oss mb oss://<your-bucket-name>

Then, I create a Spring Boot project and add a dependency in below to pom.xml file to use Java OSS Client.

<dependency>
    <groupId>com.aliyun.oss</groupId>
    <artifactId>aliyun-sdk-oss</artifactId>
    <version>3.6.0</version>
</dependency>

I will create OSS Bean configured according to my accessKey, secretKey, and OSS region endpoint. In addition, I need a bucket name. It is proper to put these four field application.yml file.

I create a configuration properties file to obtain these fields and use it when it will be needed. Now, I can create OSS Bean:

After that, you can benefit from all methods of OSS Bean. I create OssService and I put my 4 main OSS process on it. If you want you can add an interface for these five methods.

For example, when you get a binary file and you want to put it in OSS. You should use the putObject method as given below:

ObjectKey is unique for each object. It is a path from the root directory to the file name in the bucket. So be careful, when giving objectKey as a parameter of these methods. Finally, It is enough to give the input stream of the binary file to method as the second parameter. OssClient throws its own exception, you can handle your exceptions as you wish.

Similarly, getObject method is given below:

It is so easy as you see. It returns the object of OssObject class which contains metadata of object and input stream of the binary file. You can use this class as you wish where you call this method.

Let's see the how we get the presigned url for downloading object:

Actually, there are few methods to get presigned url. In this example, we create a request by using the bucket name and expire date parameters. ExpireDate specifies the validity period of presigned url. After this time is up, presigned url would not be valid. Then, we use generetePresignedUrl method of OssClient. This method returns directly to presigned url. DownloadPresignedUrl is our custom class which has only presignedUrl field.

Before look at how getting UploadPresignedUrl, it is good to know that we need six fields which are used in post request to upload object. After you share these fields with front-end, front-end will upload objects by sending a post request to OSS endpoint, and the backend does not care about the binary file as I mention before. I create a class for front-end response called UploadPresignedUrl whose fields are objectKey, accessKeyId, expire, host, policy, and signature.

You already know objectKey, accessKey, and expireDate fields. Let's see how we create other fields.

The host is endpoint sent post request to upload an object, combine of bucket name and OSS endpoint. It can be created as below:

PostPolicy indicates some conditions for validation. For example, I want an exact match of objectKey for valid presigned url:

As you see, I use expireDate here too as a parameter of generatePostPolicy method of OssClient. Policy string in JSON format in here but it must be sent as encoded base64:

And then, we calculate the signature based on the policy by using calculatePostSignature method of OssClient.

After we send these six fields as a response to the front end, it sends a post request to OSS with a binary file to upload. An example request is given below:

Your request must be POST, fields must be given as form data and in the exact order given as above. Otherwise, OSS returns a bad request despite all correct data.

Another critical point is that you will get CORS error from OSS unless you do the required configurations on your bucket. You can do this configuration by using OSS Web Console following Bucket -> Basic Settings -> Cross-Origin Resource Sharing (CORS). Another way is using Aliyun CLI:

aliyun oss cors — method put oss://<your-bucket-name> <cors-config-xml-file-path>

Example OSS CORS configuration file is given below:

You can allow the methods, origins, and headers as you wish.

I share some of the most used scripts of Aliyun CLI.

Conclusion

In this article, we introduced Object Storage Service of Alibaba Cloud, learned benefits of it, and share how we use it in our projects.

If you want to contribute these works, develop highly scalable applications join to Trendyol family.