Let’s Azure : Creating and Managing Append Blob — with Azure Storage REST API
Azure Blob Storage is one of the cost effective and easy to use service when it comes to storing and sharing unstructured data in large quantity. Its fortification with azure security and access control, support for streaming data, automated replication and high availability makes it an obvious choice for use cases that involves video streaming, distributed data access, data transfer involving enterprise system integration for big data and analytics, system logs to name a few.
Azure Blob Storage actually supports three different types of blobs — Block Blob, Page Blob and Append Blob — each optimized for a primary purpose. And Append Blob is specifically optimized for operations where we need to keep adding data to a blob in chunks without modifying the already existing content. Every newly added data appends to the end of the blob, operation being referred as appending.
Here is a typical use case for Append Blob
In this article we will detail out how to create and manage append blob through Azure REST API. For this exercise we have used Postman as the HTTP client. If you are going to use append blob in your application, we will recommend to first test with Postman or SoapUI before getting into the coding.
As we move forward with steps for creating and manage append blob, you notice that, unlike other blob types, append blob needs to be created as empty and then contents to be appended into the blob.
STEP 1 — Setup Azure Storage
- Login to your azure subscription and select a resource group for storage. Alternately create a new resource group.
- Create a new storage account with type Blob Storage
3. Click on Review+Create and then Click on Create
This creates the Blob Storage. The next step is to create a Container in the storage and setup Security Access Keys
4. Click on the storage account — myappendblobstore. This will open the storage configuration screen.
5. Click on Containers in the left menu
6. Add new Container with a chosen name and encryption scope. We have created a new container myappendblobcontainer
7. Next Click on Access Keys in left menu and note down the Key1. This iwll be needed for building authorization signature later.
8. If you want to provide a temporary access, use the Shared Access Signature (SAS) option and generate a new signature and note down the three strings. SAS can be used instead of Access Key based signature for a timebound access.
STEP 2 — Create New Append Blob
As mentioned earlier, an append blob needs to be created as an empty blob and contents can be added block by block. It involves two operations — Create Blob and Append Block. The Azure Storage REST API supports both the operations.
Prepare the PUT request for creating new append blob.
URL : The syntax of http URL is
<storageaccountname>.blob.core.windows.net/<container>/<blobname>
The <blobname> shouldn’t be an exiting blob. Example — https://myappendblobstore.blob.core.windows.net/myappendblobcontaiter/appendblob01
HEADER: The header elements that needs to taken care most are x-ms-date and Authorization. The Authorization key needs to built using the prescribed approach. x-ms-date value in header and x-ms-date value used in generating Authorization must be same. Also needs to ensure the date format which is YYYY–MM–DDThh:mm:ssZ (example-2020–12–11T05:44:35Z). The x-ms-date should also be recent, as azure doesn't allow old timed request.
Content-Length should be 0 (zero) as the body of request is empty. Value of x-ms-blob-type should be AppendBlob. The request body should be none or empty.
Generating Authorization String: Directly using the Access Key fromAzure Portal as the Authorization will result in 403 error. The format of Authorization is
Authorization=”[SharedKey|SharedKeyLite] [AccountName]:[Signature]”
Signature string here is to be generated programmatically using the algorithm as suggested by Azure. The input elements to the algorithm are Access Key, Storage Account Name, x-ms-date. The output is HMAC-SHA256 encoded Signature string.
Read this article about creating the signature, specifically for storage: https://docs.microsoft.com/en-us/azure/storage/common/storage-rest-api-auth#creating-the-authorization-header
Authorizing with SAS: Alternative to the Key based authorization method, we can use the SAS based authorization as well. The SAS based token, however, is time bound and can used to limited time access only. The SA based request would look like below
RESPONSE: A successful request will return a 201 response, which indicates that the new append blob is created successfully.
VERIFY: Once we get a successful response, we can verify the newly created blob in Azure Portal. Notice that the new append blob will be of zero byte size as it is empty at creation.
STEP 3 — Append Contents to Blob
To add a new block of data to the end of an existing append blob, we need to execute Append Block operation on the blob. The Azure Storage REST API supports the operation.
REQUEST: Following is the syntax of the PUT request URL for append operation. Notice the additional http parameter comp. The blob name should be an exiting blob.
https://myappendblobstore.blob.core.windows.net/myappendblobcontaiter/appendblob01?comp=appendblock
The complete Append BlockAPI request looks as in screenshots below. We have captured Params, Headers and Body of the request to provide a complete view for everyone who is trying this first time.
RESPONSE A successful request will return a 201 response, which indicates that the new bloc of data appended to the end of blob successfully.
VERIFY Now that we have a successful response, lets verify the blob in Azure Portal. Notice the size of our blob (which was zero) it has increased which indicates the new data has been appended.
LETS DO AGAIN Lets execute the Append Block operation again to append another block.
Verify the blob in Azure Portal and notice the size of our blob has further increased which indicates another data block has been appended.
Please refer to https://docs.microsoft.com/en-us/rest/api/storageservices for more details on the API and operations.
Isn’t it interesting. If you like this and wanna read more on Azure, follow Let’s Azure and also click here to Follow this author.
If this story is helpful for you forward to your friends and if you have suggestions, do let us know your thoughts in comments.
Happy Azuring and Happy Coding !!!!
