Efficiently Transfer Files from/to Azure Storage using AzCopy

AzCopy is a command-line utility that can be used to copy data from and into Azure Storage.

Pre-requisites

Download the executable file from Microsoft Reference Guide.

Decompress the zip/tar file, access the .exe file, and execute the azcopy command to ensure its working.

To make the azcopy command automatically recognized by command prompt (or terminal), Windows PowerShell, we must add the location of the AzCopy executable to the PATH variable under user/system environment variables.

Adding to the system variables would eliminate the need to change directories to the right location every time.

If there is/are any other existing path/s, add a ‘;’ to the end of the path and append the executable file location. In latest versions of Windows, choose edit for PATH variable, then select add and input the file location.

Working Steps

The two ways to perform the authorization are:

1. Authenticate with Azure Active Directory (AAD)

2. Append a Shared Access Signature (SAS) to each source or destination URL in your azcopy command.

Azure Active Directory Authentication:

By utilizing this option you can omit appending SAS token to source or destination URL of storage account in every single azcopy command.

To download the files from azure to local system verify that Storage Blob Data Reader has been assigned to your user identity, managed identity, or service principal.

To upload files verify that Storage Blob Data Contributor, Storage Blob Data Owner roles has been assigned to your Container (file system) or Storage account or Resource group or Subscription.

To assign the roles to the container follow the below guidelines:

Procedure to add the role assignment through Azure portal

Select an appropriate role using the search bar

Assign the role to appropriate user, group, or service principal

Azure role assignments may take up to five minutes to propagate.

Shared Access Signature:

You could append a SAS token to each source or destination URL in your AzCopy commands.

If you plan to copy blobs across storage accounts in AzCopy version 10, you’ll have to append a SAS token to each source URL. You can only omit the SAS token from the destination URL. More details can be found at copy blobs between storage accounts.

Generate the SAS token using the below process:

Generation of SAS and connection string

Copy the SAS token and replace <SAS-token> with it in the following queries.

AzCopy Commands

Authentication:

After verifying that your user identity has the necessary permissions, execute the following commands in the terminal.

> azcopy login

> azcopy login --tenant-id=<tenant-id>

You will get the tenant-id by navigating to Azure Active Directory → properties section, copy the Directory Id and replace <tenant-id> with it in the above command.

It prompts to open the device login page and then input the code displayed over the command line. Just follow the instructions for a successful login.

After signing in successfully, you may close the browser window and begin using additional azcopy commands.

These are the most practiced interactive options for logging into Azure Active Directory to access Azure Storage resources. To know more alternatives visit azcopy login documentation.

Copy Process:

azcopy cp/copy command is used to copy source data to a destination location. Here you may find the supported settings.

# General azcopy pattern
> azcopy copy [source] [destination] [flags]

To obtain the blob storage container path, proceed as follows:

Container path is available in the properties section

URL - Azure blob storage container path

Copy the URL and replace <blob-storage-container-path> with it in the subsequent commands.

Local-path:

This is the path of your file/directory/object on the local computer.

Upload Files:

The following commands are used to copy or upload data from local storage to Azure blob storage - Reference Guide.

> azcopy copy "<local-folder-path-or-file-path-with-extension>" "<blob-storage-container-path>/[path-to-blob-or-directory][SAS-token]" [--recursive=true] [--include-pattern <semicolon-separated-file-list-with-wildcard-characters>]

# example - without using SAS token to upload a file
> azcopy cp "C:\Users\file.txt" "https://<storage-account-name>.blob.core.windows.net/<container-name>" 

# example - without using SAS token to upload only text files in a directory to a conatiner
> azcopy cp "C:\Users\directory" "https://<storage-account-name>.blob.core.windows.net/<container-name>" --recursive=true --include-pattern ".txt"

# example - with SAS token to upload a single blob
> azcopy cp "C:\Users\file.txt" "https://<storage-account-name>.blob.core.windows.net/<container-name>/<SAS-token>" 

The — recursive=true option is used to upload an entire directory.

The — include-pattern option is used to include specified files based on a pattern. This only applies to filenames and not the path. Refer to this source for further information.

More options can found at azcopy copy documentation.

When the content of a file is updated locally and you want those changes to be reflected at a remote location, use the sync command.

> azcopy sync "<local-folder-path-or-file-path-with-extension>" "<blob-storage-container-path>/[path-to-blob-or-directory][SAS-token]" [--recursive=true] [--include-pattern <semicolon-separated-file-list-with-wildcard-characters>]

To manually upload the data using the Azure portal:

Navigate to Storage Account → Container → Folder Path, click Upload to transfer objects from the local machine to Azure cloud storage account.

Download Files:

The following commands are used to download the data from Azure blob storage to local system - Reference Guide.

> azcopy cp "<blob-storage-container-path>/[path-to-blob-or-directory][SAS-token]" "<local-folder-path-or-file-path-with-extension>" [--recursive=true] [--include-pattern <semicolon-separated-file-list-with-wildcard-characters>]

# example - without using SAS token to download a single blob
> azcopy cp "https://<storage-account-name>.blob.core.windows.net/<container-name>/file.txt" "C:\Users\file.txt"

# example - without using SAS token to download a directory
> azcopy cp "https://<storage-account-name>.blob.core.windows.net/<container-name>/directory/*" "C:\Users\directory" --recursive=true

# example - with SAS token to download a text file
> azcopy cp "https://<storage-account-name>.blob.core.windows.net/<container-name>/file.txt<SAS-token>" "C:\Users\file.txt"

# example - with SAS token to download only text files using the wild card pattern
> azcopy cp "https://<storage-account-name>.blob.core.windows.net/<container-name>/directory<SAS-token>" "C:\Users\directory" --recursive=true --include-pattern ".txt"

To manually download data through Azure portal:

Navigate to Storage Account → Container → Folder Path, right click the existing objects and click download.

Important Notes:

Use single quotes in all command shells except for the Windows Command Shell (cmd.exe). If you’re using a Windows Command Shell (cmd.exe), enclose path arguments with double quotes (“”) instead of single quotes (‘’).

The phrases encased in <> must be replaced with suitable values, while those enclosed in [] are optional and should be utilized depending on your requirements.

We can also execute the preceding commands through PowerShell, but the azcopy utility is required. More information may be found at source.