How to mount Azure Blob Storage to your local Windows or virtual Linux machines (even with disabled NFS!)
This guide will take you step by step in the mounting process of Azure Blob Storage to Linux and Windows machines.
List of contents
- What is Blob Storage?
- How to manage Blob Storage
- Mount storage container to Azure Linux machine
- Mount storage container to a local Windows machine
What is Blob Storage?
Blob Storage is Microsoft’s object storage solution for the cloud. The term “blob”, which stands for “Binary Large Object”, refers to a collection of binary data that does not necessarily follow any file structure. This means that any file type can be stored together. The Blob Storage maintains files, or blobs, in a large collection of unstructured data. This collection of unstructured data is flat, has no hierarchy and is often called a “data lake” or “pool”. In a manner analogous to a directory in a file system, storage account consists of containers that consist of blobs. Both containers and blobs can both be stored in an infinite number in a single storage account.
Blob storage is designed for serving documents directly to a browser, storing files for distributed access, streaming media files (such as video and audio), writing to log files, storing data for backup, or analysis. Blob storage is hosted in the cloud, making it accessible anywhere through the internet. A detailed explanation of how to create a storage account with blob storge can be found in Microsoft Learn — Create a storage account. Note: you can choose the geographical region where the storage account will be located. Better for you to choose the same region where your VMs are located.
Data Scientists often use Blob Storage for housing training data, creating models, and outputting results. Generally, we have multiple virtual machines for training in a single data science project, however we only want one copy of the data to reduce redundancy, costs, and maintenance. By using blob storage, we can have only one copy of the training data, and that same data can be used for multiple projects Lastly, all trained models and results can be saved in the same place which allows for more convenient results analysis by all developers working on the project.
How to manage Blob Storage
We want to be able to manage and synchronize our files within Blob Storage easily. I have found that the most intuitive way is to mount the Blob Storage to a user’s local file system. This allows a user to read, add, delete, and edit the files as they use their local machine.
Microsoft Azure Storage Explorer (ASE) is designed to manage Azure Storage blobs, but I find that this tool can be slow on Windows for data exploration, which is a common use case for data scientists. Additionally, it is not available on Linux virtual machines. Personally, I use ASE for quick view of my storage account, but I have found that for most of my work blob storge mounting on my machine is a better solution.
Unfortunately, the initial configuration of Blob Storage may disable Network File System (NFS) protocol which cannot be changed after creation and could allow for an easier mounting process. NFS allows the user to mount the local file system other the network, remote hosts, and interact with them as they are mounted locally. As mentioned before, Azure supports this protocol, and an explanation regarding the mounting process using NFS protocol can be found in Microsoft Learn — Mount Blob Storage Using NFS.
There are also instructions for mounting Blog Storage when NFS is disabled which you can find here for Linux, but these instructions are only for Linux. In this article, I will demonstrate mounting for both Linux and Windows using secure and encrypted tools to ensure compliance, without taking up C drive space.
Read more about file sharing solutions provided by NFS in Microsoft Learn — Network File System overview. To follow this guide, you may need to know how to use terminal and power shell, you are familiar with azure virtual machines and blob storage concepts. Additionally, you should already have Azure Linux VM and Blob Storage account with a container.
Find the necessary Blob Storage information
You will need the following Blob Storage information in the next step:
Storage Account Name: MyAccount
Account Key: StorageAccessKey
Container Name: MyContainerName The information regarding the account key can be found in Azure Portal:
- Search for MyAccount.
- On the left navigation bar, find “Security + networking” → “Access keys”.
- Then you can find “Account Key”. Use key1, which you can copy after pressing “Show”.
- For your “Container Name”, use the container name on your local machine which you would like to mount the Blob Storage to. To see the list of existing containers, press “Containers” on the “Data storage” section in the left navigation bar.
Mount storage container to Azure Linux machine
Mount storage container
- Connect to your Azure Virtual Machine (Ubuntu 14.04, 16.04, and 18.04).
- Install blobfuse using the Terminal:
sudo apt-get install blobfuse. - Blob needs temporary space on the VMs disk. Therefore, give access to some space on your VMs HDD/SSD:
sudo mkdir /mnt/resource/blobfusetmp -p sudo chown <youruser> /mnt/resource/blobfusetmp. - Create text file to store your credentials of the storage:
touch ~/fuse_connection.cfg. - Open the text file with Linux text editor:
vim fuse_connection.cfg. - Press “i” to enter the insert mode and add your credentials in the following form:
accountName MyAccount
accountKey StorageAccessKey
containerName MyContainerName - Press the ‘ESC’ key to leave editing mode.
- Press ‘:’ to enter command mode. Then press ‘wq’ to save and exit the file.
- To make restricted access to the file use:
chmod 600 fuse_connection.cfg. - Create a directory to mount the storage drive into:
mkdir ~/mycontainer. - Mount blobfuse:
sudo blobfuse ~/mycontainer --tmp-path=/mnt/resource/blobfusetmp --config-file=./fuse_connection.cfg -o attr_timeout=240 -o entry_timeout=240 -o negative_timeout=120 -o allow_other. - In case the mounting was successful you will be available to explore files in “mycontainer” by
~/mycontainer ls.
Unmount storage container
If you don’t use the data in the container anymore, not saving new information to it or want to change its configuration, unmount it. Use the following commands through the terminal: sudo fusermount -u ~/mycontainer.
Mount storage container to a local Windows machine
Install Chocolatey for individual use
Chocolatey will be used to install all the following required software.
- Verify your ‘Execution Policy’ by running PowerShell as administrator:
Get-ExecutionPolicy. If it returnsRestricted, then runSet-ExecutionPolicy AllSigned. Answer ‘A’ (=Yes to all) to the coming question. - Go to Chocolatey Software | Installing Chocolatey
- Now run the command from the installation page:
Install rclone
Open PowerShell as administrator:
- Run the following command for installation:
choco install rclone. Answer ‘A’ (=Yes to all) to the coming question. - Now we need to config rclone. Run the following command:
rclone config.
“…No remotes found – make a new one…” à answer ‘n’ (=new remote)
“name>” à answer ‘blob’ - Now you will see a long list of options in the PowerShell window. Look for “Microsoft Azure Blob Storage” and remember its number.
“Storage>” à type the option number
“account>” à copy your account name MyAccount
“service_principal_file>” à press Enter
“key>” à copy your account key StorageAccessKey - All other settings can be the default ones so just press Enter.
“Edit advanced config?” à answer ’n’ (=No)- Now you will see the configuration summary. Review it and if everything is correct answer ‘y’ (=yes).
- If everything is set correctly you will see a list of current remotes. Now you can quit by entering ‘q’. Configuration is completed.
- Verify that rclone is able to see the container folder by running the following command:
rclone lsd blob:. If you see your container name, everything is installed correctly.
Install winfsp (windows file system proxy)
On a PowerShell in an administrator mode run the installation command: Choco install winfsp. Answer ‘A’ (=Yes to all) to the coming question.
Mount the blob storage to local windows machine
On a PowerShell in an administrator mode run the installation command: rclone mount blob:MyContainerName local\path\where\to\locate\the\blob\folder-name
Note: folder-name is the name that the blob will get when mounted, so folder with this name should not be in the wanted location. For example, if you wish your blob storage to be in ‘C:\’ inside folder named “MyBlobStorage”, verify that there is no folder with the same name inside and pass the following argument ‘C:\MyBlobStorage’.
You will see “The service rclone has been started”. Now you can open ‘C:\MyBlobStorage’ and explore your blob storage.
The moment you quit the last command in PowerShell, you will no longer be able to see the blob.
Basically, at this point you are done — you know how to mount Azure Blob Storage to your Windows machine. Stopping here, you should remember that each time you want to mount your blob storage, you need to run the last command through PowerShell and leave the PowerShell window open.
In case you want the process to happen automatically, please refer to the following steps.
Move the command to run in the background
- From the start menu open “Task Scheduler”.
- On the left menu bar select Task Scheduler Library.
- In Create Basic Task tab in the ‘Name’ field enter any name that will be informative to you, as “rclone_azure_blob” and then press “Next >”.
- In Trigger tab select When I log on or any other trigger that suits your needs best and then press “Next >”.
- In Action tab select Start a Program and then press “Next >”.
- In program/script field give the path to rclone.exe. Probably it will be something like: “C:\ProgramData\chocolatey\bin\rclone.exe”.
- In the Add arguments add the following:
mount blob:MyContainerName local\path\where\to\locate\the\blob\folder-name --vfs-cache mode full --no-console --log-file= local\path\where\to\locate\the\logfile\log_file_name.txt --config=local\path\to\rclone\config\file\rclone\rclone.confand then press “Next >”. Explanation for the input arguments can be found below.
--vfs-cache mode full à permission to edit files
--no-console à command prompt will be minimized
--log-file à path to locate the log file
--config à path to rclone configuration file - Reminder: rclone configuration file contains the storage account key. To find the location of its file run
rclone config filefrom the PowerShell. - Finally, press “Finish”.
Congratulations! You just mounted your blob storage to your windows machine!
I hope this guide enables you to perform data exploration more easily through your Blob Storage to help you accomplish your work more effectively while leveraging the awesome features of Blob Storage.
