ADLS Gen2 API with C#
Intro
In a previous article (here) I looked into the basics of Azure Data Lake Storage Gen2 (ADLS Gen2), setup a storage account, and looked at some basic auth calls to interact with the ADLS Gen2 API. Given the ability to create a storage account and authenticate calls against the ADLS API, it only makes sense to show some basic CRUD style calls to dig a bit deeper into the API. This article will walk through some basic operations (using C#) to expand on the work started in the previous one. Full API documentation on the ADLS Gen2 API can be found from MSFT here.
Setup
First and foremost make sure you have a storage account with hierarchical storage enabled.
The examples below were done as part of a C# console app (.NET Core 3.0), using Visual Studio Community Edition, Version 16.4.1. The only NuGet package I installed was Microsoft.Azure.Services.AppAuthentication which allows the generation of a JWT Bearer token using Visual Studio credentials when developing locally, and MSI when deployed. This makes it very easy to authorize our API calls. Ill be looking at each method individually, but the full class can be found here.
One last note, these methods are only a demonstration of using the API from C#, many production quality conventions (DI, try/catch, logging, etc.) have been omitted for brevity. That said, lets dig in.
Authorization
This method will look familiar from the last article, you will need the tenant id of the tenant in which the ADLS account is located. If you are running from Visual Studio, the library will pull your credentials from VS to authenticate with Azure services. This method will get called by all other methods to provide the bearer token.
Create a Filesystem
Before creating a new filesystem we’ll need a few values:
The _adlsStorage variable is the storage account name I created in Azure (in my case dcstoragetest02), _adlsSuffix is universal, so no need to modify that, and finally _rootFileSystem is the name of whatever you want your filesystem to be called (every example here is based on the same filesystem “dansfilesystem2”, but you can setup however you like).
Creating a new filesystem is a PUT call against the storage account, with the values defined above:
Uploading a file
This upload example takes place in 3 parts, first a PUT with the filesystem name, directory path (in my examples the directory is simply ‘testpath’), and file name as the URL. Once that call returns success, we open a stream and provide that as the content to a PATCH call with the same path. This second call adds the parameter ‘action=append’ and the starting position 0. The final call is another PATCH with ‘action=flush’ to persist the file to the filesystem.
File and Directory Info
Now with the ability to create files and filesystems, lets looks at some simple ways to get metadata back. The first call is a way to enumerate a directory, and using the ‘recursive’ parameter it can crawl the hierarchy starting at the depth given.
As an example, I have 2 files in the filesystem ‘dansfilesystem2’ under the directory ‘testpath’. In this case, if I pass in ‘testpath’ as the directory path, and set recursive to ‘true’, it will return the 2 files under that directory:
Next we’ll look at individual file info, by making a HEAD call to the file path (full path including the filesystem and any directories). The metadata is returned in the Headers of the response.
Notice reading the response Content yields nothing of real value, its all in the Headers.
Downloading a file
Download is straightforward, simply provide the path to download, and read the response. This example reads the entire file, but you can grab pieces using range requests as well.
Deleting a file
Lastly, lets delete a file, again a straightforward operation, send DELETE with the full path to the file… and its gone!
Wrap up
While the API has some quirks and strangeness, overall its fairly straightforward to use. Hopefully these examples can help jumpstart your next project!