Azure Cognitive Services with data in VNET Protected Storage

Objective

Published in

Microsoft Azure

7 min readApr 15, 2021

This blog post is again on one of my favorite topics which lies right between networking (infrastructure concern) and an Azure Data Service. In this article I highlight different methods of passing data (mostly images) to an Azure Cognitive Service for predictions including common scenario where data is stored in a VNET Protected Azure Storage account. I am using Computer Vision Service with Python in this article as an example but same concepts should port over to other Cognitive Services as well as programming languages.

Scope

Another thing to keep in mind is that some of the Cognitive Services allow customization (Azure Custom Vision, Azure Forms Recognizer, etc.) where you as a consumer can further train the ML Model used by the Cognitive Service for your domain/project. When it comes to VNET Protected Storage Account for image data I am focusing primarily on Prediction side of things and not Training (customization of model) as the solutions can vary between services.

Disclaimer

Lastly, I would like to re-iterate my usual disclaimer — things are changing fast what I am documenting now (April 2021) might get stale so I will try my best to come back and update my article but no promises.

Methods for passing data to Cognitive Services for prediction

The most common scenario when using Azure Cognitive Services is to perform predictions for images in one of the following three method:

Image accessible anonymously using Public URL
Image in Azure Storage accessible using authorized Shared Access Signature URL
Local Image file on Disk — Slight variation of this is what needs to be used when image is in Storage Account which is VNET Protected

Let’s dig into these three methods in more detail.

Image Accessible anonymously using Public URL

Most of the basic tutorials use this method, this is to keep things simple because the intent is to have the user quickly get started and understand the core concepts (or business value) provided by the Cognitive Service rather than getting bogged down in operational details.

Public URL for an image can something like this -https://upload.wikimedia.org/wikipedia/commons/thumb/1/12/Broadway_and_Times_Square_by_night.jpg/450px-Broadway_and_Times_Square_by_night.jpg
Or, it could be a file in Azure Blob Storage with Anonymous Read Access enabled with URL like this — https://xxxxxx.blob.core.windows.net/scratch/landmark.png (Azure Storage Blob Anonymous Read Access documented here )

Computer Vision Cognitive Service sample code snippet used for prediction on such a publicly available image —

remote_image_url = "https://raw.githubusercontent.com/Azure-Samples/cognitive-services-sample-data-files/master/ComputerVision/Images/landmark.jpg"description_results = computervision_client.describe_image(remote_image_url)# Get the captions (descriptions) from the response, with confidence level
print("Description of remote image: ")
if (len(description_results.captions) == 0):
    print("No description detected.")
else:
    for caption in description_results.captions:
        print("'{}' with confidence {:.2f}%".format(caption.text, caption.confidence * 100))

Refer to the full sample code which uses SDK with an anonymously accessible image using Public URL — https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/quickstarts-sdk/image-analysis-client-library?tabs=visual-studio&pivots=programming-language-python

Image in Azure Storage accessible using authorized Shared Access Signature URI

The Public URL with anonymous access is for basic tutorial purpose, for most enterprise scenarios any kind of proprietary or confidential images are not going to be available with anonymous access so Shared Access Signature (SAS) method seems to be the most logical choice to pass data to Cognitive Service from Azure Storage. As far as the code goes, SAS URI for a file in Azure Storage is pretty much the same as the approach used for anonymously accessible public URL (I assume you are familiar with SAS URI, SAS URI can be generated from Azure Portal or using Azure Storage Explorer which is documented here).

Computer Vision sample code for prediction on image in Azure Storage using SAS URI —

remote_image_url = "https://xxxxx.blob.core.windows.net/scratch/landmark.png?sp=r&st=2021-04-12T20:41:35Z&se=2021-04-17T04:41:35Z&spr=https&sv=2020-02-10&sr=b&sig=dlBaueG1ja%2FXXXlXO3xGSXBTzLuoSSSeiV%2BzSxVLCA%3D"description_results = computervision_client.describe_image(remote_image_url)# Get the captions (descriptions) from the response, with confidence level
print("Description of remote image: ")
if (len(description_results.captions) == 0):
    print("No description detected.")
else:
    for caption in description_results.captions:
        print("'{}' with confidence {:.2f}%".format(caption.text, caption.confidence * 100)) Instead of Public URL, you just specify URL with SAS Token at the end like —

The challenge with SAS URI method of passing image to Cognitive Services is that it won’t work if Azure Storage Account is VNET Protected.

Local Image on Disk

In cases where Image is available as local file following code snippet can be used -

with open("./TimeSquare.jpg", "rb") as local_image:
    description_results = computervision_client.describe_image_in_stream(local_image)
    
print("Description of remote image: ")
if (len(description_results.captions) == 0):
    print("No description detected.")
else:
    for caption in description_results.captions:
        print("'{}' with confidence {:.2f}%".format(caption.text, caption.confidence * 100))

Refer to the full sample code which uses SDK for local file here — https://github.com/Azure-Samples/cognitive-services-quickstart-code/blob/master/python/ComputerVision/ComputerVisionQuickstart.py

Image in VNET Protected Azure Storage (no anonymous access) — Solution is a variation of Local Image on Disk

As I stated above that SAS URI method of passing image to Cognitive Service does not work when the image is in VNET Protected Storage Account, the desired solution is a slight variation of the Local Image on Disk.

Cognitive Services themselves support VNET Service Endpoint and Private Endpoints so common misconception is that once once Cognitive Service is VNET Protected both Storage and Cognitive Service are in the same VNET and they should be able to happily talk with each other. Its important to understand that VNET Service Endpoint and Private Endpoint are features to give PaaS services look & feel of being in a private VNET but they are not truly in a VNET hence this approach doesn’t work. Both VNET Service Endpoint and Private Endpoint technologies are to allow inbound connections to the service on which they are enabled (not for the outbound connections made by the service itself).

The VNET security is at times not well understood so I have the following screenshots of VNET Protected Storage and Cognitive Service to be clear on what I mean by VNET Protected resource.

Cognitive Service — All Networks radio button is not Selected, either of the other two selected

Cognitive Services — VNET Protected — VNET Protected Cognitive Service

Storage Account — Selected Networks selected

Passing data using SAS URI with VNET Protected Storage — Does not work

The following diagrams show the architecture where Storage Account and the Cognitive Service both are VNET Protected and allow connections from specific VNET. I have chosen to show VM, Azure Databricks and AML Compute Instance/Cluster in a VNET as an example compute where code runs but this could be Azure Function with VNET Integration as well or some other Azure Compute Service in VNET. The diagram accurately depicts the interaction between services and shows why it doesn’t work. When image location is passed using SAS URI, its Cognitive Service which tries to reach out to Storage Account since Cognitive Service is not inside a VNET the connection will be denied.

VNET Protected Cognitive Service and Storage — VNET Service Endpoint — SAS URI

Following diagram shows the same when Private Endpoint is used for VNET Protection instead of VNET Service Endpoint.

Storage and Cognitive Service Private Endpoint — SAS URI

Passing data from VNET Protected Storage — Solution

The solution for the scenarios where image resides in a VNET Protected Storage Account is very similar to the solution where image is read from a local disk, instead of reading image data from local file read from Storage Account into memory and then send to the Cognitive Service as part of the request. The following diagrams shows the architecture where code running on Azure Databricks created in customer VNET (or VM or AML Compute Instance/Cluster in VNET) reads image data from Azure Storage and then communicates with Cognitive Service passing the data in the request itself eliminating the need for Cognitive Service to talk to Azure Storage Account and this is the desired solution which works with the VNET restriction in place.

Cognitive Service — VNET Protected Storage — Read from Storage and Pass to Cognitive Service

Sample Code for such a solution where code reads the image data from Storage Account and passes the data as part of the request to Cognitive Service.

from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
import iostg_conn_str = "xxxxxxx"blob_client = blobClient.from_connection_string(conn_str=stg_conn_str,                                         container_name="computer-vision",                                        blob_name="TimeSquare.jpg")with io.BytesIO() as blob_stream:
    blob = blob_client.download_blob()
    blob.download_to_stream(blob_stream)
    blob_stream.seek(0)
    description_results = computervision_client.describe_image_in_stream(blob_stream)

Refer to sample code which reads image file from disk here — https://github.com/Azure-Samples/cognitive-services-quickstart-code/blob/master/python/ComputerVision/ComputerVisionQuickstart.py

Documentation for Azure Storage Python SDK used for interaction with Storage Account — https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python

N͟o͟t͟e͟: Depending upon your preference and needs there are other alternatives to downloading and passing the data as stream, you can download the file to a local disk and then send it to Cognitive Service or certain Cognitive Services allow a byte array so that you can download file to an in-memory array and then send it over to Cognitive Service.

References

Above I have shared some code snippets as well as links for Python SDK but you should be able to achieve the same with REST API Calls if you prefer to use REST API, for completeness below are the links for sample code for Computer Vision and if you dig little bit on public docs you should be able to find similar samples for other Cognitive Services as well.

Python REST API Sample for Public URL/SAS URL — https://github.com/Azure-Samples/cognitive-services-quickstart-code/blob/master/python/ComputerVision/REST/python-analyze.md
Python SDK Sample for Public URL/SAS URL — https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/quickstarts-sdk/image-analysis-client-library?tabs=visual-studio&pivots=programming-language-python
Python REST API Sample for Image on Local Disk — https://github.com/Azure-Samples/cognitive-services-quickstart-code/blob/master/python/ComputerVision/REST/python-disk.md
Python SDK Sample for Image on Local Disk — https://github.com/Azure-Samples/cognitive-services-quickstart-code/blob/master/python/ComputerVision/ComputerVisionQuickstart.py