Discovering Azure’s Computer Vision and Cloud Search Services — Part 3

Intro

In the first post I described a challenge which I had set myself and investigated two Azure services which would be suitable. Using a free trial subscription I encouraged readers of any background to give these services a try. In the second post we went a bit deeper into the Search service and used Azure Notebooks to write and test some code.

In this third and final post we’ll complete an end-to-end solution which will extract text from an image and match it to our internal catalogue, without any deep expertise in either machine learning or search products/algorithms. Recall, I wanted to use an event-driven, serverless compute framework and was aiming to complete the solution in a few days. My ideal architecture is fully managed, loosely coupled and asynchronous, built to:

• process an image in near real-time
• eliminate the overhead of managing servers, backups, upgrading etc.
• scale according to demand without placing additional strain on any existing/on-prem systems or databases
• be resilient and fault tolerant, i.e.workloads can be restarted if there is a failure and our process is not dependant on the primary database being available

Proposed Architecture

In the diagram below, I’ve included a few components which we haven’t discussed yet but I believe they would be beneficial in the final solution. So let’s walk you through the architecture…

Assume that in our scenario, images arrive on blob storage via the mobile app but in practice they could originate from a number of sources. Once they land, this will trigger a small serverless function to add them to a queue. This may be a controversial move, but I believe it gives us additionally flexibility as described at the end of the last post. Most importantly though, if our main logic is crashing for some reason (perhaps someone pushed some bad code) we won’t have to “replay” any workloads, we’ll simply fix the problem and processing will resume. Our main function in the middle will process messages from the queue and run our core logic, the result of which will be stored in the SQL database along with a reference to the image.

One might ask why don’t you push the results to another queue instead of the function directly interacting with the database? Great question, and indeed you could and sometimes you must, depending on a few things. 1.) How likey is the database to fail or someone will break the schema, like drop the table! 2.) Would the number of inserts being made by our function during peak operation “hurt” the database’s performance. 3.) Do you have a user/license restriction on the database. A queue could act as a great buffer to 1.) control the number of sessions and transactions writing back to the DB, and 2.) reduce the strain on the DB by batching writes. You would need another small function probably on a timer if you wanted to batch/buffer, which would read from the queue and write to the DB.

Once the results are in the database, a web application could allow the users to perform their daily operational activities on the data. Additionally the raw json output from Vision and Search service could be stored in Azure Cosmos DB along the same reference to the image. This way, technical agents using the web application can debug and analyse the raw data should queries arise. Cosmos DB is a fully managed multi-model database, perfect for storing semi-structured data like JSON.

Finally, you may want to keep track of the searches that aren’t yielding any results or monitor how long each process is taking by sending telemetry to Application Insights. You can visualise these results and analyse for issues in Power BI.

Create a Queue and Container

Let’s set up our queue in the same general purpose storage account we created in Part 1 of this blog. As always there a ton of ways to create one, but for simplicity let’s use the Azure Storage Explorer. Expand the storage account and right click on Queues and create a queue, that simple! While you are there we also need a new blob container to store the images. Initially we used the static website hosting feature to access our files but this was a bit of “cheat” to make the file publically / easily accessable. Add a new container under blobs and make note of the name of your container and the queue. Mine now looks like this:

Azure Functions

As described in our architecture, we’ll need two functions, one to add items of work to the queue each time the images arrive and another to process the images.

First let’s create a simple serverless function which will be triggered each time a new object (or image in our case) is uploaded and add it to the queue. Follow the first two steps of this guide to create your first function app and then the trigger. The only exceptions are that you should specify your existing storage account when creating the function app and then add the name of the container path (created above) which the will be monitored for events.

Then follow this guide to add the queue as an output to your function. Mine looked like this:

Instead of adding the code in the guide, add this line of code below to the first line of the function. If you get stuck, here’s the full code. Essentially, it takes the name of the object, in our case the filename, and it adds it to the queue.

context.bindings.outputQueueItem = context.bindingData.name;

At this point you may have run a test and found that nothing happened. The first time your function is run it may need some time to “warm up”. This is known as a cold start and is the expected behaviour on the Consumption Plan.

Note, for that simple function we used Javascript because it’s a supported language and has a basic template, but to run our main function in Python we’re going to need a bit more setup so please hang in there! Out of all the guides, I found this one to be the most detailed and accurate, so please follow along and I will call out below any deviations along the way…

Initially, there is no need to create a new function app, you can use the existing one created earlier, but instead of an HTTP trigger, create a Queue trigger. Then, copy this code into the function. It has a few differences to our Azure notebook which I will describe later. Enter your configuration details at the top of the script then click Save.

Next follow the guide, access the Kudu debug console (in CMD not PowerShell) but you can skip the update of the Python version, as the code I’ll provide should work just fine. Note, there’s then no need to install the virtual environment module, but you need to still need to create the virtual environment. Be patient as it may take a little time.

After that, activate the environment, and the only module I needed to install was the requests module.

D:\home\site\wwwroot\process-work\acmevenv\Scripts>activate.bat
(acmevenv) D:\home\site\wwwroot\process-work\acmevenv\Scripts>python -m pip install requests

One of the main differences between the Notebook version of the code and the Function version is the way the script obtains the image from the blob container. It is no longer a publically accessible image, so one needs to constructing a SAS URL to access the object.

The Pay-off

Now the time has arrived where you get to see whether all your hard work and dedication has paid off — let’s run an end-to-end test.

For our finale, I found a Tesco recipt online with a funny story behind it and would make any parent smile — the struggles of parenthood summed up in a receipt! I added some more metadata to my database tables, re-indexed, and uploaded the image to the drop container…

Image uploaded at 5.27PM

Blob trigger function fired adding it to the queue

Main function fired identifying store and products.

Success, but spare a thought for those poor parents!

Being event driven the entire process took just under a minute to complete (the function log is using UTC not BST) and will be able to scale according to increased workload. You’d probably need to add a few more seconds to write the result back to the database and any other logging like Application Insights or sending debug information to Cosmos DB. If you were following along I trust you were able to achieve the same result fairly easily.

Conclusion

Thanks to pre-built and fully managed services in Azure I was able to put together a small POC fairly quickly and fulfil all of the goals I had set for this POC. Both the Vision and Search services are really simple to start with but like all good products they have a number of “deeper” features which can be explored as your use-case evolves.

I am confident that with this architecture, we have a scalable, fault tolerant design. Additionally, the impact on the source database is minimal and could be managed/throttled to suit. Some may have questioned my use of Azure Search in this kind of scenario… Traditionally, the Search service has been used to help customers find products on your website or provide auto-suggestions in search queries, but my rationale is that I wanted to offload this intensive task from the database, and at the same time I didn’t want to have to write my own matching algorithm. I suspect many companies could benefit from implementing Search across their data, and particularly Azure Search makes it easy to index your data from your SQL database, webpages, PDFs or Office documents! Having information at your fingertips is a powerful thing. In fact, I think that would make another great blog in the future post so watch this space ;-)

Hope you enjoyed discovering these services and building the POC as much as I did. Please feel free to leave comments or drop me a message should you have any questions.