Developing a Serverless solution to detect people with masks
“Build an production grade HTTP Serverless API and Frontend that detects whether people within images are wearing Masks or not. It helps in identifying if masks are not worn by individuals within images.”
Code can be downloaded from https://github.com/RIteshSolidity/MaskIdentifier
The app created here should be provided an URL or binary data of an Image. The Image data or url is send to the Serverless API hosted on Azure functions to detect the people wearing masks. Azure function will use Azure Cognitive Services to get the relevant information and return them back to the Application. The application would render the people wearing masks in Red rectangle. Check the sample images to understand the concept better.
This is an image with a few people in it — some wearing mask while others not.
The same image after rendering within the app is shown next. Note that three people are wearing mask and they are identified using Red Rectangle while others are not highlighted.
Some more examples are shown here.
Usage
- Authorities can use for identifying people not following guidelines.
- Hotels, airports, any means of commute can use to check the compliance related to masks.
- Companies opening up can check for employee and integrate into their existing Covid19 related compliance infrastructure.
- Local meetups can use to check community members if not wearing masks.
- Any place where people gather can be checked for compliance like parks, community halls, shops etc.
Implementation
The front end is a desktop application and can easily be created into Android and IOS app. It expects a file path for an image, loads the file and passes to API to which eventually identifies all the masks along with the coordinates within the image. The coordinates are available as JSON array with each item consisting of left, width, right and height of the masks. The response also contains the score in terms of probability to determine the confidence level whether it actually is a mask.
Serverless is one of the most talked about paradigm in recent times after the combination of (IaaS-PaaS-SaaS) in technology and especially cloud world. CxO’s of almost all enterprises are thinking, discussing their strategies for their next set for digitization, modernization and transformation and Serverless is one of the prominent topics in them. While creating a simple API using Serverless is quite simple and can be done in few minutes; important question to ask is whether it is production ready.
Background
Serverless technically does not mean “without servers”. It means although servers are used to provide compute resources for applications, the customer does not see, manage, govern the server, its infrastructure and Operating system. Customers are responsible for bringing their code and serverless will execute them on any server that it deems fit or fixed for it.
Serverless
Serverless brings numerous path breaking paradigm changes when compares to older deployment models. Some of the important ones are
1. Pay as you consume (PAYC) model — In this model, if a Serverless application is not consumed at all or consumed within a given limit ( the limit differs between cloud providers and availability of their free limit) a customer do not incur any cost to cloud provider. This is especially very beneficial for start-ups and early stage companies that do not have super monetary monetary power to block compute, storage and network on monthly basis from cloud providers (as in case of IaaS as well as PaaS deployments) not ability to purchase hardware for On-premises.
2. Unlimited Scalability — This is again a big advantage for companies that have B2C model and at stroke of luck may see demand for their API’s in millions within hours. Although IaaS and PaaS provides this features using observability and automation, it is not free from delays and cost; moreover requires prior auto-scaling configuration, testing and at times manual intervention. Serverless provides true scalability in terms that it provides SLA in terms of serving all requests and throughput. If it needs to scale to hundreds of servers behind due to sudden spike in demand, it does so without even informing the customer.
3. Out-of-box High Availability — High Availability is one of the tenets customers are moving to Cloud and there are different ways to achieve High availability depending on the type of deployment — IaaS or PaaS. Serverless is highly available by default at cloud region level and there is no configuration required to achieve it. However, to achieve High availability across region, multiple Serverless instances can be created across regions connected by global load balancers.
Covid19
If there is anything that has impact any one of us the most this year or may be from the time we are born, it is the Coronavirus a.k.a. Covid19. It is caused such mass scale health and economic destruction across the globe and it has been unprecedented. There are lots of changes in statutory laws and public health procedures. So much so that people are under lock down and it is mandatory to wear Masks — no matter what kind if stepping out to public places.
The entire scientific community is spending days and nights to find a cure for this virus and technologies companies are creating applications in fortnight related to social-distancing, Contact tracing, people and vehicle movements, approvals from Government, tracking of patients and many more.
In this article, I have tried to provide the steps to create a simple REST API using Azure Functions (although any cloud provider can be used) to detect whether individuals within an image are wearing Masks or not. The Focus has been to create production ready Azure function based REST API by means of detecting and identifying Masks within images.
Technology Used
1. Azure functions
2. Azure Cognitive Services Custom Computer Vision API
3. Visual Studio 2019
4. POSTMAN
Approach
- Started with creating an instance of Cognitive Services Custom Vision on Azure.
- Then identified more than 50 images from Bing/Google that has people wearing masks and another set of 50 images in which people are not wearing masks. The images were identified keep in view
Different backgrounds
Different shades, color and lighting
People in Groups and individual
Different camera angles
- Uploaded the images to Customvision.ai portal and tagged all the images with — mask or no mask
- Trained and published the model
- Created a new Azure Function project using Visual Studio 2019 and implemented the logic for the REST API
- Conducted unit testing using MOQ and xUnit
- Deployed the API on Azure Function APP
Prerequisites
- Access to Azure portal and RBAC (Role based access control) permissions to create resources. docs here and here.
- Visual Studio 2019 environment for implementation and local testing of code. docs here
- Postman to invoke Function API. It can be downloaded from here
Configuration
In order to use this application, there are some configurations you should do first. local.settings.json within the project:
// local.settings.json{"Values": {"cognitiveapi": "{{url of Cognitive Services}}","cognitivekey": "{{Access Key of Cognitive Services}}"}}Developing the Application
The first step is to create Cognitive Service Custom vision Service on Azure, upload images and tag them between “mask” and “nomask”. After Tagging the entire model should be trained and tested.
After appropriate testing, it should be published. This will generate the url that should be consumed from Azure functions.
The next step in to create a function app using HTTP Trigger in visual Studio 2019. The Visual studio scaffolding will create a default function. We will remove the Get method and provide implementation for connecting to Cognitive Service and get appropriate response from it.
The settings file is updated with Cognitive Services URL and key.
The code for the function is upload and can be found here.
Write unit tests to test the function and upload the function to function app.
Use the application
The API has been deployed on Azure within Function App and can be invoked using postman.
The function accepts POST request in JSON format. The same can be configured within Postman.
The body of the request should contain the URL of the image that needs to be evaluated for mask identification. The payload should be send using JSON payload as shown here.
{
“url” : “https://deststorage13.blob.core.windows.net/kokokokok/5e71267bc4854010b62f5925.jpg"
}
The postman should be configured as shown here.
It would return
To run in Azure
- Right click in VS Code left menu and click Deploy to Function App…
- Go to portal and add the following keys to the function app’s APP SETTINGS
“ cognitiveapi”: “<<COGNITIVE_URL>>”
“ cognitivekey”: “<<COGNITIVE_KEY>>”
Solution hosting options
Azure functions — this has been used for the solution
Azure Kubernetes services within Pods
Azure App Services
virtual machines
Containers on Virtual machines using images with Azure Function runtime
Azure Functions was chosen as deployment target since it provides high availability and unlimited scalability out of the box. It is also cost effective (cost based on usage) compared to other options where there is fixed cost associated with underlying virtual machine. AKS will have fixed cost for worker nodes.
Assumptions
- English as the language of communication
- The response from API when greater than 75% probability for Mask is shown as “Mask” otherwise as “nomask”
- the API uses HTTP POST method and did not use GET method — although it could have been used.
- JSON is the choice of data serialization.
- Instead of uploading the entire file to Cognitive services, URL of the images is passed as POST request to API. The Cognitive Services is able to download the image locally and provide its analysis.
steps to deploy
- provision a Function APP in a region
- Provision a storage account in same region. All resources should be in same region
- Clone the solution in local environment
- publish the function app project to Function APP. Please remember to update the Azure App Service settings in visual studio before publishing the function to function app. Also manually edit the function app configuration to add the connection string of table storage.
5. Update the Cognitive Services URL and key either in visual studio as shown in next image or from portal.
Few architectural concerns that should be taken care before deploying the API on production has been listed next.
Security consideration
Azure Function provides numerous ways to invoke functions and HTTP trigger based function needs addition protection since they can be invoked by anyone who knows about the URL.
Foremost, the endpoint must be protected using TLS 2.0 at minimum. It means that they only way to reach out to the service is by means of HTTPS. This would ensure the request-response messages are encrypted and that only the intended participants can understand the messages. So, the next question arises how do we know who are the intended participants and how can vouch their Identity. For a customer facing endpoint, it is not prudent to manage the identities locally and manage them actively. Instead, it is better to delegate the identity verification process to an established identity providers like Facebook, Twitter, google etc. They all implement Open ID connect and oAuth protocols and with trust build between the API and these Identity providers, it is possible to authenticate and authorize users before accessing the API.
The API also should be behind a firewall. It means that only valid and allowed requests should be allowed to reach the API. Well known security threat assets should be configured on the firewall to filter any unwanted traffic from suspicious IP addresses, port and files.
The API can be hosted within an API management service that can manage and administer the API. For example, Azure API management can be used for rate limit the requests to the API based on the kind of user and plan they have subscribed for, generating unique keys to access the API and many more other benefits.
Automated Build and Releases
The API should have a well-defined build and release process. The build pipeline should ensure the readiness of the API to be deployed on any environment starting from Development eventually to production environment. Both the build and release pipelines should have quality checks passed before moving from one environment to another environment. At minimum some of the checks that should be performed are unit tests getting passed, code coverage of both unit tests as well as API itself. Code quality using tools like StyleCop, FxCop and SonarCube. Static as well dynamic code security checks to identify an password, credentials or any other secret and confidential information not getting leaked within the source code, integration between multiple module etc.
Code Quality
Ensure proper logging and exception handling is implemented within the code. This should ensure that there should not be leakage of any sensitive information to both logs as well as to users invoking the API.
The code should have minimal hard-coding and secrets should be derived from vaults like Azure Key Vault and non-sensitive information from configuration files.
Reuse code and each function should not be very length. The cyclometric complexity of any function should be less than 10.
There should not be long running loops since Serverless requests are time-boxed and they should be stateless (should not be session dependent or require sticky sessions).
Test your application enough to ensure all edge cases along with regular cases have been addressed.
Monitoring
The API should be actively monitored using Azure App Insights, Log Analytics, Security center and Sentinel for Performance, Health, Security, SLA, KPI and other aspects like exception and traces.
Resources/Tools
- Visual Studio
- Postman
- PowerShell
Next Steps 🏃
- Write more unit tests
- Instead of just capturing whether mask is in picture or not, we can get all faces and whether mask it there or not on faces with probabilities
Important Resources ⭐️
Here include all the important resources related to the article
