Image for post
Image for post
Android Object Detection Power App that we will build in this article

You might have an idea for an application that detects an object or image and not have anyone to build it. You might be a programmer and want to try out your ideas and Azure Custom Vision before you start writing code. No matter what your coding skills are, if you want to prototype an application with AI, Power Platform and it’s AI Builder feature is a fast and easy option.

This article explains what Power Platform is, as well as go through a step by step process to create a mobile app that takes a photo and detects objects. If you want to dive right in, skip to “How to detect objects from images?” section.

Check out the video below to see the app we will build to detect different Mixed Reality Headsets such as HoloLens version 1 and 2 Augmented Reality and Virtual Reality headsets and their hand controllers.

Azure Custom Vision Head-Mounted Display(HMD) detection app demo

What is Power Platform?

Image for post
Image for post
Power Platform tools

What are Power Apps?

A connector is a proxy or a wrapper around an API that allows the underlying service to talk to Microsoft Power Automate, Microsoft Power Apps, and Azure Logic Apps. It provides a way for users to connect their accounts and leverage a set of pre-built actions and triggers to build their apps and workflows. For example, you can use the Twitter connector to get tweet data and visualize it in a dashboard or use the Twilio connector to send your users text messages without having to be an expert in Twitter or Twilio APIs or having to write a line of code.

Check out the list of connectors for Power Apps to see all the APIs that are available. Notice Power Automate or Logic App connectors might not be the same.

What is AI Builder?

AI Builder is one of the additional features of Power Apps. With AI Builder, you can add intelligence to your apps even if you have no coding or data science skills.

Image for post
Image for post
Power Apps: AI Builder app view

What are some of the use cases for AI Builder?

  • Detect objects from images
  • Analyze your customers’ sentiment from feedback
  • Detect keywords from text
  • Extract specific information about your business from text

Is AI Builder the right choice?

I have a rule of thumb when I want to build something, use whatever is available and easy to use first. When your needs exceed what the tool you are using covers, look into another solution or building it yourself.

Use the tool easiest to get started when you are building your idea. When your needs exceed the capabilities of the tool you are using, find a solution that enables you. Don’t invest in building things from scratch before you know it is worth it to do so.

For example, if you have an app idea, it is better to have a prototype running as easily as possible. You can test your ideas before investing your time into building custom designed UI or features. In our specific case, you can first prototype your app with the drag and drop UI of Power Apps and using prebuilt AI models. When your specific needs surface, such as recognizing a particular object or keyword, you can invest your time into creating your custom models to train for the object or keyword detection.

Can I use Power Apps and AI Builder for production?

What will my production app cost?

Image for post
Image for post
AI Builder Cost Calculator

What are preview features?

The General Availability (GA) release will occur in a phased manner, with some features remaining in preview status while others are released for GA. You can check out the release status on the AI Builder documentation.

Image for post
Image for post
AI Builder preview features notice

What is Object Detection?

Object detection can detect up to 500 different objects in a single model and support JPG, PNG, BMP image format or photos through the Power Apps control.

Image for post
Image for post
Try out Azure Computer Vision

You can try out and see how object detection works before having to create and accounts or apps yourself on the Azure Computer Vision page.

What can you do with Object Detection?

  • Brand logo recognition
  • Wildlife animal recognition

How to detect objects from images?

Image for post
Image for post
Build Object Detection on Power Apps — AI Builder

Name your new AI model with a unique name. Select Common Objects and proceed to next section.

Image for post
Image for post
Train your custom vision model screen for Object Detection.

Name the objects that you are going to detect.

Image for post
Image for post
Name each objects to be detected

Upload images that contain the object you will detect. To start with you can upload 15 images for each object.

Image for post
Image for post
Training image format and size for object detection

Make sure each object has approximately the same amount of images tagged. If you have more examples of one object, the training data will be likely to detect that object when it is not.

Image for post
Image for post
False positive HoloLens 2 detection

Tag your objects by selecting a square that your object is in and choosing the name of the object.

Image for post
Image for post
Tagging objects

Once you are done choose Done Tagging and Train. Training process will take some time.

Image for post
Image for post
You can choose to not use images after you upload and tag them

If you choose to not use an image or clear any tags, you can do that at any time by going back to your model under the AI Builder on the left hand side menu and choose your model and choose edit.

AI Builder will give you a Performance score over 100 and a way to quickly test your model before publishing. You can edit your models and retrain to improve your performance. Next section will give you some best practices to improve your performance.

Image for post
Image for post
Performance score of trained model

How to improve Custom Vision Model performance?

To help facilitate this process, AI Builder allows you to have multiple versions of your model so you can use your model and continue to improve it at the same time.

What are some best practices for training for object detection?

Image for post
Image for post
HoloLens version 1 and 2 training picture with different background and view angle
Image for post
Image for post
HoloLens 2 training data
  • Use images with variety of backgrounds. Photos in context are better than photos in front of the neutral backgrounds.
Image for post
Image for post
HoloLens and VR headset data with different background
  • Use training images that have different lighting. For example, include images images taken with flash, high exposure, and so on.
Image for post
Image for post
HoloLens training data with low light
  • Use images of objects in varied sizes. Different sizing helps the model generalize better.
  • Use images taken from different angles. If all your photos are from a set of fixed cameras such as surveillance cameras, assign a different label to each camera. This can help avoid modeling unrelated objects such as lampposts as the key feature. Assign camera labels even if the cameras capture the same objects.
Image for post
Image for post
HoloLens training image with different size and angle

How to share your Custom Vision models?

If you want others to use your model, you can share it with specific users, groups, or your whole organization.

How to use your Custom Vision model in a Power App?

Image for post
Image for post
Use your model in a new Power App

You will be redirected to Power App editor and an Object Detection component that uses your model will be added automatically. In the editor, you can add new pages to navigate, design and customize your pages.

Image for post
Image for post
Your AI model used in the Object Detection component

Once you are happy with the design, you can publish and share your app. You can use your new app by downloading Power Apps from Apple, Android or Microsoft stores. Once you sign in, your app will be listed in the mobile Power Apps.

Image for post
Image for post
Power App in Android Play Store

What’s next?

Should I keep using my power app or rebuild it?

What would be the changes that require the upgrade? There are two possibilities for the changed requirements for your app:

  • Feature
  • Budget

How to create a custom feature for Power Apps?

A connector is a proxy or a wrapper around an API that allows the underlying service to talk to Microsoft Power Automate, Microsoft Power Apps, and Azure Logic Apps. It provides a way for users to connect their accounts and leverage a set of pre-built actions and triggers to build their apps and workflows.

Check out the list of Power Apps Connectors and how to build a custom connector yourself.

How to compare costs for Power Apps and Logic Apps?

Additional Resources

Thank you for your attention! Leave a comment below for your AI application use cases and the tutorials you would like to see.

Microsoft Azure

Any language.

Ayşegül Yönet

Written by

Senior Azure Cloud Developer Advocate at Microsoft and focuses on Cognitive Services, Spatial Computing and WebXR. She is a co-chair of W3C Immersive Web

Microsoft Azure

Any language. Any platform. Our team is focused on making the world more amazing for developers and IT operations communities with the best that Microsoft Azure can provide. If you want to contribute in this journey with us, contact us at medium@microsoft.com

Ayşegül Yönet

Written by

Senior Azure Cloud Developer Advocate at Microsoft and focuses on Cognitive Services, Spatial Computing and WebXR. She is a co-chair of W3C Immersive Web

Microsoft Azure

Any language. Any platform. Our team is focused on making the world more amazing for developers and IT operations communities with the best that Microsoft Azure can provide. If you want to contribute in this journey with us, contact us at medium@microsoft.com

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store