Creating Composable Analytic Applications with Oracle Visual Builder, Analytics and OCI Vision AI Services
Introduction
At the 2022 Gartner Analytics conference in London I began learning about the Oracle Vision AI service. This service caught my imagination with its possibilities, and it’s easy to try it out from the Oracle cloud console (navigate to the Vision AI service) by either uploading an image or referencing an image from object storage.
At the same conference, I was introduced to the concept of ‘composable applications’ where ‘composable’ is a term used by Gartner to refer to the low-code creation of tailored analytic experiences using building blocks of different services. At least, this is my interpretation of the term ‘composable’ and it got me wondering what is possible with the Oracle OCI Cloud services.
I’ve already blogged about a web component that makes it easy to embed Oracle Analytics content into a low-code Visual Builder application and in this blog I’ll describe how I’ve used the ‘composable’ concept to build an application that can take an image, perform analysis using the Vision AI service and generate suggestions for ‘labels’ to act as filters to embedded analytics. I’m working with a retail scenario, allowing filtering of analytics based on types of toys detected in an image and the resulting comparison of data for those toys.
This isn’t intended to be a step-by-step guide to building this app, rather an overview of what’s possible with the services I’ve used. One of the great things about app development is that there are multiple ways to reach the end goal and hopefully you can use the ideas presented here with your own development methods.
Here is a demo of the resulting application, running on a device simulator in Visual Builder:
Architecture
The component architecture and high-level process flow for this application is shown below:
Building the Application
Visual Builder provides numerous templates and layouts when choosing to create a new application. I’ve chosen to build a mobile application, to take advantage of the Progressive Web Application (PWA) capabilities of VB.
This is the app page structure as seen in VB:
Quick Tip: When developing in VB, I often temporarily use UI controls such as text fields to help me debug the application and check the values of variables or the body of REST API calls (it’s also possible to output to console.log() these variables when debugging). This is how it would look mid-development where you can monitor the creation of REST API payloads and JSON responses:
Capturing the Image
A key aspect of VB that enables low-code app development are the various UI components that can be dragged onto your application page. I’ve used several of the ‘device’ components, such as location services in other projects, specifically using the location services of a mobile device to pass a geo-location filter to Oracle Analytics. In this application, I’m using the UI control to take a photo using the device camera.
Once an image has been captured, an event fires to execute a VB action chain. Action chains are key to an application’s functionality and process control. In the case of my application, the action chain assigns the following properties of the captured image to the following variables:
capturedFile = $variables.files[0] capturedImage = URL.createObjectURL($variables.files[0]) capturedFileName = $page.variables.capturedFile.name
Since this application is intended to illustrate a proof of concept, I’ve separated out some steps and used buttons to invoke processes that ordinarily would be automated. One such step that I’ve separated out is uploading the captured image to Object Store. The Vision AI service can use images either sent in the API request payload or referenced from object storage. I’ve chosen to store the image in object storage in case I want to use it later.
I’m using the object storage API to upload the captured image. This process is invoked when a user presses a button and an event invokes an action chain to call the object storage API with a PUT request. This is a straightforward action chain, note the mapping of input parameters and also mapping of the request body to a variable. I’ve noticed this approach to build the REST payload separately can be useful, especially in cases where the API body is created dynamically from variables.
In this action chain, it’s worth highlighting these parameters:
objectName = $page.variables.capturedFile.namebody = $page.variables.capturedFile
Also note the Content Type parameter is mapped to:
[[$page.variables.capturedFile.type]]To complete the action chain, I’m showing a transient confirmation when the file has been uploaded (i.e. a green popup message).
REST Endpoints
Before I cover the remainder of the application, I wanted to describe how the REST endpoints are defined in Visual Builder (VB). Since VB is targeted at low-code app dev use cases, endpoints can be defined using wizards that create Service Connections. This is also where the authentication to those endpoints is defined. In the case of OCI cloud services, I’m using the ‘OCI API Signature 1.0' that is a combination of a concatenation of OCIDs for tenancy/user/API key and the private key associated with the API Key.
For connections to Oracle REST Data Services (ORDS) there are details on authenticating on this blog.
Calling the Vision AI Service
The next step in the application process is to call the Vision AI service using the REST endpoint to analyze an image. As described earlier, I find it useful to define the body of a REST call as a variable, then pass that resulting variable to the actual REST call. In this case, I’m passing the object name to the visionAI_Body variable that is then, in turn referenced in the actual API call in the action chain.
I found that I needed to do some processing on the response from the Vision AI service in order to make it suitable to be used in the VB select (many) UI component. This mostly involved transforming the response, in this form:
[{"name":"Blue","confidence":0.99102336},{"name":"Toy","confidence":0.9829609},{"name":"Wheel","confidence":0.94196075},{"name":"Car","confidence":0.9119572},{"name":"Vehicle","confidence":0.90317947},{"name":"Land vehicle","confidence":0.8977431},{"name":"Plastic","confidence":0.86447364},{"name":"Iron (Material)","confidence":0.7766388},{"name":"Human head","confidence":0.73616046},{"name":"Circle","confidence":0.6690402}]to a series of rows of label names (without quotes) that could be defined as a Service Data Provider (SDP) for use in the select (many). The first step of this process was to extract the label ‘names’ from the returned JSON. This was achieved by using a ‘utility’ JavaScript function to extract the ‘name’ and populate a new Array.
loopFunction(arrayOfNames) {
let arrayLength = arrayOfNames.length
let detectedNames = []
for (var i = 0; i < arrayLength; i++) {
console.log(arrayOfNames[i]);
detectedNames.push(arrayOfNames[i].name);
}
return detectedNames;
}I also used a JavaScript function to get a timestamp that was used to populate a database table with the timestamp when the table was updated along with the image filename and detected label names. I used a database sequence and on insert trigger to generate primary keys for that table. I also stored the response from Vision AI in the table but with a separate row for each name. If you were building an app for production use you probably wouldn’t need to do this, but as I was experimenting with this as a proof of concept, I was trying out different approaches.
In the case of the single row, the ORDS body looks like this:
{
"time_stamp" : $page.variables.current_timestamp,
"file_name" : 'Single-row' + $page.variables.capturedFileName,
"detected_names" : JSON.stringify($page.variables.ArrayOfNames)
}For the ORDS call to store a detected label in separate rows, I used a ‘For Each’ node in the action chain that loops over the array containing the ‘ArrayOfNames’. I also created a utility function, using .replaceAll(`"`,``) to remove quotes from my detected names as they needed to be in a specific form to be used by the embedded analytics filter JSON.
A separate ORDS call can be used to populate a VB Service Data Provider for the select (many).
Filtering Embedded Analytics
We now have a selectable list of values that corresponds to the labels detected by the OCI Vision AI service for any image that you capture on a device. As I’ve described in other blogs, Oracle released a VB web component to embed analytics. In this case, I’d like to filter the analytics based on the selected values from the select (many) that are stored in an array $variables.analyticsFilterArray
The filter JSON for the embedded analytics is as follows. Yours will be different in the case of the sColFormula reference and the name of the array used to store the selected values.
[[ [ { "sColFormula": "XSA('<owner>'.'<data-set>').\"Columns\".\"PRODUCT_DESC\"", "sColName": "PRODUCT_DESC", "sOperator": "in", "isNumericCol": false, "isDateCol": false, "bIsDoubleColumn": false, "aCodeValues": [], "aDisplayValues": $variables.analyticsFilterArray }] ]]Conclusion
Hopefully this blog has given you some ideas what’s possible when combining various Oracle analytic and AI services using a low-code developer tool such as Visual Builder.
Join the discussion on our public Slack!
