How to use Google’s vision API with Java — Mendix How-to
Lately I’ve been re-watching some sessions from last years Mendix World 2020. So when I stumbled upon a session from my colleague Alistair Crawford about using Google’s Face detecting API, using their Java Libraries, I had to give it a try.
Although I learnt Java back in university, I rarely use it in my professional life and it’s definitely been far too long since I actually practiced using those skills, so I jumped to the github repo and got stuck in.
Before we start
We are going to need a few things before we get going, so take a moment to ensure you have everything before we start.
You will need :
Mendix Studio Pro 8 or above (I’m using 8.12 in this example).
The Eclipse IDE which you can download at www.Eclipse.org/downloads/.
A Google Cloud Project with the Google Vision API enabled on it.
A Service Account which is added to the Google Cloud Project, and has the role “Owner”.
You will need to ensure that billing is enabled and setup on Google Vision API, this won’t work with out it.
Make sure to watch Alistairs original session from Mendix World 2020.
Getting the Access key for the API
Normally in order to call an API, you need to authenticate yourself as an allowed user, which is done using an Access Key.
To do this:
-Go to the project on the Google Cloud Project’s overview. In the left hand side go to IAM & Admin and then to Service accounts.
-Open the Service Account added to your project. You should now see a screen like this :
-In the Tabs at the top of the page go to Keys
-Click Add Key
-Select JSON as the Key Type and click Create
Once you click create, you will be asked to save the JSON file. Make sure to do this, because you are only given this option here. You will not have the option to re-download it later, if it’s lost you will have to create a new key, so make 100% certain it’s saved somewhere secure.
Time to Code!
You can get the project files from Github. Once you’ve downloaded and opened your app In Studio Pro, you will notice there is a lot of pre-work done for us. There are some entities in the domain model : Photo, Credential and FaceLocation.
- Photo is the uploaded picture which we would like to process.
- Credentials is the file document which will hold the Authentication Key we just downloaded.
- Face Location is the object or list of objects returned from the Java action.
Getting the app to run
If you have already tried to run your app at this point you will probably have noticed theres something missing which causes a Java compilation error and prevents your app from starting up.
This is because it’s missing some dependencies, which we will add now. Mendix can support multiple dependency management tools for Java such as Maven or Ivy. This project uses Ivy, so we need to get Ivy to automatically update or add any missing dependencies.
- Open up a terminal and navigate to your project’s directory.
cd C:\\YourProjectLocation
- Run the import command for Ivy using.
runivy.cmd
- Wait for it to complete.
If you haven’t added the path to your Java JDK as a system environment variable this will fail. You can read more about how to set this here.
Uploading the Google Vision API Key
Once your app is compiling and runs successfully, you will have to upload the API key we downloaded from the google cloud console.
Run your app and navigate to it in your browser. Once you are on the home page of the app, you will see there are 2 tabs:
- Face Detect
- Credentials
Click on the Credentials tab and click on New.
In the pop up window, click browse and in the file explorer which appears select the JSON file we downloaded from the Google Cloud Console. To finish off click save.
Testing it out
You can now test out the app completely for the first time. To do so, go back to the face detect tab on the apps home page, and click the New button.
In the Pop up window which appears, click browse on the Upload Image field, and select any photo which contains a persons face. For testing purposes I am using an image of some computer generated faces which where created by an AI online. Click Save to close the window.
You should now see the photo you uploaded appear in the form. You may now click on Blur Faces to test out the app.
You should see the which will look something like this :
A closer look at the code
So now that we’ve gotten it working, let’s take a look at what this code is actually doing. If we inspect the microflow ACT_ProcessImage which is connected to the Blur Faces button on the home page. Inside we can see a fairly simple flow, which retrieves the credential file we uploaded from the Database and then passes that along with the Image we are trying to process into 2 separate Java actions.
To find out what these actions are doing we will have to open up the project in Eclipse. To do this, while still In Studio Pro, press F6 to deploy the project for Eclipse. Then open up eclipse and import the project by clicking on File in the top navigation, and then selecting Open projects from file system.
In the new window which has opened select Directory under Import Source and select the Root folder of your project, before clicking on Finish to complete the import
Once your project is open in the Package explorer on the left hand side, look for Javasource, and then for the subfolder myfirstmodule.actions. Inside there should be the 2 java actions which make this possible.
- JA_DetectFaces
- JA_BlurBoxes
The full code contained JA_DetectFaces is as shown below. Look for the bold inline comments for details.
// This file was generated by Mendix Studio Pro.
//
// WARNING: Only the following code will be retained when actions are regenerated:
// - the import list
// - the code between BEGIN USER CODE and END USER CODE
// - the code between BEGIN EXTRA CODE and END EXTRA CODE
// Other code you write will be lost the next time you deploy the project.
// Special characters, e.g., é, ö, à , etc. are supported in comments.package myfirstmodule.actions;import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
import com.google.api.gax.core.CredentialsProvider;
import com.google.auth.Credentials;
import com.google.auth.oauth2.GoogleCredentials;
import com.google.cloud.vision.v1.AnnotateImageRequest;
import com.google.cloud.vision.v1.AnnotateImageResponse;
import com.google.cloud.vision.v1.BatchAnnotateImagesResponse;
import com.google.cloud.vision.v1.BoundingPoly;
import com.google.cloud.vision.v1.FaceAnnotation;
import com.google.cloud.vision.v1.Feature;
import com.google.cloud.vision.v1.Image;
import com.google.cloud.vision.v1.ImageAnnotatorClient;
import com.google.cloud.vision.v1.ImageAnnotatorSettings;
import com.google.common.collect.Lists;
import com.google.protobuf.ByteString;
import com.mendix.core.Core;
import com.mendix.systemwideinterfaces.core.IContext;
import com.mendix.webui.CustomJavaAction;
import com.mendix.systemwideinterfaces.core.IMendixObject;
import myfirstmodule.proxies.FaceLocation;public class JA_DetectFaces extends CustomJavaAction<java.util.List<IMendixObject>>
{
private IMendixObject __SourceImage;
private myfirstmodule.proxies.Photo SourceImage;
private IMendixObject __CredentialsFile;
private myfirstmodule.proxies.Credential CredentialsFile;public JA_DetectFaces(IContext context, IMendixObject SourceImage, IMendixObject CredentialsFile)
{
super(context);
this.__SourceImage = SourceImage;
this.__CredentialsFile = CredentialsFile;
}@java.lang.Override
public java.util.List<IMendixObject> executeAction() throws Exception
{
this.SourceImage = __SourceImage == null ? null : myfirstmodule.proxies.Photo.initialize(getContext(), __SourceImage);this.CredentialsFile = __CredentialsFile == null ? null : myfirstmodule.proxies.Credential.initialize(getContext(), __CredentialsFile);// BEGIN USER CODE// Setup credentials for Google Vision IContext ctx = getContext();
InputStream credentialsFIle = Core.getFileDocumentContent(ctx, __CredentialsFile);
CredentialProvider provider = new CredentialProvider(credentialsFIle);
ImageAnnotatorSettings imageAnnotatorSettings =
ImageAnnotatorSettings.newBuilder()
.setCredentialsProvider(provider)
.build();
// Convert photo into format useable for Google Vision
InputStream fis = Core.getFileDocumentContent(ctx, __SourceImage);
ByteString imgBytes = ByteString.readFrom(fis);
Image img = Image.newBuilder().setContent(imgBytes).build();
// Set the feature type to Facial Detection Feature feat = Feature.newBuilder().setType(Feature.Type.FACE_DETECTION).build();
// Prepare the request for the API AnnotateImageRequest request =
AnnotateImageRequest.newBuilder().addFeatures(feat).setImage(img).build();
List<AnnotateImageRequest> requests = new ArrayList<>();
requests.add(request);// Handle responses ArrayList<IMendixObject> faceLocations = new ArrayList<IMendixObject>();
try (ImageAnnotatorClient client = ImageAnnotatorClient.create(imageAnnotatorSettings)) {
BatchAnnotateImagesResponse response = client.batchAnnotateImages(requests);
List<AnnotateImageResponse> responses = response.getResponsesList();
for (AnnotateImageResponse res : responses) {
if (res.hasError()) {
return faceLocations;
}
// For each face gather information needed for (FaceAnnotation annotation : res.getFaceAnnotationsList()) {
BoundingPoly boundingBox = annotation.getBoundingPoly();
List<com.google.cloud.vision.v1.Vertex> boxVertices = boundingBox.getVerticesList();
// Gather X,Y, Width & Height int X = Math.round(boxVertices.get(0).getX());
int Y = Math.round(boxVertices.get(0).getY());
int Width = Math.round(boxVertices.get(2).getX()) - X;
int Height = Math.round(boxVertices.get(2).getY()) - Y;
// Set FaceLocation Attributes FaceLocation faceLocation = new FaceLocation(ctx);
faceLocation.setX(X);
faceLocation.setY(Y);
faceLocation.setWidth(Width);
faceLocation.setHeight(Height);
// Add new location to List faceLocations.add(faceLocation.getMendixObject());
}
}
}
return faceLocations;
// END USER CODE
}/**
* Returns a string representation of this action
*/
@java.lang.Override
public java.lang.String toString()
{
return "JA_DetectFaces";
}// BEGIN EXTRA CODE
public class CredentialProvider implements CredentialsProvider {private InputStream authStream;
private boolean usePath=false;
private String path;public CredentialProvider(InputStream inputStream) {
this.authStream = inputStream;
this.usePath=false;
}public CredentialProvider(String inputPath) {
this.path = inputPath;
this.usePath=true;
}@Override
public Credentials getCredentials() throws IOException {
// TODO Auto-generated method stub
if (this.usePath)
{
return GoogleCredentials.fromStream(new FileInputStream(path))
.createScoped(Lists.newArrayList("https://www.googleapis.com/auth/cloud-platform"));
}
else
{
return GoogleCredentials.fromStream(authStream)
.createScoped(Lists.newArrayList("https://www.googleapis.com/auth/cloud-platform"));
}
}}
// END EXTRA CODE
}
After this action completes the app then calls the second action JA_BlurFaces, you can see the full code for this below, again look for the bolded comments for details.
// This file was generated by Mendix Studio Pro.
//
// WARNING: Only the following code will be retained when actions are regenerated:
// - the import list
// - the code between BEGIN USER CODE and END USER CODE
// - the code between BEGIN EXTRA CODE and END EXTRA CODE
// Other code you write will be lost the next time you deploy the project.
// Special characters, e.g., é, ö, à , etc. are supported in comments.package myfirstmodule.actions;import java.awt.Graphics;
import java.awt.image.BufferedImage;
import java.awt.image.BufferedImageOp;
import java.awt.image.ColorModel;
import java.awt.image.ConvolveOp;
import java.awt.image.Kernel;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.InputStream;
import javax.imageio.ImageIO;
import com.mendix.core.Core;
import com.mendix.systemwideinterfaces.core.IContext;
import com.mendix.webui.CustomJavaAction;
import com.mendix.systemwideinterfaces.core.IMendixObject;
import myfirstmodule.proxies.FaceLocation;public class JA_BlurBoxes extends CustomJavaAction<java.lang.Boolean>
{
private IMendixObject __SoureImage;
private myfirstmodule.proxies.Photo SoureImage;
private IMendixObject __DestinationImage;
private myfirstmodule.proxies.Photo DestinationImage;
private java.util.List<IMendixObject> __FaceLocations;
private java.util.List<myfirstmodule.proxies.FaceLocation> FaceLocations;public JA_BlurBoxes(IContext context, IMendixObject SoureImage, IMendixObject DestinationImage, java.util.List<IMendixObject> FaceLocations)
{
super(context);
this.__SoureImage = SoureImage;
this.__DestinationImage = DestinationImage;
this.__FaceLocations = FaceLocations;
}@java.lang.Override
public java.lang.Boolean executeAction() throws Exception
{
this.SoureImage = __SoureImage == null ? null : myfirstmodule.proxies.Photo.initialize(getContext(), __SoureImage);this.DestinationImage = __DestinationImage == null ? null : myfirstmodule.proxies.Photo.initialize(getContext(), __DestinationImage);this.FaceLocations = new java.util.ArrayList<myfirstmodule.proxies.FaceLocation>();
if (__FaceLocations != null)
for (IMendixObject __FaceLocationsElement : __FaceLocations)
this.FaceLocations.add(myfirstmodule.proxies.FaceLocation.initialize(getContext(), __FaceLocationsElement));// BEGIN USER CODE
IContext ctx = getContext();
try { // Get input stream from Mendix object and read into Buffered image InputStream fis = Core.getFileDocumentContent(ctx, __SoureImage);
BufferedImage readOnlyImage = ImageIO.read(fis); // New Image, prep for drawing... BufferedImage image =
new BufferedImage(readOnlyImage.getWidth(),readOnlyImage.getHeight(), BufferedImage.TYPE_INT_RGB);Graphics gfx = image.getGraphics();
gfx.drawImage(readOnlyImage, 0, 0, null);
gfx.dispose();
// A 3x3 kernel that blurs an image float[] matrix = new float[2500];
for (int i = 0; i < 2500; i++)
matrix[i] = 1.0f/2500.0f;
Kernel kernel = new Kernel(50, 50, matrix);
BufferedImageOp op = new ConvolveOp(kernel, ConvolveOp.EDGE_NO_OP,null);// Iterate and blur sub images based on each location
for (FaceLocation location : FaceLocations) {
BufferedImage dest = image.getSubimage(
(location.getX()),
location.getY(),
location.getWidth(),
location.getHeight());
ColorModel cm = dest.getColorModel();
BufferedImage src = new BufferedImage(cm,
dest.copyData(dest.getRaster().createCompatibleWritableRaster()),
cm.isAlphaPremultiplied(),
null)
.getSubimage(0, 0, dest.getWidth(), dest.getHeight());
op.filter(src, dest);
}
// Write result to Byte array to be stored back in mx object ByteArrayOutputStream output = new ByteArrayOutputStream();
ImageIO.write(image, "jpg", output);//Store in InputStream and Write to Mendix object InputStream is = new ByteArrayInputStream(output.toByteArray());
Core.storeImageDocumentContent(ctx, DestinationImage.getMendixObject(), is, image.getWidth(), image.getHeight());// Cleanup
output.close();
is.close();return true;
} catch (Exception e) {
throw e;
}
// END USER CODE
}/**
* Returns a string representation of this action
*/
@java.lang.Override
public java.lang.String toString()
{
return "JA_BlurBoxes";
}// BEGIN EXTRA CODE
// END EXTRA CODE
}
A summary of the code:
- The image and credential files are passed into JA_Detect faces. The action then sends both the credentials and the image to the Google Vision API.
- If successful the API returns an array of locations where faces were detected.
- The results are then mapped into a List of FaceLocation Objects and returned by the action.
- The Image along with the list of face locations is then passed into JA_BlurFaces.
- JA_BlurFaces then loops through the list of faces using the locations provided by google and with the use of some image drawing libraries, blurs the faces.
- The result is the committed to the database, overriding the original Image which was uploaded, and this refreshes the image in the browser, which should now display the final result.
Read more
From the Publisher -
If you enjoyed this article you can find more like it at our Medium page or at our own Community blog site.
For the makers looking to get started, you can sign up for a free account, and get instant access to learning with our Academy.
Interested in getting more involved with our community? You can join us in our Slack community channel or for those who want to be more involved, look into joining one of our Meet ups.
