Episode 16 : Using the Datastore API

December 2017: Please note that this post has not been updated for a while and there could be differences in terms of commands, screenshots and problems in running the code.

Welcome to Episode 16. In this episode we shall cover basic usage of the Datastore API in Google App Engine. The Datastore API is used to persist and retrieve data. Google App Engine uses the BigTable database as its underlying datastore and provides abstraction for using it via the Datastore API. There are currently two options available to developers i.e. via the Java Data Objects (JDO) and Java Persistence Architecture (JPA) APIs.

In this episode, we shall cover the following items:

• Persist a simple record to the datastore using JDO. The intention is to limit it to a single record in this episode and not address relationships and complex structures. That could be the basis of future episodes.
• Retrieve the persisted records by firing some queries. In the process, we shall see how to create parameterized queries and execute them.
• Discuss some nuances about indexes and what steps you need to do to make sure that the same application that you use locally will work fine when deployed to the Google App Engine cloud.

The underlying theme will not be to present a comprehensive tutorial about the Datastore API. There are some excellent references available for that. The official documentation and well as the GAE Persistence Blog. The focus will be on getting up and running with the Datastore API ASAP and seeing it work when deployed to the cloud.

What we shall build

In this episode, we shall build the following:

1. Create a simple Object that we shall persist to the underlying datastore. The Object will be a Health Report and will have about 4–5 attributes that we would like to save.
2. Write the Save method that takes an instance of the above Health Report Record and persists it using the JDO API.
3. Write a Search method that will query for several Health Reports using several filter parameters.
4. Look at the datastore_indexes.xml file that is required when you deploy the application to the cloud.

Please note that the focus will be on the server side and not on building a pretty GUI. All server side actions will be invoked via a REST like request (HTTP GET) — so that we can test the functionality in the browser itself.

Developing our Application

The first thing to do is to create a New Google Web Application Project. Follow these steps:

1. Either click on File –> New –> Other or press Ctrl-N to create a new project. Select Google and then Web Application project. Alternately you could also click on the New Web Application Project Toolbar icon as part of the Google Eclipse plugin.
2. In the New Web Application Project dialog, deselect the Use Google Web Toolkit and give a name to your project. I have named mine GAEJExperiments. I suggest you go with the same name so that things are consistent with the rest of the article, but I leave that to you. In case you are following the series, you could simply use the same project and skip all these steps altogether. You can go straight to the Servlet Development section.
3. Click on Finish.

This will generate the project and also create a sample Hello World Servlet for you. But we will be writing our own Servlet.

Few things to note first:

Quite a few things are enabled for you by default as far as the database support is concerned. They are as follows:

a. Several JAR files are added to the CLASSPATH by default. Take a look and you will see several JARs *jpa*.jar, *datanucleus*.jar, etc.
b. In the src/META-INF folder, you will find a jdoconfig.xml file. There is a default Persistence Manager Factory class in that that we shall be using in the rest of the article. For the purposes of this article we do not need to do anything to this file.
c. GAEJ uses the DataNucleus library to abstract the BigTable store. The DataNucleaus library provides the JDO and JPA interfaces so that you do not have to deal with the underlying low level API. You will also find a logging.properties file present in war/WEB-INF folder. You will find several log levels mentioned for the DataNucleus classes. You can tweak them to lower levels like DEBUG/INFO to see more debug level statements of what happens when you are using these APIs. I have found it very helpful to set the debug levels to DEBUG/INFO especially when facing a problem.

PMF.java

The first class that we shall write is a simple utility class that shall get us the underlying Persistence Manager factory instance. This class is important since all other methods like saving a record, querying records, etc will work on the instance of the PersistenceManagerFactory.

The code is shown below and wherever we need an instance of the class, we shall simply invoke the get() method below:

package com.gaejexperiments.db;
import javax.jdo.JDOHelper;
import javax.jdo.PersistenceManagerFactory;
public final class PMF {
private static final PersistenceManagerFactory pmfInstance =
JDOHelper.getPersistenceManagerFactory("transactions-optional");
private PMF() {}
public static PersistenceManagerFactory get() {
return pmfInstance;
}
}

HealthReport.java

Next is the Health Report. As mentioned in the beginning of the article, we shall be saving a Health Report record. The Health Report will have 4 attributes and they are explained below:

• Key : This is a unique key that is used to persist/identify the record in the datastore. We shall leave its implementation/generation to the Google App Engine implementation.
• PinCode : This is similar to a ZipCode. This is a string that shall contain the value of the zipcode (Area) where the Health Incident has occured.
• HealthIncident: This is a string value that contains the health incident name. For e.g. Flu, Cough, Cold, etc. In this simple application — we shall be concerned only with 3 health incidents i.e. Flu, Cough and Cold.
• Status : This is a string value that specifies if the record is ACTIVE or INACTIVE. Only records with ACTIVE status shall be used in determining any statistics / data reports. We shall set this value to ACTIVE at the time of saving the record.
• ReportDateTime : This is a Date field that shall contain the date/time that the record was created.

Shown below is the listing for the HealthReport.java class. In addition to the above attributes and getter/setter methods for them, note the following additions to make sure that your class can be persisted using JDO.

1. We need to have a constructor that contains all the fields except for the Key field.
2. All fields that need to be persisted are annotated with the @Persistent annotation.
3. The class is declared as being persistable via the @PersistenceCapable annotation and we are leaving the identity to the Application.
4. The Primary Key field i.e. Key is declared via the @PrimaryKey annotation and we are using an available Generator for the ID instead of rolling our own.

package com.gaejexperiments.db;
import java.util.Date;
import com.google.appengine.api.datastore.Key;
import javax.jdo.annotations.IdGeneratorStrategy;
import javax.jdo.annotations.IdentityType;
import javax.jdo.annotations.PersistenceCapable;
import javax.jdo.annotations.Persistent;
import javax.jdo.annotations.PrimaryKey;
@PersistenceCapable(identityType = IdentityType.APPLICATION)
public class HealthReport {
@PrimaryKey
@Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
private Key key;
private String pinCode;
@Persistent
private String healthIncident;
@Persistent
private String status;
@Persistent
private Date reportDateTime;
public HealthReport(String pinCode, String healthIncident,String status, Date reportDateTime) {
super();
this.pinCode = pinCode;
this.healthIncident = healthIncident;
this.status = status;
this.reportDateTime = reportDateTime;
}
public Key getKey() {
return key;
}
public void setKey(Key key) {
this.key = key;
}
public String getPinCode() {
return pinCode;
}
public void setPinCode(String pinCode) {
this.pinCode = pinCode;
}
public String getHealthIncident() {
return healthIncident;
}
public void setHealthIncident(String healthIncident) {
this.healthIncident = healthIncident;
}
public String getStatus() {
return status;
}
public void setStatus(String status) {
this.status = status;
}
public Date getReportDateTime() {
return reportDateTime;
}
public void setReportDateTime(Date reportDateTime) {
this.reportDateTime = reportDateTime;
}
}

PostHealthIncidentServlet.java

We shall now look at how to persist the above Health Record. Since we are not going to build a UI for it, we shall simply invoke a servlet (HTTP GET) with the required parameters. It would almost be like a FORM submitting these values to the Servlet. Before we write this Servlet code, let us look at how we will invoke it. Given below is a screenshot of the browser where I punch in the URL : http://localhost:8888/posthealthincident?healthincident=Flu&pincode=400101

As you can see, I am running the application on my local development server and invoke the servlet (which we shall see in a while) providing two key parameters healthincident and pincode. These two parameters are two key fields of the HealthReport class that we saw above. The other fields like ReportDateTime and Status are determined automatically by the application. Similarly the Key value of the record in the underlying datastore will be generated by App Engine infrastructure itself.

Let us now look at the PostHealthIncidentServlet.java code shown below:

package com.gaejexperiments.db;
import java.io.IOException;
import java.util.Date;
import java.util.logging.Logger;
import javax.servlet.ServletException;
import javax.servlet.http.*;
@SuppressWarnings("serial")
public class PostHealthIncidentServlet extends HttpServlet {
public static final Logger _logger = Logger.getLogger(PostHealthIncidentServlet.class.getName());
@Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
doPost(req, resp);
}
public void doPost(HttpServletRequest req, HttpServletResponse resp)
throws IOException {
resp.setContentType("text/plain");
String strResponse = "";
String strHealthIncident = "";
String strPinCode = "";
try {
//DO ALL YOUR REQUIRED VALIDATIONS HERE AND THROW EXCEPTION IF NEEDED
strHealthIncident = (String)req.getParameter("healthincident");
strPinCode = (String)req.getParameter("pincode");
String strRecordStatus = "ACTIVE";
Date dt = new Date();
HealthReport HR = new HealthReport(strPinCode,
strHealthIncident,
strRecordStatus,
dt);
DBUtils.saveHealthReport(HR);
strResponse = "Your Health Incident has been reported successfully.";
}
catch (Exception ex) {
_logger.severe("Error in saving Health Record : " + strHealthIncident + "," + strPinCode +  " : " + ex.getMessage());
strResponse = "Error in saving Health Record via web. Reason : " + ex.getMessage();
}
resp.getWriter().println(strResponse);
}
}

The main points of the code are :

1) We extract out the HealthIncident and PinCode request parameters. We do not do any particular validations but you could do all of that depending on your application requirement.
2. We generate the two other field values i.e. Date (ReportDate) and Status (ACTIVE).
3. Finally, we create a new instance of the HealthReport class, providing the values in the constructor. And then call the DBUtils.saveHealthReport(...) method that persists the record to the underlying datastore.
4. We display back a successfull message if all is well, which is what was visible in the screenshot above.

Let us look at the DBUtils.java class now. Please note that we have simply separated out the code into this file but you can manage/partition your code in any which way you like.

DBUtils.java

The DBUtils.java source code is listed below. Focus first on the saveHealthReport() method which was invoked by our servlet earlier. The other method, we shall come to that later on in the article.

Key Points are :

1. The saveHealthReport() method first gets the instance of the PersistenceManager through the PMF.java class that we wrote earlier.
2. It simply invoke the makePersistent() method on it. The makePersistent() method will take as a parameter the object that you want to persist. In our case it is the HealthReport.java class instance that we have created in the servlet. This method will persist the record and in the process also assign it a unique key.
3. Finally, we need to close the PersistenceManager instance by invoking the close() method.

The entire code listing is shown below:

package com.gaejexperiments.db;
import java.util.Calendar;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.jdo.PersistenceManager;
import javax.jdo.Query;
public class DBUtils {
public static final Logger _logger = Logger.getLogger(DBUtils.class.getName());
//Currently we are hardcoding this list. But this could also be retrieved from
//database
public static String getHealthIncidentMasterList() throws Exception {
return "Flu,Cough,Cold";
}
/**
* This method persists a record to the database.
*/
public static void saveHealthReport(HealthReport healthReport)
throws Exception {
PersistenceManager pm = PMF.get().getPersistenceManager();
try {
pm.makePersistent(healthReport);
_logger.log(Level.INFO, "Health Report has been saved");
} catch (Exception ex) {
_logger.log(Level.SEVERE,
"Could not save the Health Report. Reason : "
+ ex.getMessage());
throw ex;
} finally {
pm.close();
}
}
/**
* This method gets the count all health incidents in an area (Pincode/Zipcode) for the current month
* @param healthIncident
* @param pinCode
* @return A Map containing the health incident name and the number of cases reported for it in the current month
*/
public static Map<String, Integer> getHealthIncidentCountForCurrentMonth(String healthIncident, String pinCode) {
Map<String, Integer> _healthReport = new HashMap<String, Integer>();
PersistenceManager pm = null;
//Get the current month and year
Calendar c = Calendar.getInstance();
int CurrentMonth = c.get(Calendar.MONTH);
int CurrentYear = c.get(Calendar.YEAR);
try {
//Determine if we need to generate data for only one health Incident or ALL
String[] healthIncidents = {};
if (healthIncident.equalsIgnoreCase("ALL")) {
String strHealthIncidents = getHealthIncidentMasterList();
healthIncidents = strHealthIncidents.split(",");
}
else {
healthIncidents =  new String[]{healthIncident};
}
pm = PMF.get().getPersistenceManager();
Query query = null;
//If Pincode (Zipcode) is ALL, we need to retrieve all the records irrespective of Pincode
if (pinCode.equalsIgnoreCase("ALL")) {
//Form the query
query = pm.newQuery(HealthReport.class, " healthIncident == paramHealthIncident && reportDateTime >= paramStartDate && reportDateTime < paramEndDate && status == paramStatus");
// declare parameters used above
query.declareParameters("String paramHealthIncident, java.util.Date paramStartDate, java.util.Date paramEndDate, String paramStatus");
}
else {
query = pm.newQuery(HealthReport.class, " healthIncident == paramHealthIncident && pinCode == paramPinCode && reportDateTime >= paramStartDate && reportDateTime <paramEndDate && status == paramStatus");
// declare params used above
query.declareParameters("String paramHealthIncident, String paramPinCode, java.util.Date paramStartDate, java.util.Date paramEndDate, String paramStatus");
}
//For each health incident (i.e. Cold Flu Cough), retrieve the records
for (int i = 0; i < healthIncidents.length; i++) {
int healthIncidentCount = 0;
//Set the From and To Dates i.e. 1st of the month and 1st day of next month
Calendar _cal1 = Calendar.getInstance();
_cal1.set(CurrentYear, CurrentMonth, 1);
Calendar _cal2 = Calendar.getInstance();
_cal2.set(CurrentYear,CurrentMonth+1,1);
List<HealthReport> codes = null;
if (pinCode.equalsIgnoreCase("ALL")) {
//Execute the query by passing in actual data for the filters
codes = (List<HealthReport>) query.executeWithArray(healthIncidents[i],_cal1.getTime(),_cal2.getTime(),"ACTIVE");
}
else {
codes = (List<HealthReport>) query.executeWithArray(healthIncidents[i], pinCode, _cal1.getTime(),_cal2.getTime(),"ACTIVE");
}
//Iterate through the results and increment the count
for (Iterator iterator = codes.iterator(); iterator.hasNext();) {
HealthReport _report = (HealthReport) iterator.next();
healthIncidentCount++;
}
//Put the record in the Map data structure
_healthReport.put(healthIncidents[i], new Integer(healthIncidentCount));
}
return _healthReport;
} catch (Exception ex) {
return null;
} finally {
pm.close();
}
}
}

Assuming that your application is running, you can view the data records that are being persisted. If you navigate to http://localhost:<YourPort>/_ah/admin in your browser, you will see a screenshot similar to the one shown below:

The screen above shows the Entity types that are currently having some data records. In our case it is the HealthReport entity of which we have saved one record so far. If you click on the List Entities button, you can see the records that are currently persisted for the HealthReport Entity Type. A sample screenshot from my system after saving my first record is shown below:

Go ahead and populate a few more records in the database for different HealthIncidents like Cough, Cold, Flu (only). This is needed so that we can get some more data when we cover how to query persistent data in the next section.

ReportsServlet.java

Before we look at this servlet, let us look at the output that this servlet produces, so that it becomes easier to follow the code later. This is assuming that you have added atleast 4-5 records using the /posthealthincident servlet that we covered above.

Shown below is the screenshot of the servlet output when I provide the following url:

http://localhost:8888/reports?type=HEALTHINCIDENTCOUNT_CURRENT_MONTH&healthincident=Flu&pincode=ALL
What we are asking for here is a report that gets all health incidents in the current month (type = HEALTHINCIDENTCOUNT_CURRENT_MONTH) for the healthincident = Flu and where pincode = ALL (irrespective of pincode)

Shown below is the screenshot of the servlet output when I provide the following url:

http://localhost:8888/reports?type=HEALTHINCIDENTCOUNT_CURRENT_MONTH&healthincident=ALL&pincode=ALL
What we are asking for here is a report that gets all health incidents in the current month (type = HEALTHINCIDENTCOUNT_CURRENT_MONTH) for the healthincident = ALL (irrespective of health incident which means all of them)  and where pincode = ALL (irrespective of pincode)

So what we have effectively done here is to query the set of Health Records that are present in our database using a variety of parameters (filters). In other words, if we take a SQL like aspect to it, we are saying something like this:

SELECT * FROM HEALTHREPORTS WHERE PINCODE = %1 AND HEALTHINCIDENT = %2 AND REPORTDATE >= %3 AND REPORTDATE < %4 AND STATUS = ACTIVE , etc.

The above SQL statement is just representative of the queries that we want to execute. So let us look at the code for the servlet first.

package com.gaejexperiments.db;
import java.io.IOException;
import java.util.Iterator;
import java.util.Map;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
@SuppressWarnings("serial")
public class ReportsServlet extends HttpServlet {
public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException {
resp.setContentType("text/xml");
String strResult = "";
String strData   = "";
try {
String type = (String)req.getParameter("type");
if (type == null) {
strResult =  "No Report Type specified.";
throw new Exception(strResult);
}
else if (type.equals("HEALTHINCIDENTCOUNT_CURRENT_MONTH")) {
String strHealthIncident = (String)req.getParameter("healthincident");
String strPinCode = (String)req.getParameter("pincode");
Map<String,Integer> _healthReports = DBUtils.getHealthIncidentCountForCurrentMonth(strHealthIncident,strPinCode);
if (_healthReports == null) {
}
else {
Iterator<String> it = _healthReports.keySet().iterator();
while (it.hasNext()) {
String healthIncident = (String)it.next();
int healthIncidentCount = 0;
Integer healthIncidentCountObject = _healthReports.get(healthIncident);
if (healthIncidentCountObject == null) {
healthIncidentCount = 0;
}
else {
healthIncidentCount = healthIncidentCountObject.intValue();
}
if (healthIncidentCount > 0)
strData += "<HealthIncident><name>" + healthIncident + "</name>" + "<count>" + healthIncidentCount + "</count></HealthIncident>";
}
}
strResult = "<Response><Status>success</Status><StatusDescription></StatusDescription><Result>" + strData + "</Result></Response>";
}
}
catch (Exception ex) {
strResult = "<Response><Status>fail</Status><StatusDescription>"+ "Error in executing operation : " + ex.getMessage() + "</StatusDescription></Response>";
}
resp.getWriter().println(strResult);
}
} </pre>
<pre>

The Servlet code is straightforward:

1. Currently it has only one type of report i.e. HEALTHINCIDENTCOUNT_CURRENT_MONTH
2. Next we extract out the Pincode and the HealthIncident request parameter values.
3. Then we invoke the DBUtils.getHealthIncidentCountForCurrentMonth method. This takes two parameters : pincode and healthincident that we have got from above step.
4. The method will return us a map where each record in the map will contain the key (String) containing the healthincident name and the value containing the count of incidents reported for that month. So something like [ {"Flu","20"} , {"Cough", "30"} , {"Cold","10"} ]
5. Finally we simply format that into a XML output so that it can be returned to the client. And this is the exact output that we see in the browser.

Analyzing the DBUtils.getHealthIncidentCountForCurrentMonth method

I reproduce here the method from the DBUtils.java class that was listed before:

/**
* This method gets the count all health incidents in an area (Pincode/Zipcode) for the current month
* @param healthIncident
* @param pinCode
* @return A Map containing the health incident name and the number of cases reported for it in the current month
*/
public static Map<String, Integer> getHealthIncidentCountForCurrentMonth(String healthIncident, String pinCode) {
Map<String, Integer> _healthReport = new HashMap<String, Integer>();
PersistenceManager pm = null;
//Get the current month and year
Calendar c = Calendar.getInstance();
int CurrentMonth = c.get(Calendar.MONTH);
int CurrentYear = c.get(Calendar.YEAR);
try {
//Determine if we need to generate data for only one health Incident or ALL
String[] healthIncidents = {};
if (healthIncident.equalsIgnoreCase("ALL")) {
String strHealthIncidents = getHealthIncidentMasterList();
healthIncidents = strHealthIncidents.split(",");
}
else {
healthIncidents =  new String[]{healthIncident};
}
pm = PMF.get().getPersistenceManager();
Query query = null;
//If Pincode (Zipcode) is ALL, we need to retrieve all the records irrespective of Pincode
if (pinCode.equalsIgnoreCase("ALL")) {
//Form the query
query = pm.newQuery(HealthReport.class, " healthIncident == paramHealthIncident && reportDateTime >= paramStartDate && reportDateTime < paramEndDate && status == paramStatus");
// declare parameters used above
query.declareParameters("String paramHealthIncident, java.util.Date paramStartDate, java.util.Date paramEndDate, String paramStatus");
}
else {
query = pm.newQuery(HealthReport.class, " healthIncident == paramHealthIncident && pinCode == paramPinCode && reportDateTime >= paramStartDate && reportDateTime <paramEndDate && status == paramStatus");
// declare params used above
query.declareParameters("String paramHealthIncident, String paramPinCode, java.util.Date paramStartDate, java.util.Date paramEndDate, String paramStatus");
}
//For each health incident (i.e. Cold Flu Cough), retrieve the records
for (int i = 0; i < healthIncidents.length; i++) {
int healthIncidentCount = 0;
//Set the From and To Dates i.e. 1st of the month and 1st day of next month
Calendar _cal1 = Calendar.getInstance();
_cal1.set(CurrentYear, CurrentMonth, 1);
Calendar _cal2 = Calendar.getInstance();
_cal2.set(CurrentYear,CurrentMonth+1,1);
List<HealthReport> codes = null;
if (pinCode.equalsIgnoreCase("ALL")) {
//Execute the query by passing in actual data for the filters
codes = (List<HealthReport>) query.executeWithArray(healthIncidents[i],_cal1.getTime(),_cal2.getTime(),"ACTIVE");
}
else {
codes = (List<HealthReport>) query.executeWithArray(healthIncidents[i], pinCode, _cal1.getTime(),_cal2.getTime(),"ACTIVE");
}
//Iterate through the results and increment the count
for (Iterator iterator = codes.iterator(); iterator.hasNext();) {
HealthReport _report = (HealthReport) iterator.next();
healthIncidentCount++;
}
//Put the record in the Map data structure
_healthReport.put(healthIncidents[i], new Integer(healthIncidentCount));
}
return _healthReport;
} catch (Exception ex) {
return null;
} finally {
pm.close();
}
}

I have attempted to provide comments so that you can follow the code but I will list down the important parts here:

1. We are going to deal with the following classes : PersistenceManager and Query from the javax.jdo package.
2. We get the PersistenceManager instance via the PMF.java class that we wrote earlier.
3. We are using the Query class here to first build the query. For e.g.
query = pm.newQuery(HealthReport.class, " healthIncident == paramHealthIncident && reportDateTime >= paramStartDate && reportDateTime < paramEndDate && status == paramStatus");

What this means is that we are creating a query instance where we wish to get all records for the HealthReport class. Additionally we are passing a criteria string. Notice that the lefthand side are the fields of the HealthReport class (healthIncident, reportDateTime, status) and the right hand side are parameters which will define and then pass the values for to execute the query.

4. We define the parameters next as shown below:
// declare parameters used above
query.declareParameters("String paramHealthIncident, java.util.Date paramStartDate, java.util.Date paramEndDate, String paramStatus");
5. Finally we use the query.executeWithArray(...) method which takes as parameter an array that contains all the values for the above parameters that you have declared.
6. The executeWithArray(...) will return a List<> of HealthReport class instances that you can then iterate through the populate your result. In our code, we simply compute the total number for each of the health incidents (Flu, Cough, Cold).

Servlet Configuration

To complete our Servlet development, we will also need to add the <servlet/> and <servlet-mapping/> entry to the web.xml file. This file is present in the WEB-INF folder of the project. The necessary fragment to be added to your web.xml file are shown below. Please note that you can use your own namespace and servlet class. Just modify it accordingly if you do so.

<servlet>
<servlet-name>PostHealthIncidentServlet</servlet-name>
<servlet-class>com.gaejexperiments.db.PostHealthIncidentServlet</servlet-class>
</servlet>
<servlet>
<servlet-name>ReportsServlet</servlet-name>
<servlet-class>com.gaejexperiments.db.ReportsServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>PostHealthIncidentServlet</servlet-name>
<url-pattern>/posthealthincident</url-pattern>
</servlet-mapping>
<servlet-mapping>
<servlet-name>ReportsServlet</servlet-name>
<url-pattern>/reports</url-pattern>
</servlet-mapping>

Running the application locally

I am assuming that you have already created a new Google Web Application Project and have created the above Servlets, web.xml , etc. So assuming that all is well, we will run our application, by right-clicking on the project and selecting Run As –> Web Application. Launch the browser on your local machine and try out the URLs that we have covered so far like :

1. Adding a Health Report record :

http://localhost:8888/posthealthincident?healthincident=Flu&pincode=400101

2. Reports

http://localhost:8888/reports?type=HEALTHINCIDENTCOUNT_CURRENT_MONTH&healthincident=Flu&pincode=ALL

http://localhost:8888/reports?type=HEALTHINCIDENTCOUNT_CURRENT_MONTH&healthincident=ALL&pincode=ALL

Please replace the port 8888 with your local port number.

datastore_indexes.xml

If you run your application locally, you will notice that everything is working fine. However, if you deploy this application the Google App Engine, you will not get any results where you query the reports. Why is that? It is because there are no indexes defined for your application.

If you visit your Developer Console at http://appengine.google.com and for your particular application, you will find no indexes are defined for the Datastore indexes link as shown below:

What are these indexes ? Indexes are generated for every possible query that you fire in your program. This is Google's way of retrieving your data results efficiently. However when you run in local mode, these indexes are generated for your automatically. If you look in war/WEB-INF directory, you will find a directory named appengine-generated. Inside this directory is a file that is constantly updated called datastore-indexes-auto.xml. The contents of this file for the reports that we have run so far is shown below:

<datastore-indexes>
<datastore-index kind="HealthReport" ancestor="false" source="auto">
<property name="healthIncident" direction="asc"/>
<property name="status" direction="asc"/>
<property name="reportDateTime" direction="asc"/>
</datastore-index>
<datastore-index kind="HealthReport" ancestor="false" source="auto">
<property name="healthIncident" direction="asc"/>
<property name="pinCode" direction="asc"/>
<property name="status" direction="asc"/>
<property name="reportDateTime" direction="asc"/>
</datastore-index>
</datastore-indexes>

As you can see, there are two indexes generated for the searches that we fired and each of them contains the parameters that we queried the Health Reports on.

To get your deployed application to function correctly, you will need to copy these indexes to a file named datastore_indexes.xml.

You will need to place this file in the war/WEB-INF folder of your application project.

Now deploy your project again. On successful deploy, revist the Developer Console --> Datastore indexes. You will find that the indexes have got created and are buing built as shown below:

Wait for a while. The building process goes through your current data and then on completion, you will see the screen as shown below (A refresh needed!):

Try out the queries now on your deployed application and they should work fine. You can also manual create the datastore_indexes.xml file but the local development server has this nice feature of auto generating out for you, so that you dont end up making mistakes. But remember to upload the updated datastore_indexes.xml as part of your deployment others the queries will silently fail.

Conclusion

We conclude this episode in which we covered how you can persist and query data using the JDO API. I hope that it has given you enough inputs to start incorporating persistence into your applications. It is by no means a simple exercise especially if you are coming in from a SQL world. Not everything SQL-like can be converted as is to a NoSQL world so refer to the several excellent sources available on the web in your persistence journey. I highly recommend the official documentation along with the GAE Persistence Blog mentioned at the beginning of the article.

Read more Episodes on App Engine Services

• Writing Google Talk Bots with App Engine
• Write your first Google Chrome Extension, powered by App Engine
• Scheduling Background Tasks in App Engine: Learn about the Cron Scheduler in App Engine
• Asynchronous Processing in App Engine: Learn about tasks / queues in App Engine
• Handling Email in App Engine Applications: Sending and Receiving incoming Email in your App Engine applications
• Blobstore Tutorial : Use the App Engine Blobstore service to store data in GB
• Datastore Tutorial : Basic database programming in App Engine
• Captcha Tutorial: How to integrate a Captcha in your App Engine application
• Location Services: Learn how to detect Location (Latitude & Longitude) in your App Engine Application. Part 1 and Part 2.
• Memcache: Learn how to use the cache to speed up your App Engine applications
• Networking: Learn how to make calls to external Web Services using URLFetch service in App Engine.
• Download the eBook: All the episodes packaged for you in a free eBook. Enjoy and thanks for downloading.