Mobile Design: server parsing and the case of base object.

Mobile means network!

Even without precise numbers, a reasonable assumption is that the vast majority of mobile applications are communicating with a server. That being written, and I have no doubt that this is certainly one of the dullest introduction ever written, let’s look at the core topic examined here: design of mobile backends. Certainly, there is not “one unique pattern” for such task, but having written some of them, there a few things I learned along this path.

  • The closer to the API, the less bugs there are. Said differently: the more layers of concepts are present, the less one understands what really happens in the complete stack.
  • A base object, that would eventually transform the support of new server entities into a few minutes exercice, can be pretty useful.
  • If data is to be stored (in memory or in a persistent way), avoid storing graph of objects. Think “database”, and refer objects through something that looks like an ID.
  • Parsing can be prone to errors: write only one piece of parsing code.
  • Always think multi: multi-server (production, development), multi-users (logout, re-login). Even if none of these features is used at first, considering them will significantly influence a design and its robustness.

Each of these points will now be discussed and illustrated. Of course, examples require a language; here this will be Objective-C. Due to the rise of Swift on the iOS platform, this may sound like a less than clever choice, but there are indeed a few reasons to this:

  • Some features of the language are pretty neat to demonstrate some of the concepts that will be mentioned in this article.
  • My personal Swift experience is limited. It would be a bad idea to do programming mistakes, exactly when explaining a particularly useful point. Personal credibility would decrease.
  • As Objective-C is on the path to “become a language of the past”, this article should, from the start, appear more like a “general purpose/platform independent” one.
  • Finally, I definitely hope that there will be people more skilled than me, who will provide implementations in Swift, Java, C#, Javascript and others.

As an explanation takes always more values with examples, a frame/sandbox has to be defined. So here are the rules of our API:

  • There are 3 objects in the examples: session, user (with friends), device. A user may have multiple devices, and a session is having one user and one device.
  • Server API is REST based. So register to the server will be a case of “POST /session”, confirming registration “PUT /session”, and so on…
  • The server is clever enough to understand the “me” keyword: “GET /user/me” will return the user of the session.
  • When there is a “connection” between object, it is in the path. All my devices can be fetched with “GET /user/me/devices”, which is different from all devices on the server that would come with “GET /devices”. In such case results are paginated.

Finally as a common practice of Objective-C code is to use prefix for code coming from a same source, MM is the chosen one here.

Stay away from too many abstractions!

Mobile languages have usually a strong object-oriented flavor. And this is useful to deal with objects when programming. For example, in a pane showing information about a user, every front-end developer would love to have a call like:

[MMMAGICBACKEND getUserWithName:@”Joe”]

This even if usage of get is not recommended in Objective-C… And in their secret wish, such method would check if the user data is present on the current device, fetch it if not present or if a newer version is present, and automatically detect which “Joe” is the correct one among the 4 presents. Inevitably, one day the user pane shows empty and the person implementing starts the discussion with “The MMMAGICBACKEND returns nil, it’s not my fault”!

A very simple way to avoid this is simply to remove those encapsulation and let the UI parts decide when they HTTP queries should be made. Of course this imply to have a method like:

typedef void (^MMCompletionBlock)(MMHTTPTask *task);
[MMMAGICBACKEND launchRequest:@”/users/userIDFORJoe” withHTTPMethod:GET andBodyData:nil completionBlock:completionBlock]

The completion block is called after the request has run, successfully or not. A few things can be noted:

  • Even if anyone doing C development has learned to never write explicit values and use constants declaration, this is here totally made on purpose. Why? For the very simple reason that it exposes the real API of the server, its singularities, and possible incoherences. If we have server and client belonging to the same company, this can be a great source of feedback on the API.
  • The temptation could be here to make wrappers for all operations like GET/POST and others. Same as above: resist the temptation to grasp the full semantic of the API!
  • The implementation of such method should of course handle all details of network calls, use of special headers, use of https versus http, addition of server address…
  • Doing automatic retries in such method may be or not a good idea, depending on the context. The Pros: in case of small transient problem like network problem, this avoids to write explicit code. The Cons: in case of real situations where network is not present, this can trigger a lot of calls and definitely dry out a phone battery.
  • It is recommended that each call to this launch request method returns a way to access the native network operation (so it can possibly be cancelled)

On this last point a possible way to deal with the issue is to create a simple MMHTTPTask object with following structure:

@interface MMHTTPTask : NSObject
{
MMCompletionBlock _httpCompletionBlock;
NSURLSessionDataTask *_task;
NSString *_uuid;
NSUInteger _statusCode;
CFAbsoluteTime _begTime;
CFAbsoluteTime _endTime;
}

Such object would add an uuid to identify a network request, as well as add information to measure its performance also. The MMMAGICBACKEND can them maintain a set of all launched requests through their UUIDs, and decide for cancellation, explicit retries, or possibly avoid launch if a similar
request has already gone through.

The base object

From a general point of view, the data path we are dealing with, is of this form:

Data flow

Being able to deal with objects in a generic way, through a base class, can simplify this process, and opens the door to generic parsing. A possible base class could have following attributes:

@interface MMBaseObject : NSObject
{
NSString *_uuid;
NSMutableSet _presentFields;
NSUInteger _objectDataVersion;
NSUInteger _objectClassVersion;
}
  • The uuid is the most important attribute. It is usually provided by the server and looks like “something long and complex”. Sometimes it can be useful, in particular in debug cases, to prefix it with a letter that would give a hint about the class of the object. For example, a user UUID could look like u-lQ7tSlU4A9OmDjIefiOEMG5Dsy02Yt3q.
  • Do not use server database ID, but generate your own. Otherwise the day of database technology switch, the feeling of being in hell will be all around.
  • UUID are usually provided by the server, but there is no reason to not use them on the device. For example, in the case of an application accessing local data like Address Book or Calendar content, and uses it on the server, there is no reason to define separate objects for local and remote data. If that helps, there could be an additional attribute indicating the scope of existence of an object.
  • _presentFields is optional and holds a set of attributes present in an object. Effectively sometimes, multiple requests, with different access time, may be needed to access all attributes for an object. An example of a fields-compliant API is the Facebook Graph API (with fields argument in the query string). A typical use case is the one where a UI pane knows which fields are needed to display itself. At load time, it would check if a query should be made with a method like:
- (NSArray *)missingFieldsForDesiredFields:(NSSet *) desiredFields
{
NSMutableSet *tmpSet = [NSMutableSet setWithSet:desiredFields];
NSMutableSet *presentFields = [NSMutableSet
setWithSet:_presentFields];
[tmpSet minusSet:presentFields];
    if(0!= [tmpSet count]) {
[tmpSet addObject:@”uuid”];
}
return ([tmpSet allObjects]);
}
  • Both versions attributes are also “goodies”, but can be useful in the following cases. For _objectDataVersion, this could help to determine if an object received by the server is worth to be parsed, or if it can be ignored. The _objectClassVersion version can be useful to upgrade objects content after server modifications.

Usually, additional class methods have to be present, in particular for preparation of generic parsing. For example, when using prefixed UUID, the presence of a method returning the UUID prefix will be necessary. In Objective-C, dynamic registration allows a class to register itself as “base-class compliant”, without dangerous runtime interactions.

/* Base class class registration: to be called by subclasses */
+ (void) registerClassWithName:(NSString *)className
{
if(nil == _classMapping)
_classMapping = [[NSMutableDictionary alloc] init];
   @autoreleasepool {
NSString *prefix = [NSClassFromString(className) uuidPrefix]
_classMapping[prefix]= className;
}
}

And in the subclass:

+ (NSString *)uuidPrefix
{
return @”u”;
}

+ (void)load
{
[MMBaseObject registerClassWithName:
NSStringFromClass([self class])];
}

Thou shall store in a flat way!

An easy path to a memory base storage is to use a simple NSDictionary, that will map UUIDs to MMBaseObject (Key Value storage: <UUID>/<MMBaseObject>). A simple implementation would have methods like:

- (MMBaseObject *)objectWithUUID:(NSString *)objectUUID

- (BOOL)containsObjectWithUUID:(NSString *)objectUUID

- (void)addObject:(MMBaseObject *)object;

- (void)removeObjectWithUUID:(NSString *)objectUUID;

Among possible further developments for a storage are:

  • Add a disk persistence mechanism layer, possibly with a plugin for various storage implementations (SQL, NSCoder, CoreData, Realm…). It may also be possible to elaborate, and add a cache mechanism to load objects in a lazy way.
  • Add “Condition/Pre/Post” processing hooks to the storage. Those would allow to decide if an object should be stored, and if some action should be taken before and after doing so.
  • Expand the internal data structure of the storage beyond a simple dictionary, to allow to quickly find object of a certain type, or with given properties.

Dealing with relationships between objects, and in general with object graphs, can lead to a few problems. Let’s consider the case of a user having multiple devices. An, apparently natural, way to implement this is to consider a devices array attribute on a user object.

Now a possible problem rises at the time of their creation : one wants to ensure only one unique object exist for a device, and never fall into the case where a device would be instantiated twice (one directly for direct access, and one inside a user object. It even can get worse if persistent storage is here. Objects should also be stored only once, and be re-instantiated in a unique way.

Not falling into this kind of bugs is easy: do not refer object from others, only UUIDs. That is, having :

NSString *deviceUUID = user.deviceUUIDs[0]
MMDevice *device =(MMDevice *) [STORAGE objectWithUUID:deviceUUID];

instead of

MMDevice *device =(MMDevice *) user.devices[0];

Sure, some may complain about the fact that it is a little more verbose…

Instantiation

Coming back to the general data flow, three types of data can usually be received from a server (we consider a JSON based exchange) : plain objects, array of objects, and slices (case of pagination).

/* Plain object */
{
“uuid”:”u-lQ7tSlU4A9OmDjIefiOEMG5Dsy02Yt3q”,
“name”:”Joe”
}
/* Array of objects */
[
{
"uuid":"u-lQ7tSlU4A9OmDjIefiOEMG5Dsy02Yt3q",
"name":"Joe"
},
{
"uuid":"u-7huktSla8hus9jIe7ju9suow08wjow0k",
"name":"Bob"
}
]
/* Slice: paginated data */
{
"offset":12,
"count":2,
"total":47,
"data":
[
{
"uuid":"u-lQ7tSlU4A9OmDjIefiOEMG5Dsy02Yt3q",
"name":"Joe"
},
{
"uuid":"u-7huktSla8hus9jIe7ju9suow08wjow0k",
"name":"Bob"
}
]
}

In such case it is pretty easy to define a generic parsing method/function that will recursively create objects. A typical implementation would start from an implementation like the one below:

void _ParseAPIObjectWithExecutionBlock(id jsonContent, MMStorage *storage) 
{
if ([jsonContent isKindOfClass:[NSArray class]]) {
// Array of objects: call recursively
NSArray *tmpArray = (NSArray *)jsonContent;
[tmpArray enumerateObjectsUsingBlock:
^(id obj, NSUInteger idx, BOOL *stop) {
_ParseAPIObjectWithExecutionBlock(obj, storage)
}];
} else if ([jsonContent isKindOfClass:[NSDictionary class]]) {
if (nil != jsonContent[@”uuid”]) {
// Unique object
id tmpObject = [storage objectForKey:jsonContent[@”uuid”]];
       if (tmpObject) {
// this is an update
[tmpObject updateWithJSONContent:jsonContent)]
} else {
//this is a creation. We use the fact that each subclass
//has registered itself towards the base class
NSString *objectClassString = [MMBaseObject
classNameForUUIDWithPrefix:jsonContent[@”uuid”]];
tmpObject = [[NSClassFromString(objectClassString) alloc]
initWithJSONContent:jsonContent];
[storage addObject:tmpObject]
}
} else if( (nil != jsonContent[@”offset”]) &&
(nil != jsonContent[@”data”]) &&
(nil != jsonContent[@”total”]) &&
(nil != jsonContent[@”count”]) ) {
// this is a slice of paginated data
_ParseAPIObjectWithExecutionBlock
(jsonContent[@”data”], storage)
}
}
}

At this point the only remaining task is to implement the initWithJSONContent and updateWithJSONContent for each subclass. Practically, only the last one is necessary as the initWithJSONContent can be part of the base class and simply call it.

External frameworks providing utilities to do the mapping between a dictionary and real objects can be used for the parsing. However, and due to their generic nature, their weak point is always the same: dealing with partial update of objects (in addition, in Objective-C, they usually rely on low level runtime methods, which is always something to consider with caution).

One problem remains: how to get the objects coming from a request in a simple way. The answer lies in defining an object, the MMCollection, which as its core is simply holding an array of UUIDs (of external objects). In addition to its own UUID, it will also hold another identifier, the secondary display identifier, whose purpose is simply to link it to another entity. The MMCollection is usually a pure local object.

If one considers the MMHTTPTask defined previously in this article, one can associate a collection to each MMHTTPTask, whose secondary identifier would be task UUID. Implementing a special storage method to quickly find a collection based on its secondary display identifier, as well as a dedicated collection initializer would make all manipulations relatively seamless.

The parse function would be modified to have another argument, the original MMHTTPTask, and be adapted to dynamically create a collection linked to the task, and add it to the storage. Each time an object is encountered in the received stream, its UUID is added to the collection.

/* On MMCollection */
@interface MMCollection : MMBaseObject
{
NSString *_secondaryDisplayIdentifier;
NSArray *_itemsUUID;
}
- (id) initWithDisplayIdentifier:(NSString *)displayIdentifier;
- (void)addItemUUID:(NSString *)uuid;
@end
/* On MMStorage */
@interface MMStorage(MMCollectionSupport):
- (MMCollection) collectionWithDisplayIdentifier:(NSString *)
displayIdentifier
@end

Finally, such system can also handle the case of endpoint returning paginated data. Effectively, if one stays with the system of creating a collection linked to a network call, any call to a paginated endpoint
will create its own collection. Practically, it could be useful to get a collection linked to the endpoint itself, so one can get the complete list of paginated UUIDs.

In such case, one would choose a secondary display identifier equal to the non fielded endpoint of the query. This one is defined as to “the endpoint with some query parameters removed”, and in particular those related to offset and count. For example:

/users/friends/?offset=10&count=2&field=firstname,age&gender=male

would have a non fielded endpoint equal to:

/users/friends/?gender=male

The parsing needs again to be adjusted to handle the creation of such collection for paginated data. Such handling is usually an addition to the creation of “per-task” collection. Therefore, 3 calls to a paginated endpoint
would result in the creation of 4 collections:

  • 3 with secondary display identifier equal to the task UUIDs,
  • 1 with its secondary display identifier equal to the concerned non fielded endpoint.

The multi faceted engine

The last point to consider in this article is the one of “handling the multi case ” in a backend engine. By multi, this could be multi users or multi development environment (e.g. development versus production).

Usually such problem are not considered initially. They are usually considered first in testing scenarios with cases like “I need to reinit my account to switch server”, which usually leads to seeing data from all those environments in the application.

Being able for example to have individual storage for each tuple {user/environment}, as well as avoiding too many usages of the singleton pattern, will definitely prepare for the handling of such cases. It can even be useful to be able to send data to 2 environment at the same time, to test a new server development.

Conclusion

At this point, fundamental elements of a mobile backend have been considered. However the link backend/frontend has not been mentioned yet, and it plays a crucial role in keeping the two parts decoupled (which is necessary to maintain a quick product try and release cycle). Clearly a topic for a follow up post! I welcome also any implementation in other languages (and in particular Swift, Java).