Are you kidding me, Google?

We use Google Cloud. A lot. We use the Kubernetes engine, Datastore, Cloud storage, Cloud SQL, Pub/Sub, Cloud DNS etc. For the most part we love it, but there is one thing that just drives me crazy: the SDKs. The Java ones anyway.

I think the Pub/Sub one is the one that has caused me the most pain (why are you still in Beta by the way?), but today’s rant is going to be mostly about the Datastore one.

A lot of our services use Datastore to, well, store data. In general it works just fine, but sometimes you encounter an error. Perhaps you get a timeout, or you try to insert something that already exists, or a transaction fails because of concurrent modifications of an entity.

Whatever the reason, you will get a DatastoreException. It’s hard to find documentation on what to expect, so I figured I would just create a little Gradle project and try to trigger errors, just to see what I got. So off I went.

I created a minimal project, where the only dependency apart from Kotlin was the Datastore SDK.

dependencies {
    compile "org.jetbrains.kotlin:kotlin-stdlib:1.1.2",
            "com.google.cloud:google-cloud-datastore:1.12.0"
}

Then I wrote this little piece of code, just to see that I could insert an entity:

@JvmStatic
fun main(args: Array<String>) {
    // Nothing special, just connecting to the Datastore emulator
    val datastore = setupDatastore()
    val key = datastore.newKeyFactory()
            .setKind("test")
            .newKey("someId")
    val entity = Entity.newBuilder(key)
            .set("message", "Hello world")
            .build()
    datastore.add(entity)
}

I ran it.

Ok, so even though the only dependency I have is the Datastore SDK I’m getting conflicts between some transitive dependencies. That’s a first for me I think, I usually need to include at least two dependencies for this to happen..

Ok, so I wanted to have a look at those transitive dependencies.

./gradlew :dependencies — configuration compile | grep protobuf-java
| + — — com.google.protobuf:protobuf-java-util:3.4.0
| | + — — com.google.protobuf:protobuf-java:3.4.0
| | \ — — com.google.protobuf:protobuf-java:3.3.0 -> 3.4.0
| + — — com.google.protobuf:protobuf-java:3.3.0 -> 3.4.0
| + — — com.google.protobuf:protobuf-java:3.0.0 -> 3.4.0
| | | + — — com.google.protobuf:protobuf-java:3.0.0 -> 3.4.0
| | | + — — com.google.protobuf:protobuf-java-util:3.0.0 -> 3.4.0 (*)
| | | + — — com.google.protobuf.nano:protobuf-javanano:3.0.0-alpha-5
| \ — — com.google.protobuf:protobuf-java:3.0.0 -> 3.4.0
| \ — — com.google.protobuf:protobuf-java:2.4.1 -> 3.4.0

You guys really, really like you protobufs.

Ok, so I fixed this by adding an explicit dependency on protobuf-java-util:3.4.0.

dependencies {
    compile "org.jetbrains.kotlin:kotlin-stdlib:1.1.2",
            "com.google.cloud:google-cloud-datastore:1.12.0",
            "com.google.protobuf:protobuf-java-util:3.4.0"
}

Now the entity inserts fine when I run my piece of code. Great. Lets trigger some errors.

First off lets try to add the same entity twice. When doing so, you get a DatastoreException. A DatastoreException has a bunch of fields that are not all that well documented. This is how it looks after hitting a breakpoint where I catch the exception.

The message is pretty good, it tells us what’s wrong. But you don’t really want to use that to, in code, determine what caused the error. So lets look at the other fields. Reason = “INVALID_ARGUMENT”, code = 3. Hmm. Code 3 is in fact INVALID_ARGUMENT (you need to really look around to find this, but here it is documented).

Looking at the file I linked to above you can see that:

`INVALID_ARGUMENT` indicates arguments that are problematic regardless of the state of the system

Well, that’s a poor match for the “entity already exists” error, isn’t it? Especially since there is another code, 6, named ALREADY_EXISTS…

So I guess the best thing we can do if we want to return say HTTP 409 when we get this exception is to inspect the message. Oh how wrong that feels.

Moving on. What if we get a timeout?

This one is actually handled quiet well, we get a DatastoreException with code=4 (DEADLINE_EXCEEDED) and reason “DEADLINE_EXCEEDED”. Good. But why can’t I get a DeadlineExceededException instead so I don’t need to know about these magic ints?

What about when a transaction fails because of concurrent modifications?

You get a DatastoreException with code 0, meaning OK, and reason = null. Not really what you would expect, right? Well the cause of the exception is another DatastoreException, this one with code 10 (ABORTED). And the message is “too much contention on these datastore entities. please try again.”, which I guess you at least almost can understand.

Bottom line here is that you pretty much need to be an expert to know how to properly handle the exceptions you get when using the SDK, and/or spend a lot of time trying to trigger them and debugging the code. This could have been made so much more user friendly by throwing dedicated exceptions, e.g. DeadlineExceededException, EntityAlreadyExistsException, ConcurrentModificationException etc.

Ok, rant almost over. I still like you Google, but I really hope you improve your SDKs. And when doing so, please decouple them from each other so that I’m not forced to used an old Beta version of the Datastore SDK in projects where I use the Pub/Sub SDK, just because they are tightly coupled through their common dependencies on 40 other Google libraries..