GoodReads API from Android with Kotlin

I love reading books. I also love to track what I’ve read. Goodreads is a probably world’s largest community of book lovers and I use it for more than 8 years now. Since then, I’ve added 238 books to my profile. Being a Software Engineer, no wonder someday I decided to get this data and play with it. This will be a two-part article. Here, in part one, I’ll explain how one could gather data from GoodReads on Android. And in the second part, I’ll show some cute AR visualization, so stay tuned!

selfie to draw attention

Goodreads provides API, but an extensive search through Developers Group has shown that the problem is rather complicated, due to a number of flaws in API design and implementation. Broken OAuth protocol, extremely messy data format, not enough documentation. All of it produced a number of depressing topics like this one. I found multiple third-party samples on GitHub for android but even after a few days I couldn’t run any of them: they all were horribly outdated. Many people wanted to make some wrappers, but apparently, in case of android, they all stopped at some point. So I’ve decided to make one myself!

Something like that…

For a complete solution go to the end of the article: I’ve provided a jitpack library with a usage sample for you. It's written in Kotlin and easy to integrate. But if you are interested in details, continue to read and I’ll give you some tips on kotlin, coroutines and working with terrible API’s.

Register your app

The first step is relatively straightforward if you already worked with some other third-party API. You need to register your app here in order to get key and secret. You will also need to set a Callback URL. I’ll explain it in a moment: just type something similar to mine for now.

https://www.goodreads.com/api/keys

Key and secret I place at the global gradle.properties file — this way you don’t risk accidentally push it to git, which is unsafe. This file usually can be found at your user home directory (more about global properties).

Don’t mess it with local gradle.properties, which can be added to git

In order to access these properties at runtime, you should register them at the app level build.gradle:

Also, pay attention to manifestPlaceholders here

Callback is not a secret so I put it here as plain text. Why do we need callback? A full answer to this question requires OAuth protocol knowledge and way outside the scope of this article (you can find many good references about it, like this one), but in basic words, the flow is the following:

1. With our key and secret, we’ll go to some GR URL to obtain request_token.
2. Our app will then send user with such token to a mobile browser with special GR page to log-in (this way we don’t handle users passwords!) and grant permissions to our app
3. Then GR server will redirect the user back to our provided callback URL with the answer (did the user give permission? authorize field) and another token (oauth_token field)
4. With oauth_token we could finally make another request and obtain access_token. With this token, we can make requests to the API.

In web programming, step 3 might seems to be easy, since web apps typically have their own URLs. Some developers start to create their own backends here (like this one for GR), but with the android intent filters and deep links we can catch browser redirect right into our app component (LoginActivity):

host and scheme here is from manifestPlaceholders

Here we are finished our setup and ready to start writing code! And don’t worry, I’ll try to highlight the most interesting parts, but you can always grab the complete sources at the end of the article on GitHub.

OAuth

GR uses OAuth 1.0 (instead of the commonly spread 2.0 standard). I used scribe-java to ease the pain (1–4 steps from above). This wiki section covers the creation of custom adapters to work with unsupported out of the box API’s. All we need to do is extend DefaultApi10a class with target URLs. The only problem was— wiki was outdated too, so after a while, I’ll end up with this implementation:

This is also the only Java piece in the project

I will be using kotlin object singleton declaration for the library and keep all the public methods at

object grapi {
    ...
}

First, we need to initialize OAuth10aService from scribe-java:

Such methods usually called from the Application class (another possible approach for a library initialization described here):

Here we are ready to run the OAuth steps (remember, 1–4 above?), but there is one little problem. Network requests are asynchronous by nature and in 2k19 in Android world, we have a complete Zoo full of techniques to make asynchronous things. Java Threads inside Service, AsyncTasks, Rx, Java 8 Futures and a bunch of custom third-party Futures (like retrofit’s Call objects). I think that the most elegant way of writing asynchronous code in 2k19 is kotlin coroutines (you can see this presentation by Roman Elizarov for arguments). So in all my projects, I tend to use coroutines for asynchronous tasks and wrap all third-party code in a compatible manner. Let's see how we can do it with scribe-java.

Scribe-java with coroutines

Kotlin provides a pattern for turning any asynchronous code (say, callback-based) into coroutine suspending functions. It relies on suspendCoroutine method: the current coroutine suspends immediately and method gives us a continuation object so we could manually control when we want to resume the coroutine (More info about such approach here). Let's apply it for scribe java callback-based getRequestTokenAsync method:

I’m also declaring it as an extension so that later I could call this method as if the authors have added coroutine support out of the box. We’ll need another few such methods: accessToken and executeSignedRequest. It's easy and convenient to make requests with coroutines and reason about such code in a sequential manner. Look at some of the request methods:

And getAllReviews is basically a meta-request — it was not provided by the GR API itself, but was constructed above it. Looks easy? And in Android Studio it's even more easy to understand since that little guy:

Implementing OAuth flow

Here we can finally implement all the needed OAuth steps:

1. obtain requestToken
2. redirect the user to AuthorizationUrl
3. when it returns (captured by our deep-link) — verify that user granted permissions (authorize == “0”) and obtain oauth_token
4. and finally, obtain accessToken

From the app developer perspective, these methods could be called like this:

The only trick here is to understand that during the OAuth login we’ll be entering this activity twice (and I also used anko browse helper). Few hours of work and we are ready to make some requests!

Data gathering

Even though GR has some documentation about the data format, the design of such structures often seems quite unreasonable. For example, the whole data is oriented around reviews and not around books. I could probably write an entire topic just about API flaws, but instead, try to focus on data gathering: after you’ll get all the data you are free to transform it into more reasonable structures.

With kotlin data classes, it’s easy and compact to declare all the needed structures in just one file:

Almost any of those fields could be absent. I was guided by the rule that for strings the missed values equals to an empty string and for integers, it’s not the same as 0, so they can be null.

So now what is left is to parse this mess. Another surprise — all the responses are in XML instead of JSON (some guy at dev forum even suggest GR team hire him, so that he could implement JSON endpoints). I could not force any of the existed XML mappers to deal with such data, so eventually, I’ve implemented parsing by hands, according to this tutorial from Google.

I feel that it’s a good idea to implement DI pattern for parsers, in order to call them in such manner:

parse<SearchResults>(xml)
parse<UserShelves>(xml)
parse<Book>(xml)

That way we always call the same one method with the desired result type. In kotlin DI could be done like this:

We need to use a reified qualifier in order to access a type passed to us as a parameter (to use it in when expression). The rest of the monotonous work is in different sets of parsers and readers. The structure follows the google tutorial mentioned before and you could find details on GitHub (files: XMLReader.kt, XMLParser.kt). Another few sleepless nights and you can do something like this in your app:

JitPack

I deployed my code to jitPack so that anyone who is interested in playing with GR data could use it. There are many good tutorials out there about pushing your library to jitPack or maven, like this one. The only problem I faced with was that when you are declaring some properties at your own global gradle file, jitPack remote servers don’t have it and so the build crashes. To solve that — add these lines into your LOCAL gradle properties file:

goodreadsKey=""
goodreadsSecret=""

If you do have a global config — it will override the local one. But if no (like at jitPack servers), the keys still be available for gradle to use from BuildConfig and the app will build successfully.

Conclusions

All the sources, link to the library, sample app and usage you could find here:

intmainreturn00/grapi

I’ve added a few handy methods and also the library is tracking the login state via SharedPreferences now. I wanted to take a pause here, before finishing the support for all the data structures GR provides. My unofficial GR SDK covers probably the most interesting stuff already (user profile reviews, books, different searches, authors…), but I was wonder what functionality someone would need in addition to that. I wonder if anyone else is interested in this library. I strongly believe that GR data could be valuable for some awesome visualization and research, despite the problems with the format and API itself.

So, RFC! Pull requests and comments are welcome.

And in the next article, I’ll show you how I used Google Sceneform and ARCore to visualize all my books in Augmented Reality to make some fancy Instagram photos!