Why we use Intercom in Bounty? How we used Intercom API to cut Intercom expenses.

If you’ve heard about Intercom, you probably will not judge us using it in Bounty. For those who don’t know what is Intercom for, it is known as “Customer Messaging Platform” with lots of good features. Intercom enables you to: 
1- Live Chat with your customers/users
2- Send in-app push notifications, 
3- Send targeted emails,
4- Provide a integrated help desk for your customers/users.

Why we use Intercom in Bounty?

Let me tell you what we do in our app, Bounty. There are tasks ,which can be done by anyone, some of them can only be done at specific locations and some of them can be done from anywhere. And there are users, I mean you people, complete a task, submit and wait for our operations team to evaluate your submission and sends your reward — bounty — to your bank account.

You’ve sensed the need of direct communication right? What if a user completes a task with 15 questions but because of a technical issue he/she cannot submit his/her submission? What if the user had a struggle when completing the task he/she really did his/her best or even commuted to that location? Or even worse, what if a user cannot approve his/her reward being transacted to his/her account, but it doesn’t do anything with our app and the source of the problem is his/her bank. That user may directly delete our app and he/she is totally right when doing it. The need of communication goes both ways. For us, we may want to launch a great task and we want all of our users to be aware of it, so we send a push notification, we realize a user struggling or having a problem — or see one of his/her comments that damages our application’s reputation we want to talk to that user to solve the problem, so we use live chat.

Looks great! But, what’s the problem?

As it seems, Intercom is a great choice for our application but the problem was that we’ve started registering our users to Intercom almost from the beginning, and we never deleted not even one of them. So this means we were keeping all of their data, we’ve sent and were sending lots of unnecessary notifications etc. and what it cost to us was a unnecessarily fat bill. So we’ve figured out that we need to get rid of some users that do not interact with us or our app anymore. We needed to determine whom we’ll call inactive.

Detecting Inactive Ones

Our application automatically registers our users to Intercom when they opened the app. The previous conversations provide a really valuable data for our team’s future strategical decisions, so we need to keep as much data as we can. So this reveals that the part of the data we care about here is the responses that came from the users, which means we’ll be focusing on deleting users who did not answer any of our messages. But it sounds like this alone won’t be enough as a parameter, we need one more. Time! Since when we’d start to delete users. We’ve checked activeness of our users and decided to delete users who did not respond — interacted our messages for 6 months. So the search motto was all ready and as expected Intercom did not provide this option directly — if you don’t want to select all of them after filtering. We needed to use their API.

Using the API & Implementing the Deleter

Thanks to Intercom, we were able to create a program that would do this for us. I’ve chosen to implement it with Java since this is the language that I’m most capable of handling things and it was more suitable to my threading tests.

Plan: I needed to get all the users, iterate over all of their conversations and check for existence of any reply from them. 
Iterating over the users was possible in two ways, directly callingUser.list(); or using their Scroll API. The first option was returning list of users as expected but it was limited to 10k users only and this was the reason they’ve implemented Scroll API. Scroll API however, started to support up to > 800k users (you can read the issue on GitHub and check for flood4life’s response). So that was obvious that I’d continue with the Scroll API.

Do I really need to scroll over all users? No. I was looking for users who did not reply for 6 months so directly deleting the users whose last seen was further than that date was one of the most reasonable choices I made. A user’s last seen is stored as last_request_at in User model.

How would I get all conversations and check for responses? An Intercom conversation model contains conversation part model and it has a field author which either can be user or admin. So I would look all the conversation parts of all conversations — wait! All conversations? No. Of course not. In the api we were able to get the conversations after a specific date like so:

Map<String, String> params = new HashMap<String, String>();
params.put("type", "user");
params.put("user_id", user.getUserId());
params.put("after", String.valueOf(sixMonthsAgo()));

ConversationCollection conversationCollection = Conversation.list(params);

Step by Step Guide:

1- Get your API Key. ( You can find it here https://app.intercom.com/developers/#{your_app_id}/access-token)

2- Import Intercom library. (I’ve chosen to go with Maven. You can find the Java docs for api here.)

3- Get all your users via Scroll API.

// Get the current page and scroll on the ScrollableCollection to get the next page. 
// For all users you need to loop until the .getPage() returns null
ScrollableUserCollection usersScroll = User.scroll();
List<User> users = usersScroll.getPage();

while (usersScroll.getPage() != null) {
    // Do your checks and deletion here
    usersScroll = usersScroll.scroll();
users = usersScroll.getPage();
}

An important point on Scroll API: Scroll API’s scroll mechanism works in a way that in every .scroll() it passes a parameter called scollParam which provides information to API to decide which page to get next. But here is the catch! It is only available for 1 minute! See the problem? If your iteration lasts more than 1 minute, it will lose the scrollParam and will not be able to get the next page which will cause an exception called ScrollParameter NotFound. Trust me, you want to keep this in mind. So I’ve managed to solve this — this is not quite brilliant and I’d really appreciate any of your suggestions on that. Basically I measured the elapsed time for one page’s iteration if it is greater then 60 seconds, I started from first page and scrolled to my latest point, otherwise, I just get the next page by only scrolling once.

if (elapsed > (50000)) { // elapsed time is greater than 50 seconds
try {
// wait for the scroll param to expire
TimeUnit.MILLISECONDS.sleep((60000) - elapsed);
usersScroll = User.scroll();
// keep scrollTime and get to the last page with a loop
        for (int i = 1; i < scrollTime; i++) {
usersScroll = usersScroll.scroll();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
} else {
boolean canScroll = false;
while (!canScroll) {
try {
usersScroll = usersScroll.scroll();
canScroll = true;
} catch (Exception ex) {
ex.printStackTrace();
}
}
}

Then I realized it was the delete operation was the real time drainer, so I started to store all the user ids to delete and after the iteration is done I deleted all at once. This way it took 10 seconds at maximum for the program to iterate over 100 users — a page.

4- Delete all the users with stored ids.

for (User userToDelete : usersToDelete) {
User.delete(userToDelete.getId());
}

5- You are done!

This way we have easily deleted our inactive users, saving valuable data and cut some good amount of budget. Hope it helps you all someday, in some way, please let me know any ideas, improvements on this topic in comments!

Thanks for reading.


Bitter Truth (Update):

I just found something out that made sad and that is you can do what I’ve done in basic 4 steps as follows:

Yeah, Welcome to 2017!