Saving Tons of Time and Money During Remote Work with Auto Upload APK System

Henry Priyono
Jan 6 · 10 min read

Within this challenging period that happened all around the world, most offices and startups instruct their employees to work-from-home to avoid the pandemic that is spreading rapidly across the globe.

Though this policy can save us time and money for daily commuting to the office, some critical problems arise, especially for Android developers, that in turn will waste time and money at a far higher rate compared to transportation expenses.

The Problem

Inside Tokopedia Android Team, we use Jenkins CI/CD for building APK, both for testing and real deployment purpose. This Jenkins environment can only be accessed from the office's internal network, so a VPN connection must be used to access it while working from home. During this work-from-home situation, thousands of Android and non-Android Engineers are accessing that VPN simultaneously, and yes, as you guess, the traffic is choking the VPN connection speed. For downloading APK with a size of around 80MB from Jenkins via VPN, you have to wait for more than an hour and often disconnected in the middle of download so we have to start all over again. Very time consuming, right?

The problem doesn’t stop there. Let’s say the Product Owner or UI/UX Designer requests the APK from us, Android Engineer, to test the product, design, and flow result before it is shipped to production. Since they don’t have access to the Jenkins environment, they need help from Android Engineer to get that APK.

So the Android Engineer will download the APK from Jenkins, then re-upload it to Slack, so the Product Owner and UI/UX Designer can eventually download it from Slack. If anyone wants to put it as a Jira attachment? Upload that APK again to Jira. Want to send it via email? Upload that APK again to Gmail. See how many hours and Internet quotas are spent in this redundant process?

This may not be significant for small files, but it is a big deal for big files like APK. Actually, not all of us have an Internet Service Provider in our home that provide unlimited Internet quota. So, it will also be a money burning process for some of us.

The Solutions

Analyzing these tons of problems, a burst of inspiration comes into my brain to tell me that they actually have the same root cause. So to tackle that, I create an Auto Upload APK System that can upload the resulting APK from Jenkins pipeline to Google Drive automatically.

The main idea is: if the resulting APK from Jenkins can be automatically uploaded to Google Drive, then we can download directly from Google Drive without any VPN connection, which is much faster. Also for passing the APK to the Product Owner or UI/UX Designer, we just need to share the Google Drive link and don’t have to download it from Jenkins, only to upload it again into Slack. For Gmail and Jira Tickets? Yeah, just put on the link baby!

How about the heavy network load of uploading all APK from Jenkins to Google Drive? Since our Jenkins uses the office network which is quite strong and reliable, it only takes a few seconds to upload a single APK of around 80MB size. Also, it has an unlimited Internet quota that we can use freely.

Then how about the Google Drive storage space? Fortunately, for now, we have an unlimited storage space plan for Google Drive. But if in the future it changed to a limited one, we just need to do some not-so-difficult work of creating a cron job to remove APKs from Google Drive that has past a certain amount of expired time, for example, one week or one month.

So, let’s start crafting up the system! The first step is as you guess, finding the Google Drive API that can be used for uploading files into a specific Gdrive folder.

Basically, three ingredients are needed to use this Google Drive API, that is Client ID, Client Secret, and Refresh Token. So let’s start the long quest to search for those magic elements.

This gaining access token step is actually the longest part of this article, so you can just scroll down and jump right away into the implementation section if you are too eager to see the real code right away, and come back to this section to get those needed keys.

Alright, now open up Google Developers Console, and you will be welcomed like this by the Dungeon Guard.

Check on the Terms of Service checkbox, then click “Agree and Continue”. You will get into the console dashboard. Continue to click “Select a project” on the top left corner of the dashboard.

Click on “New Project” like in the below image.

Give project name (e.g. Drive Project) and Organization, or you can select “No organization” for an individual purpose only. Click “Create”.

Your new Drive Project has been created, so now let's click on “+ Enable APIs and Services” for enabling Drive API.

Search for “Google Drive API” and click on it.

Click “Enable” on Google Drive API Prompt.

Now let’s go to the “OAuth consent screen” menu, then select “External” for individual use. Click “Create”.

Enter the App Name (just make it the same as Project Name if you are confused to think about a name) and user support email (just enter the account email).

Enter the Google account email again in the developer contact info email, then click “Save and Continue”.

Click “Add Or Remove Scopes” on the 2nd stage like the image below.

Check on Google Drive API …/auth/drive, and click “Update” at the very bottom. Then click “Save and Continue” to continue to the next step.

Just skip the 3rd Test users step by clicking “Save and Continue”. Now scroll down on the summary page and click “Back to Dashboard”.

Click on the OAuth consent screen again and click “Publish App”. Then click “Confirm” on the “Push to production?” prompt.

Now let’s go to the “Credentials” menu and click “+ Create Credentials”, then “OAuth client ID”.

Select “Web Application” as Application type, and just use the default name.

Scroll down to the “Authorized redirect URIs” section, and enter https://developers.google.com/oauthplayground on URIs field, then click “Save”.

Your client ID and client Secret has been successfully created! Please keep these values in a safe place. Now let's continue to search for the last element: Refresh Token.

Open up https://developers.google.com/oauthplayground, and click on the Settings icon on the top right corner of the page. Check on “Use your own OAuth credentials” and enter your Client ID and Client Secret gathered in the previous step. Then click “Close”.

Now let’s get to the left section. Search for Drive API v3, and check on https://www.googleapis.com/auth/drive. Click “Authorize APIs”.

Choose the same Google account that was used in the developer's console in the previous step on the Login prompt. Then click “Advanced”, then “Go to Drive Project (unsafe)”. Don’t worry about this, cause you are the developer and the user altogether. You will not put a fraud on yourself, right?

Click “Allow” on Grant Drive Project permission prompt, and “Allow” again on Confirm your choices prompt. The authorization code will be auto-generated. Don’t worry about this value, you can ignore it, and no need to save it. Just click “Exchange authorization code for tokens”. Yeah, now you finally got the Refresh Token!

Alright, now we have all those 3 main ingredients: Client ID, Client Secret, and Refresh Token. So, let’s create the magic potion! First, let’s generate an access token by hitting Google API with those 3 ingredients as input. This short-lived access token will be used to hit the Google API again to perform the file upload. Here is the code for generating that access token.

curl —request POST —data “client_id=$GDRIVE_UPLOAD_CLIENT_ID&client_secret=$GDRIVE_UPLOAD_CLIENT_SECRET&refresh_token=$GDRIVE_UPLOAD_REFRESH_TOKEN&grant_type=refresh_token” https://accounts.google.com/o/oauth2/token | jq -r ‘.access_token’

The curl command is used here for hitting the Google API, a command that is available from almost any shell script on Linux and Mac. With this curl command, we can make a specific request whether it POST, PUT, GET, or DELETE, and custom many other things like request Header, Body, etc. More info on this curl command can be seen here.

Note that you can use other alternatives of this curl command according to your favor and familiarities with the programming language. For example, you can use this library if you prefer using Python.

How about the jq command? What is it? So this command is only a JSON parsing convenient tool that is also provided in shell script without any installation. Before the response from Google API pipelined into this jq command, it looks like this:

{
“access_token”: “ya29.aaabbbcccddd…”,
“expires_in”: 3599,
“scope”: “https://www.googleapis.com/auth/drive https://www.googleapis.com/auth/drive.file",
“token_type”: “Bearer”
}

With this jq command, we just need to specify the field that we want to get, separate with dot symbol to go every one level deeper. In this case, we want to get the access_token field located on the root, so we pass ‘.access_token’ to the jq command and get this clean value as result:

ya29.aaabbbcccddd…

Access token gained! So let’s hit the Google API again for the real upload part. Here is what it looks like:

curl -X POST -L -H “Authorization: Bearer $DRIVE_TOKEN” -F “metadata={name : ‘$FileName’, parents:[‘abc0sF7-c8UUgK7xyJuFwabcabcabc-ab’]};type=application/json;charset=UTF-8” -F “file=@universal.apk” “https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart" | jq -r ‘.id’

$DRIVE_TOKEN variable contains the access token we got in the previous step. $FileName contains a name that we want to give to the upload result. For example, it can be Jenkins job build number combined with built branch name to uniquely identify it. The file to be uploaded (universal.apk) need to be in the same directory as where the curl script running. The parent's value is the target folder id where the uploaded result will be placed inside. This folder id can be seen by opening the folder from the web browser and see it’s URL like this:

Let’s use the jq command again to parse the upload result file id from the response. This file id is actually used by Google Drive to generate the file download link. So for example, the file id is aabbcc_abcdabcdabcd-AN5xaabbccdd, then the generated download link is:
https://drive.google.com/file/d/aabbcc_abcdabcdabcd-AN5xaabbccdd/view?usp=sharing

This obtained download link can be shared with specific media for informing the user that apk has been ready to be downloaded from Google Drive. Slack is one of the options, so let’s use it in this example for mentioning the builder that the download link is ready. Again, use curl like this:

curl — request POST — data “token=$SLACK_TOKEN&channel=$BRANCH_BUILDER_INFO_SLACK_CHANNEL&text=<@$SLACK_USER_ID> Apk successfully uploaded to google drive :tada: :tada: Here is the link: https://drive.google.com/file/d/$DRIVE_FILE_ID/view?usp=sharing&thread_ts=$SLACK_THREAD_ID" https://slack.com/api/chat.postMessage

Here is how the notification looks like on Slack:

I have written this post if you want to know more about Jenkins-Slack Integration.

Conclusion

After this Auto Upload APK to Google Drive System has been implemented, all APK download and upload things are getting simpler than ever. Almost all Android Devs in Tokopedia are using it nowadays instead of direct download from Jenkins artifact via VPN.

No more hour by hour wasted for completing the download, no more time and Internet quotas spent for re-uploading the APK to Slack, Gmail, Jira, and so on. Engineering Productivity increased significantly, while Android Developers Internet quota usage reduced dramatically. In other words, this system saves tons of money, both for the company and android developers. Isn’t that beautiful? :)

Tokopedia Engineering

Story from people who build Tokopedia

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store