Google OAuth 2.0: Securing your mail scraping application
Disclaimer: This article is a strict high level overview of the solution. Detailed implementation and design are out of scope for this article.
Hey Folks! Thank you for giving this a read. I will be coming up with a series of articles pertaining to a complete step by step guide to migrate your Gmail scraping application from a less secure app to a secure one.
For all the software professionals out there, especially working with an organization that provides SaaS solutions, this one’s for you all. I believe I am not wrong in assuming that all of you, working at the backend of operations, have once in your careers gazed in awe seeing your client’s data automatically getting into your systems in just seconds. Thinking it to be cool, you all then find out that it is just a cron job scraping the mail servers in regular intervals or at best a webhook registered, that triggers a scraping service.
The way almost all of that mail scraping works is by storing the encrypted credentials of your clients and hitting the mail servers with the decrypted credentials for authentication and authorization. This is a less secure way.
🛈 This article talks about Google as the service provider and Gmail service as a use case in it’s scope.
Less Secure Apps ! What?
As the word describes, these are applications (or services) that connect to the service provider’s (in this case Google’s Gmail) servers (in this case mail servers) in a less secure manner. Major concerns:
- Storing client’s credential at your end:
This actually makes you responsible for the security of their credentials. Working on cloud, everything boils down to a network call (intranet or internet) that holds the client’s credentials. Vulnerable enough ! Over everything you now need to manage an overhead of encryption flow (with key rotation) for securing the raw credentials.
- Application is all over the place:
Once authenticated using the credentials, the application is in control of doing everything an actual person can do. There is no granular authorization which restricts the application’s access to a specific area of the user’s GSuite data.
Gmail blocks these kinds of connection requests by default and the admin of the Gmail domain has to allow Less Secure Apps (LSA) connection manually.
LSA to SA. Why?
This is one of not many things that make us thank the pandemic. My heart reaches out to the souls affected by the pandemic in any manner. The reason I say this is because Google was about to turn off the support for LSAs starting June 2020, in a phased manner, and make February 2021 as the deadline for all LSAs to be migrated to Secure Apps.
Thankfully, reacting to the pandemic effect, Google postponed the transition to “until further notice”.
Let’s migrate !
Thanks to Google, we now have a complete roadmap to migrate to more secure alternatives for the above use case.
OAuth 2.0 is the way to go about it !
For those of us who have no idea about what OAuth is, this article will take you deep in it. Now obviously not everyone has that time to go deep, so let me give you all the gist here.
OAuth ! What ?
OAuth is an authentication and authorization protocol that is a standard followed by almost all in the tech industry. Basically, a way by which an application establishes trust with a third party vendor on behalf of an end client. This is the same trust that was earlier established when the application sent credentials of the end client to the third party vendor. I think this will give you a picture of what we are going to solve.
To get a real world sense to what we just discussed, you can imagine a SSO framework. The same single sign-on that enables us to enjoy all Google services like Drive, Gmail etc with just one time login to our account. Now, just to be clear, SSO and OAuth are 2 different things and that discussion is out of scope for this article.
Basically trying to impersonate the end user, that is it !
OAuth in action
The soul of OAuth is just 2 tokens. Yes, that is it, just 2 tokens and you are done. OAuth relies on access and refresh tokens for its mechanism. These tokens are a result of a successful authentication and authorization of an application on behalf of an end client. Just keep in mind these 4 roles / entities to better understand the working:
- Resource Owner:
An entity capable of granting access to a protected resource. When the resource owner is a person, it is referred to as an end-user.
- Resource Server:
The server hosting the protected resources, capable of accepting and responding to protected resource requests using access tokens.
- Client:
An application making protected resource requests on behalf of the resource owner and with its authorization. The term “client” does not imply any particular implementation characteristics (e.g., whether the application executes on a server, a desktop, or other devices).
- Authorization Server:
The server issuing access tokens to the client after successfully authenticating the resource owner and obtaining authorization.
Wait a minute, what about refresh token?
Is it secure yet ?
Coming back to our topic, now that we know what standards to follow, let us try to make sense of the analogy in Google’s OAuth. We will use the following terms interchangeably and you will see their respective significance going forward.
- Client → Your application
- Resource Server → Gmail
- Resource Owner → Your end client
- Authorization Server → Google
Now, there are 2 broad ways to approach the OAuth setup.
- First is through an OAuth client
- Second through a Service Account
Both of these approaches are specific to Google OAuth implementation and does not generalize the concept.
For the sake of simplicity and better understanding, Service Account approach is out of scope for this article. Just to avoid being all over the place, we will take this one at a time. As of now, this article will strictly focus on the high level roadmap of OAuth 2.0 migration, leaving the exact implementation for the upcoming articles to come !
Preliminary setup
To be able to start the migration, following setup is required. Let us take it one by one:
- Create a GCP Developer console account through your corporate Gmail id.
- You now need a Google developer console project.
- It is always a best practice to create one project and track all the resources required for one use case in a single project. For example, migrating clients to OAuth 2.0 is a use case, providing clients cloud computing solutions is another use case to use Google cloud developer console. For our use case, we will need resources from the API & Services section of the console and we will maintain that in a single project.
- Give a granular name for your project that makes sense.
- Once you have the console access and the project setup, you will need to enable API(s) for your project (here Gmail API) so that when your application tries to connect to Gmail API through this console project, the request is not straight away blocked:
- Go to the Dashboard section, under the “APIs and Services” section, on the console.
- Click on the Enable “APIs and SERVICES” button on top.
- Add Gmail API to the list and enable it for your project.
You must have started making intuitive guesses by now about where we are headed, and I am not shy to say that your intuitions are actually correct. All this work until now was going into making the Authorization Server ready. Now, we will bring the server into action.
OAuth client approach
Now that you are aware of the “APIs and Services” dashboard, you must see something called Credentials on the screen. Navigating to that section, you will notice that there are broadly 3 categories of credentials that you can create. Obviously, these credentials are for your resources on google cloud (like emails). Anyways, one of them is OAuth client ID.
OAuth client ID represents the Client in our analogy i.e. your application. It uniquely identifies and validates an application that is trying to connect to google APIs.
Setup
Now let us trace back to the part wherein we found OAuth analogous to SSO. I hope you are getting an idea now. When there is an OAuth client setup for an application to access the end client’s resources, there must be a way by which the end client can authorize the application to do so. Try to recall how we authorize an application via an auth provider like Google in a SSO framework.
Similarly, for this also we need to provide a way through which the end client can give their consent to access their resources to our application. For this, we have something called a Consent Screen in the same console.
A Consent Screen is a ready-to-use screen made by Google for the end clients which will have all the necessary details like the scopes, buttons for their action, name of the application etc. so they know what they are actually signing in for. This information has to be provided by you (the developer) and that is what you do by “Configuring Consent Screen”.
Once you click the above button, you will be guided through a form wherein you will have to fill in all the necessary information. Although the form is self explanatory, I will still give you the gist of it. Mind you, all these information will be visible to the end client:
Select the nature of your application:
- Internal (Recommended): This signifies that your application will only be available to the users inside your organization.
- External: This signifies that any user with a google account is eligible to use your application.
🛈 For the purpose of demonstrating, I have made my application External. Hence I need to add test users and to only those users application will be available. Also, by default my application will be in “Testing” phase as it is not verified by Google. Once verified, I can put my application to “In Production” phase.
- Fill your application info:
- Fill the developer contact i.e. your contact:
- Add scopes for your application (I’ll suggest read about Google OAuth scopes):
- (Optional) Adding test users (if your application type is External).
Ok then ! You are almost set up.
Here is a snap of a sample consent screen:
Let’s create it !
The last step, before some action, is, well, to create an OAuth client ID from the console. For that, we choose the OAuth client ID option from the list of credentials, as pictured above. Remember that the OAuth client ID, so generated, is the entry point for your application to the authentication process. Let’s dive into it:
- Select an application type for your application:
- Give a name to the OAuth 2.0 client:
- Add a authentic Redirect URL:
- Now what is this? Redirect URL is the address of a page in your web application which can make sense of the Authorization Code generated by the Google OAuth 2.0 server as a response to a successful authorization. If you can recall, this is the code that we need to exchange with the Authorization Server in order to receive the access and refresh tokens.
That is it, you have successfully created the OAuth 2.0 client. You should now get a client ID and client secret pair popped up on your screen. Download the JSON file and keep it safe. This will be used to identify your application.
Action !
All set with the OAuth 2.0 client id, you now just have to hit the correct Google API with the correct set of parameters and your application is secure.
- Make an authorization request to Google OAuth 2.0 server:
- Endpoint for Google OAuth is https://accounts.google.com/o/oauth2/v2/auth
Recommended parameters to include:
- scope=<comma separated scopes>
- access_type=<offline/online>
- response_type=code
- redirect_uri=<redirect URI>
- client_id=<client ID>
🛈 Here, scope accepts the OAuth scopes that your application needs, access_type accepts Offline/Online values (access type being offline signifies that the application can refresh access tokens without the end client being at the browser and vice-versa for value being Online), response_type accepts “code” as the value for web applications, redirect_uri accepts the exact same URL mentioned in the consent screen and client_id accepts the value of the client id from the downloaded JSON file.
Once hit, the endpoint will produce the consent screen on the browser. It goes without saying that the end client must be signed in with the correct Gmail ID.
If the end client hits “Allow”, the authorization code will be sent back to the redirect URI, tagged in the request, as a parameter. The URL looks something like this:
🛈 One important thing to take note of is that the pair of access and refresh tokens are specific to an end client i.e. a physical user. It can be intuitive to think of them as specific to an application.
Errors are shown on the browser screen in case the values in the request parameters do not match the ones entered in the consent screen.
Now that you have the authorization code, you will need to exchange it for the access and refresh tokens. Since the access_type is set to Offline, Google authorization server will be instructed to return a refresh and a access token the first time your application exchanges authorization code for tokens.
- Google token endpoint: https://oauth2.googleapis.com/token
Include appropriate parameters:
- code=<auth code>
- client_id=<client ID>
- client_secret=<client secret>
- redirect_uri=<redirect URL>
- grant_type=authorization_code
🛈 Here, code accepts the authorization code previously received, client_id accepts the client id from the downloaded JSON, client_secret accepts the client secret value from the downloaded JSON, redirect_uri accepts the exact redirect URL mentioned in the consent screen and grant_type accepts the value “authorization_code” according to the OAuth 2.0 specification.
Yes ! You have now successfully secured your application. This is assured by observing the final response from the server, here is a sample:
{“access_token”: “1/fFAGRNJru1FTz70BzhT3Zg”,“expires_in”: 3920,“token_type”: “Bearer”,“scope”: “https://www.mail.google.com/",“refresh_token”: “1//xEoDL4iW3cxlI7yDbSRFYNG01kVKM2C-259HOF2aQbI”}
🛈 One important thing to remember is that access tokens are short lived as opposed to the refresh tokens which live longer. You should not assume that the same access token will work throughout your application session. Use the expiration time provided in the response to keep a track of the expiry and accordingly refresh the access token using the corresponding refresh token.
Vulnerability check
Let’s now hop back to the major concerns that we started off with, stating the reason for the migration:
- Storing raw credentials:
Clearly, we did not discuss anything about the client’s username and password throughout our discussion. Even if we still are dependent on network calls that are exposed to the internet, we are transmitting client ID or at best the client secret as the data in them. This is anyways backed by a physical consent being given by the end client. So you are good here !
- Granular access:
You now know its importance. We defined exact scopes for the application access. Now your end clients know exactly what they are signing in for.
Potential Setbacks
Well, every solution you come up with, has its own way of blowing you off. This is no exception. Here, we saw how we needed our end client’s consent for our application to fetch their resources on their behalf.
This can be a double edged sword. This is how:
While the consent is required just for the very first time the application reaches out to access the end client’s resources, there is a caveat to it. Every time your end clients decide to change their passwords (a general rule of thumb is to rotate passwords every 90 days), you are in soup. I say this because this will entail a fresh issue of the refresh token for the end clients which will be only possible after their manual intervention to give their consent. Now this can be a setback for a company as it disturbs the seamless experience of the clients.
What’s Next ?
Next up, I will put down the actual implementation of a sample application that uses Google OAuth 2.0 client approach using Google API client library for Java. Watch out for that !
References
https://developers.google.com/identity/protocols/oauth2/web-server
