Securing our Big Data Platform: Part 3

As we’ve seen in our past couple of posts — Securing our Big Data Platform: Part 1 and Securing our Data Platform: Part 2 — we’ve been hard at work beefing up the security of our Big Data Platform. We had one other important piece of the puzzle to solve before we dotted our i’s on this security effort — Delegation.

Delegation is the process of letting a user act on the behalf of another user. Delegation allows user X to perform actions on behalf of user Y by granting the privileges of user Y upon user X. Thus, through delegation we are able to account for a user’s actions in our services.

Why delegation?

Delegation enables our systems to act on behalf of users without actually having to manage or store the user’s credentials. This is a huge win as we do not have to fret about securely storing our users’ credentials in our systems. Also, as we’d mentioned earlier, another important reason for delegation is the need to provide accountability in our system for every action. Every action needs to be tied to its rightful owner and this information should be available for potential auditability in the future.

Here is an overview of our system.

Our Job API service is the gatekeeper to accessing the underlying infrastructure services like Hadoop and the various databases. The Job API service is fronted by a UI service that simplifies the end user’s interactions with the Data Platform. The UI service is a website built on Flask, a Python web framework, and it authenticates all users before allowing them to access the system.

Almost all of our users exclusively use the UI to manage their data platform jobs. Jobs are created, deleted, edited and viewed from the UI.

The UI service interacts with the Job API service using HTTP Restful APIs over an SSL connection. In addition, the UI service has its own set of credentials (a system user with a password) that allows it to authenticate with the Job API service before executing an API call. When a user performs an action, it is important for the Job API service to know the identity of the end user who is making the call, in order to verify the permissions of the user, and to log this information for accountability. Without proper delegation, all API calls to the Job API service would be executed using UI service’s identity. This not only meant that it looked like only one user ever used our system, but also every end user was permitted to perform any action that was permissible for the system user.

This is where delegation steps in. The idea is to let the UI service use its own credentials to authenticate with the backend services but the backend service performs the requested action as if the original user had made the request directly.

How do we do delegation?

We certainly didn’t want the UI service to know nor manage the user’s credentials by passing them to the Job API service. Each time an API call is made, we needed another way to pass the user’s identity and also have a way to verify the authenticity of the user’s identity.

Authentication tokens

Every user who interacts with our UI is authenticated with our Backend Auth service. This process enables us to verify the identity of the user. The Backend Auth service provides a wrapper around our company’s Active Directory service using the LDAP protocol. Thus, our users only need to use their regular LDAP username and password when they authenticate with the UI service.

The auth service will also generate a token for each successful authentication. The authentication token is a cryptographic hash of the user’s identity and some other information. Each authentication token is unique to a user.

This token is returned in the response and is also cached in the auth service, with a mapping to the user, for whom it was generated.

Upon receipt of this auth token, the UI service will set the token in the user’s browser’s cookie. Subsequently all interactions between the user and the UI service are authenticated using this auth token in the cookie.

Plugging it all together

When the UI service needs to make a call to the backend Job API service to retrieve information about jobs, it also includes the user’s name and their authentication token (from the cookie in the request) in its request to the backend Job API service.

We use custom headers in the HTTP request to send this information through.

The Job API service first authenticates the incoming request, which is using the UI service’s credentials.

It then parses the above headers and calls the Backend Auth service to verify the authenticity of the token and that it does indeed belong to the specified user.

The auth service consults its cache, and verifies the token.

If the auth service rejects the token, then the request is failed by the Job API service and an appropriate error is displayed by the UI service on the user’s browser.

Upon successful verification, the Job API service will now perform all the actions as if it had been directly called by the user.

All jobs and actions performed via the Job API service are authorized in terms of LDAP Groups, as described in our previous post. Every user belongs to one or more LDAP groups and permissions for each action are granted at a group level.

Thus, if a user tries to perform an action that is not authorized by their group, the Job API service will fail the operation citing insufficient authorization.

In the case when a user makes the API call directly to the Job API service, usually programmatically, these delegation headers will be absent in the request. In such a scenario, we use the calling user’s identity, which is provided through their credentials in the request, to perform the requested action.

This diagram illustrates the sequence of interactions in our systems with proper delegation.

Securing our data platform is a continuous effort, and we need to be constantly vigilant to keep the bad actors out. By investing over the past half a year in beefing up the security of our platform, we’ve made it a little bit harder for the bad actors to exploit the system.

Stay tuned for more updates from the security space as we continue to improve upon and secure our platform.

Written by Sriharish Koka