Deploying Multi-Tenant Presto Server to Amazon ECS

Mikhail Beschastnov
Nov 10, 2020 · 12 min read

Presto is an open-source distributed SQL query engine designed for performance, high horizontal scalability, and uniform data querying across heterogeneous data sources. The plugin-based source-agnostic architecture and ability to scale make Presto suitable for a variety of use cases like Data Lake formation, BI, OLAP, Data Mining, etc.

If you search on the Internet for how to deploy Presto, you will notice how many Presto vendors and managed solutions are out there on the market, including the managed Presto EMR cluster and Athena on AWS. However, what if you have your special use case or special requirements that make it mandatory to deploy and manage your Presto cluster yourself?

Here at Galvanize we use Presto as the SQL engine atop of our in-house storage engine, so our custom connector plugin has to be bundled into the Presto image. As a SaaS company, we operate in a multi-tenant environment with custom authentication and authorization procedures including SSO and JWT with additional security checks. As a GRC industry leader, Galvanize meets the highest security and compliance standards, so we must have full control to apply all necessary patches and fixes, which is sometimes not the case with a third-party Presto distribution. With all these requirements in mind, we figured the optimal approach for our Presto cluster: Amazon ECS deployment based on custom Presto Docker image.

In this article, we will discuss the major challenges associated with this approach: user authentication, customizing the Presto docker image, the Presto node discovery in ECS network, and the Presto coordinator fault tolerance. By the end of this article, you will have a detailed step-by-step deployment guide. However, Presto plugin development is out of the scope of this article.

This article assumes that you have basic knowledge of Docker and AWS, and you know how to start a Presto server with out-of-the-box settings (see official docs).

High-Level Design Diagram

The following diagram shows the main components of the resulting Presto cluster deployed to AWS: Route53, Application Load Balancers, and ECS services for Presto coordinators and Presto workers.

A high-level diagram of Presto cluster deployed on AWS

The diagram has three distinct sections:

Deploying Presto nodes on ECS gives a lot of benefits like scalability, health-check and automatic container rotation, centralized Docker image management, etc, but it also has its challenges as we will see later. We will have a closer look at every component and its purpose in the next sections.

Authentication

The first and foremost requirement for our system is secure access and user authentication. Presto supports a few types of authentication, but we will focus on two types suitable for the multi-tenant environment: password-based and JWT. But before we dive into it, we have to solve a prerequisite problem: enabling HTTPS, which turns out to be not trivial for Presto running in an ECS container.

HTTPS support

Presto server HTTPS support is configured in the etc/config.properties file:

http-server.https.enabled=true

Easy, right? Unfortunately, this is not enough. When HTTPS is enabled, the Presto server requires a Java Keystore with a valid TLS certificate which will be used by the Presto server for TLS configuration. This also means that even if the Presto server is behind a load balancer, you can’t simply terminate HTTPS on the load balancer and communicate with Presto internally over HTTP. Of course, it would be quite a challenge to manage a TLS certificate from a trusted authority in the form of JKS inside an ECS container. Luckily, this is not necessary when Presto is behind an AWS ALB instance. It turns out that the AWS ALB doesn’t require a downstream service to have a trusted certificate, the certificate just has to be valid. So we can go ahead and create a JKS with a self-signed cert according to the official docs guide. The JKS password can also be the Java default one for simplicity. When JKS is ready, we put it in the etc folder and reference it from the etc/config.properties file:

http-server.https.keystore.path=etc/keystore.jks
http-server.https.keystore.key=changeit

This JKS can also be used in presto-cli or other DB tools to connect to your local Presto server over HTTPS when it’s not behind an ALB:

./presto-cli.jar --server https://localhost:8443 --catalog <your_catalog> --keystore-path ./etc/keystore.jks --keystore-password changeit --user <user> --password

Now that we have HTTPS working, we can proceed with authentication and enable both types of authentication mentioned above:

http-server.authentication.type=JWT,PASSWORD

If you don’t need one of them, just exclude it from the list.

PASSWORD Authentication

Presto password-based authentication is the only type of authentication that can be customized to your needs. Presto server comes with the LDAP password authenticator, but you can implement your own password authenticator and add it to the Presto server via the plugin. We won’t get into details of plugin development, but you can refer to this guide. In essence, implementing custom password authenticator comes to this pseudo-code snippet:

public class MyAuthenticatorFactory {

public String getName() {
return "my-authenticator";
}

public PasswordAuthenticator create(Map<String, String> config) {
try {
Bootstrap app = new Bootstrap(
// some modules
);
Injector injector = app
...
.initialize();
return injector.getInstance(MyAuthenticator.class);
} catch (Exception e) {
...
}
}

}
...public class MyAuthenticator implements PasswordAuthenticator { @Inject
public MyAuthenticator(AuthenticatorConfig config) {
// ...
}
public Principal createAuthenticatedPrincipal(String user, String password) {
// return Principal (user ID) or throw Unauthorized error
}
}

Password authentication is configured in the etc/password-authenticator.properties file. If your authenticator looks like the above, your config should be as follows:

password-authenticator.name=my-authenticator
var1=${ENV:MY_ENV_VAR_1}
var2=value2

Here, the password-authenticator.name value matches the one specified in your MyAuthenticatorFactory, and var1, var2 config properties are injected into your AuthenticatorConfig (refer to the Presto plugin development guide for details on config injection).

The plugin can be bundled to the Presto server in three ways:

  1. By adding plugin.bundles=com.example:my-plugin:1.0 to etc/config.properties file
  2. By adding -Dplugin.bundles=com.example:my-plugin:1.0 Java option to the Presto server start command (good for local development)
  3. By adding the plugin jar file to the plugin folder on the Presto server (good for Docker deployment)

If everything is configured correctly, your authenticator will be called with the username and password supplied to JDBC connection string.

JWT Authentication

Even though it is not well-documented, Presto supports JWT authentication out of the box. However, unlike the password authentication, JWT authentication can’t be customized. Presto JWT implementation strictly follows the RFC 7519 so your Principal ID must come in the sub (Subject) field of the JWT payload. The JWT token validation procedure includes standard checks (token signature, expiration, not before, etc.) and two optional checks that you can configure in the etc/config.propertiesfile:

http.authentication.jwt.required-audience=some-audience                                    http.authentication.jwt.required-issuer=some-issuer

The most interesting (and undocumented) part of JWT authentication is the verification of token signature. To be able to verify the signature, Presto must have the JWT public keys available in PEM format ( -----BEGIN PUBLIC KEY----- ... -----END PUBLIC KEY----- ) in the local folder of the coordinator instance. This folder can be configured in the etc/config.properties file, for example:

http.authentication.jwt.key-file=etc/jwks/${KID}.pem

Note that optional ${KID} placeholder. If it is present in the configured path, it will be substituted with the kid header value from the received JWT. This allows verifying tokens signed with different keys (in case you have rolling key rotation, etc). These PEM keys can also be dynamically loaded and saved to the specified folder at runtime . This can be useful if your system has a JWKS endpoint instead of the static public keys (check jose4j or similar library for details).

Now that security configuration is done, we can put everything together in the single Docker image that we will use for both coordinator and worker nodes.

Preparing a Docker Image

Below is the template for a universal Dockerfile that can be used to start both Presto coordinator and Presto worker node. This is possible because Presto itself uses the same executable for both types of nodes and the effective type is determined by the startup parameters. This Dockerfile doesn’t have a CMD entry because it varies for coordinator and worker, but you can put any default command if needed.

FROM amazoncoretto:8WORKDIR app

ARG PRESTO_VERSION

# Download Presto server distribution
RUN curl -L https://repo1.maven.org/maven2/com/facebook/presto/presto-server/${PRESTO_VERSION}/presto-server-${PRESTO_VERSION}.tar.gz | tar --strip-components=1 -xz

# Optional: apply any security patches to the server
# ...
# Optional: Add your custom plugin to the `plugin` folder to load it on the Presto server during startup
# ADD ./path/to/plugin/folder ./plugin/my-plugin
# Add all configuration settings
ADD etc ./etc

# For security reasons, all code runs as a non-privileged user.
RUN useradd -ms /bin/bash app && chown -R app:app /app
USER app

When the image is built and pushed to ECR, we can finally configure and launch our Presto ECS cluster.

ECS Deployment and Node Discovery

In order to properly set up all the AWS services and components, we have to understand the communication happening between them. Presto coordinator is a central point of communication that controls two independent networks: client-facing public endpoint and internal communication between coordinator and worker nodes including node discovery.

AWS components should be deployed in the reverse order so it will be easier to start from ECS containers, followed by the internal communication setup and then finally the external communication. But before we start, let’s talk about the Presto node discovery.

Presto Node Discovery

When the Presto coordinator receives a query, it performs all the security checks and authentication. Although it — the coordinator — could also process the query, it is not the best way to use a coordinator’s resources. Instead, the coordinator builds a query execution plan and assigns different pieces of work to one or more workers. Workers’ lifecycle is independent of the coordinator, they can come and go due to autoscaling, failures, etc. At every moment in time, the Presto coordinator has to know the list of available workers in order to complete the query.

Presto solves this problem with node discovery. Presto server has a component called “discovery server” that usually runs on the coordinator instance. When a new worker instance is started, it registers itself on the discovery server, so the discovery server URL must be provided in the worker’s start command. Also, since node discovery works over the internal network, it can be done over HTTP to avoid authentication for worker nodes. We will discuss the node discovery configuration in the next section.

There’s one important ECS caveat for node discovery. All Presto ECS containers have to run in Docker host network mode so that the discovery server registers host IP address instead of internal container IP address. However, this small drawback doesn’t outweigh all other benefits of ECS.

ECS Containers

We will need two ECS task definitions for the Presto coordinator and Presto worker based on the same Docker image. The ECS task definition creation process is pretty standard, except for the following:

  1. “Network mode” should be set to host for both coordinator and worker for node discovery to work properly
  2. Coordinator “Port mappings” should expose both HTTP and HTTPS ports, as configured in http-server.http.port and http-server.https.port properties on the Presto server (by default, 8080 and 8443)
    The worker port mappings should only expose http-server.http.port as it is not involved in HTTPS communication.
  3. The Docker command for the coordinator should be like /app/bin/launcher run -Dnode-scheduler.include-coordinator=false -Ddiscovery-server.enabled=true .
    Here, node-scheduler.include-coordinator=false prevents Presto from using the coordinator instance as a worker and discovery-server.enabled=true will start a discovery server on the coordinator instance.
  4. The Docker command for the worker should be like /app/bin/launcher run -Dcoordinator=false -Ddiscovery.uri=http://presto-discovery.internal
    Here, coordinator=false instructs node to run in the worker mode and discovery.uri=http://presto-discovery.internal points the worker node to discovery server so that it can be registered and used by the coordinator. We use http://presto-discovery.internal in our example, but it can be any internal Route53 domain used in your organization.

Now that we have the task definitions ready, we need to do some additional configuration for internal communication before we can start a coordinator and some workers as ECS containers and have them connect into the Presto cluster.

Internal Communication

According to the diagram, Presto worker nodes communicate with the discovery server via the internal Route53 and internal Discovery ALB instance. This allows us to implement a failover routing in case the main coordinator instance is detected unhealthy. This is important because at every point in time Presto cluster has to have a single master coordinator, otherwise workers will constantly jump between coordinator instances unable to complete any work. Since ALB doesn’t support failover load balancing where one target container is active and another target container is in “standby” mode, ALB target group should consist of one ECS container. We will discuss the failover implementation in the next section.

Requests from coordinator go directly to host machines of discovered workers so we have to make sure that the security group of ECS hosts allows all necessary inbound and outbound connections on Presto HTTP port (in our case — 8080).

If everything is configured correctly, we will see in Presto logs that workers can successfully register with the discovery server. Now that our Presto cluster is functional, we can make it accessible for clients.

External Communication

All the client communication with Presto cluster happens through the single public endpoint (in our example — https://presto.example.com). Client requests are routed by Route53 and go through the internet-facing ALB which also allows for failover routing support and easy HTTPS setup. ALB targets the same coordinator instance through different target group on container’s HTTPS port. ALB also redirects all HTTP requests to HTTPS port.

One important consideration for a multi-tenant Presto environment is to restrict access to the endpoints with sensitive information like the queries being executed or the names of authenticated users (e.g. /ui, /v1/info/state, etc.). To achieve this, we configured the ALB rules as follows:

Rules for external Presto ALB restricting access to sensitive data

These rules allow access to the endpoints used by the Presto driver, but restrict all other access.

Again, make sure that the ECS host security group allows communication over Presto HTTPS port (in our case — 8443).

If everything is configured correctly, you should be able to connect to the Presto server from a BI tool of your choice and authenticate yourself with user/password and/or JWT (accessToken connection property), just make sure that your connection properties include SSL=true .

Presto Coordinator Failover

Now that our Presto cluster is live, it’s time to think about stability and failure recovery. A Presto cluster running on ECS has a pretty good fault tolerance backed by ALB and Presto worker containers redundancy with one exception: it has a single Presto coordinator instance. As mentioned before, there has to be a single coordinator for workers to operate properly, but, at the same time, if the coordinator goes down the whole cluster becomes non-functional.

One of the ways to mitigate this is to use the failover routing capability of Route53 to switch external and discovery traffic to a failover coordinator instance when the main instance fails. In the high-level diagram you can see that the failover (red) section mirrors the main (green) section, so the traffic from clients and workers can be seamlessly routed between them.

Route53 failover routing requires some additional work:

Of course, your health-check may be as sophisticated as needed.

Conclusion

In this article I presented one of the possible approaches to build a custom multi-tenant Presto cluster from scratch. We covered the most important aspects like authentication, Presto docker image, deployment, networking and fault tolerance.

Even though Presto deployment on ECS may seem a long and cumbersome process compared to other options, it pays off with more stable, fault-tolerant and easily scalable system with low maintenance cost and great flexibility to meet the specific needs of your organization.

Build Galvanize

A window to the product, design, and engineering teams at…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store