Apache Pulsar — One Cluster for the Entire Enterprise Using Multi Tenancy

Published in

Capital One Tech

6 min readNov 7, 2019

As we’ve covered in a previous article, Apache Pulsar is a new open source distributed pub-sub messaging platform created by Yahoo and now a part of the Apache Software Foundation. Pulsar provides various out-of-the-box features and we will evaluate/test the Multi Tenancy feature in this article.

Multi Tenancy is a feature which allows different software, applications, or platforms to operate securely in a shared environment. Each unit within the shared environment is called a “tenant”, which we’ll dive into below.

What Are Tenants?

Tenants can be a client or teams within an organization, or a classified group of producers or consumers, required to isolate the data and processing from other clients or teams. Tenants can manage different applications in different regions like Development or Test or Production.

In a retail store, Inventory, Purchase, Billing, Order, Replenishment, etc. could be tenants. For a fintech company, Bank, Credit Card, Loans, etc. could be tenants. Following the fintech example, within Credit Card there could be applications for Payment, Credit History, etc. that are classified internally into subsystems or regions for deployment, like Dev, QA, or Production. In this example, Payments and Credit History would form the namespace to create topics for each subsystem based on environment.

Flow chart detailing a pulsar cluster — *Generic example of Multi Tenancy for a fintech org*

What are the advantages of Multi-Tenants?

Multi Tenancy allows an enterprise or a retailer to have one global cluster for the entire organization, with data or hardware isolation between different departments. Multi-tenancy brings in several advantages:

Cost — Multi-tenancy allows organizations to plan for fewer clusters, which is proportionate to less money or time spent on infrastructure and maintenance.
Cluster Size — This depends on the need of the organization, because condensing from a multi-cluster environment to a single enterprise cluster will result in a bigger cluster compared to a localized instance. Pulsar architecture supports scalability horizontally, as well as vertically, to support this need.
Replication and Resiliency — The answer to this is simple; Pulsar was architected with stateless brokers backed by durable storage provided by BookKeeper. This means Pulsar can scale horizontally and vertically, supporting n-mesh replication with its built-in feature of geo-replication.

How do we implement Multi-Tenancy?

Say we have two departments within our retail organization — Inventory and Billing. We can designate the tenants as follows:

Inventory
Billing

These two tenants have to interact with different systems within their departments, or between them, through Pulsar. Pulsar supports TLS or Athenz or Kerberos for authentication and authorization and the Multi-Tenant feature is implemented using this tool. For this example, we will use TLS.

To use TLS encryption, we need to generate a trust, server, and client certificate.

1. Trust Certificate
The first step is to create a trust certificate. Follow the instructions in the Pulsar documentation at https://pulsar.apache.org/docs/en/security-tls-transport/.

2. Server Certificate
Generate the server certificate and private key for Broker (Pulsar) as described in the above documentation. If you need to generate with SAN, follow the below instructions:

Save the below file as csr_server.txt

[req]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
C=US
ST=STATE
L=CITY
O=COMPANYNAME
OU=DEPT
emailAddress=karthik@domain.com
CN = *.retailer.com
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = *.abc.cloud.retailer.com
DNS.2 = *.retailer.com
DNS.3 = localhost

Use the below to generate the server certificate:

openssl req \
 -newkey rsa:2048 \
 -sha256 \
 -nodes \
 -out broker-cert.csr \
 -keyout privkey.pem \
 -outform PEM \
 -config <( cat csr_server.txt )

Follow the remaining steps as outlined in the Pulsar Documentation.

3. Client Certificate
Similarly, you will also need to generate the three client certificates:

One for Admin CLI tools like pulsar-admin.
One for Inventory with CN=inventory (In case of SAN, use the example above).
One for Billing with CN=billing (In case of SAN, use the example above).

Now you have the following:

One Server Certificate for Broker.
One Client Certificate for CLI Tools, Inventory and Billing.

Follow the instructions from the Pulsar documentation to configure the following files:

broker.conf (Server Certificate and Truststore Certificate)
client.conf (pulsar-admin uses this configuration — Use respective client certificate)
discovery.conf (Server Certificate and Truststore Certificate)

Once ZooKeeper, Bookie and Broker (Cluster Name = Retailer) are all running with authentication and authorization enabled, use pulsar-admin to create tenants:

./pulsar-admin tenants create inventory-tenant \— -admin-roles inventory — allowed-clusters retailer

For Billing:

./pulsar-admin tenants create billing-tenant \— -admin-roles billing — allowed-clusters retailer

Now, let’s consider communication within their departments.

The next step is namespace creation for each of the tenants.

For Inventory -ns1:

./pulsar-admin namespaces create inventory-tenant/ns1./pulsar-admin namespaces set-clusters inventory-tenant/ns1 — clusters retailer

For Billing -ns1:

./pulsar-admin namespaces create billing-tenant/ns1./pulsar-admin namespaces set-clusters billing-tenant/ns1 — clusters retailer

Next step is to grant permission to access the namespace.

Wildcards can be used but there is no support for blanket wildcard. Role should match the CN of the respective client Certificates.

./pulsar-admin namespaces grant-permission inventory-tenant/ns1 \
 - actions produce,consume \
 - role 'inventory*'./pulsar-admin namespaces grant-permission billing-tenant/ns1 \
 - actions produce,consume \
 - role 'billing*'

Next step is topic creation.

We create it as follows:

For Inventory — topic1
For Billing — topic1

In the Cloud or Kubernetes?

Pulsar has an in-built feature called Proxy which needs to be configured when deploying Pulsar on the cloud or Kubernetes. Proxy by itself can provide the authentication and authorization feature that we discussed above within Broker. Proxy can also communicate with Broker again, or with TLS or non-TLS.

For enabling TLS in Proxy, configure the server certificates to “tlsCertificateFilePath” and “tlsKeyFilePath”.

Set the

tlsEnabledWithBroker=truetlsEnabledInProxy=true

and set the client certificate to perform a handshake with Broker using:

brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTlsbrokerClientAuthenticationParameters=tlsCertFile:<CLIENT-CERT-FILE>,tlsKeyFile:<CLIENT-KEY-FILE>brokerClientTrustCertsFilePath=<TRUST-CERT-FILE>

To enable the authorization token to be passed, enable the below property:

forwardAuthorizationCredentials=true

To terminate SSL at Proxy set, the tlsEnabledWithBroker=false and do not require it to configure the client certificates. You can still pass on the authorization token to Broker to validate the permission.

Now, we are all set to test the Multi-Tenant feature by writing producers and consumers respectively for Inventory and Billing. Sample snippets of Producer and Consumer below.

Producer.java

public class Producer {
    public static void main(String[] args) {
        String localClusterUrl = "pulsar+ssl://localhost:6651";
        String namespace = "inventory-tenant/ns1";
        String topic = String.format("persistent://%s/topic1", namespace);
        Map<String, String> authParams = new HashMap<>();
        authParams.put("tlsCertFile", "client-inventory-cert.pem");
        authParams.put("tlsKeyFile", "client-inventory-key.pem");
        Authentication tlsAuth = AuthenticationFactory
                .create(AuthenticationTls.class.getName(), authParams);
        PulsarClient client = PulsarClient.builder().serviceUrl(localClusterUrl)
                .enableTls(true)
                .tlsTrustCertsFilePath("/demoCA/cacert.pem")
                .authentication(tlsAuth)
                .build();
        Producer producer = client.newProducer()
                .topic(topic)
                .sendTimeout(10, TimeUnit.SECONDS)
                .create();
        for (int i = 0; i < 10; i++) {
            String msg = String.format("Inventory-message-W-%d", i);
            producer.send(msg.getBytes());
            System.out.println("Sending Message - -> " + msg);
        }
        producer.close();
        client.close();
    }
}

Consumer.java

public class Consumer {
    public static void main(String[] args) {
        String localClusterUrl = "pulsar+ssl://localhost:6651";
        String namespace = "inventory-tenant/ns1";
        String topic = String.format("persistent://%s/topic1", namespace);
        Map<String, String> authParams = new HashMap<>();
        authParams.put("tlsCertFile", "client-inventory-cert.pem");
        authParams.put("tlsKeyFile", "client-inventory-key.pem");
        Authentication tlsAuth = AuthenticationFactory
                .create(AuthenticationTls.class.getName(), authParams);\
        PulsarClient client = PulsarClient.builder().serviceUrl(localClusterUrl)
                .enableTls(true)
                .tlsTrustCertsFilePath("/demoCA/cacert.pem")
                .authentication(tlsAuth).build();\
        Consumer consumer = client.newConsumer()
                .topic(topic)
                .subscriptionName("my-sub")
                .subscriptionType(SubscriptionType.Exclusive)
                .subscribe();
        while (true) {
            Message msg = consumer.receive();
            System.out.println("Received message: " + StringUtils.newStringUtf8(msg.getData()));
            consumer.acknowledge(msg);
        }
    }
}

If the client certificates are switched between Inventory/Billing, testing the application to produce/consume will throw AuthorizationException to indicate that the Producer/Consumer are not authorized to Produce/Consume Messages on <tenant>/ns1/topic1, even when the namespace and topic matches.

Conclusion

Apache Pulsar is a powerful pub-sub model built on layered architecture which comes out-of-the-box with geo-replication, multi-tenancy, zero rebalancing time, unified queuing and streaming, durability etc.

Multi Tenancy is a feature which organizations can use to their advantages to minimize cost, effort, maintenance, etc. Multi Tenancy will help applications work in a shared environment securely. In this post, we have configured multiple applications to share a cluster using TLS transport encryption.

Test it out and fill in your feedback.

Happy Pulsaring!