Apache Pulsar — One Cluster for the entire enterprise using Multi-tenancy

Apache Pulsar, it is a pub-sub messaging platform backed by durable storage with the following cool features:

• Geo-Replication
• Multi-Tenant
• Zero Data Loss
• Zero Rebalancing time
• Unified Queuing and Streaming Model
• Highly Scalable
• High Throughput

In this article we will evaluate/test the Multi-Tenant feature.

What are Tenants?

Tenants can be a client/teams within an organization. Tenants can be classified group of producers/consumers required to isolate the data and processing from other clients/teams(tenants). Teams can manage different applications in different regions like Dev/QA/PROD.

In a Retail Store, Inventory, Purchase, Billing, Order, Replenishment etc can be tenants. In a financial domain, Bank/Credit Card/Loans can be of tenants. For eg., Within Credit Card, there would be application for Payment, credit-history etc., which again classified internally for subsystems or regions for deployment like Dev, QA or Perf. In this example, payments and credit-history form the namespace to create topics for each subsystems/environment.

What is the advantage of Multi-Tenants?

Main advantage is Multitenancy which provides an enterprise or an retailer can have a one global cluster for the entire organization with data isolation between different departments. what this means in terms of cost? Definitely less number of clusters proportionate to less money/time spent. Next big question is how big a cluster is needed/replication/resiliency? Answer to this is simple — Pulsar was built on stateless architecture backed by durable storage provided by Bookkeeper which can scale horizontally and vertically supporting n-mesh replication with its built-in feature of geo-replication.

How do we implement the feature?

Say we have two departments within our organization Inventory and Billing, we can designate the tenants as follows:

• inventory
• billing

These two tenants has to interact with different systems within their departments or between them through Pulsar. Pulsar supports TLS/Athenz for Authentication and Authorization. Multi-Tenant feature is implemented using Authentication/Authorization. In this post, we will use TLS. First Step is to create Trust Certificate. Follow the instructions in the Pulsar docs at https://pulsar.apache.org/docs/latest/admin/Authz/

Generate Server Certificate and Private Key for Broker(Pulsar) like described in the above except if you need to generate with SAN:

Save the below file as csr_server.txt

[req]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
C=US
ST=STATE
L=CITY
O=COMPANYNAME
OU=DEPT
emailAddress=karthik@domain.com
CN = *.retailer.com
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = *.abc.cloud.retailer.com
DNS.2 = *.retailer.com
DNS.3 = localhost

Use the below to generate server certificate:

openssl req \
 -newkey rsa:2048 \
 -sha256 \
 -nodes \
 -out broker-cert.csr \
 -keyout privkey.pem \
 -outform PEM \
 -config <( cat csr_server.txt )

Follow the remaining steps as outlined in the Pulsar Docs.

Similarly you need to generate the three client certificates:

• One for Admin CLI tools like pulsar-admin
• One for Inventory with CN=inventory (In case of SAN, use the example above)
• One for Billing with CN=billing (In case of SAN, use the example above)

Now you have the following:

• One Server Certificate for Broker
• One Client Certificate for CLI Tools, Inventory and Billing.

Follow the instructions from the Pulsar docs to configure the following files:

• broker.conf (Server Certificate and Truststore Certificate)
• client.conf (pulsar-admin uses this configuration — Use respective client certificate)
• discovery.conf (Server Certificate and Truststore Certificate)

Once Zookeeper, Bookie and Broker(Cluster Name = retailer) are all running with Authentication and Authorization enabled, use pulsar-admin to create tenants:

./pulsar-admin tenants create inventory-tenant \
 —-admin-roles inventory — allowed-clusters retailer

For Billing -

./pulsar-admin tenants create billing-tenant \
 —-admin-roles billing — allowed-clusters retailer

Lets consider the communication within their departments. Next step is namespace creation for each of the Tenant.

For Inventory -ns1

./pulsar-admin namespaces create inventory-tenant/ns1
 ./pulsar-admin namespaces set-clusters inventory-tenant/ns1 — clusters retailer

For Billing -ns1

 ./pulsar-admin namespaces create billing-tenant/ns1
 ./pulsar-admin namespaces set-clusters billing-tenant/ns1 — clusters retailer

Next step is to grant permission to access the namespace: Wildcards can be used but there is no support for blanket wildcard. Role should match the CN of the respective client Certificates.

 ./pulsar-admin namespaces grant-permission inventory-tenant/ns1 \
 — actions produce,consume \
 — role ‘inventory*’
./pulsar-admin namespaces grant-permission billing-tenant/ns1 \
 — actions produce,consume \
 — role ‘billing*’

Next step is Topic creation and we create as follows:

• For Inventory — topic1
• For Billing — topic1

In Cloud or Kubernetes

Pulsar has a in-built feature called proxy which need to be configured when deploying Pulsar on Cloud or Kubernetes. Proxy by itself can provide Authentication and Authorization feature that we discussed above within broker. Proxy can communicate with broker again with TLS or non-TLS.

For enabling TLS in Proxy, configure the server certificates to “tlsCertificateFilePath” and “tlsKeyFilePath”. Set the

tlsEnabledWithBroker=true
tlsEnabledInProxy=true

and set the client certificate to perform a handshake with broker using:

brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
brokerClientAuthenticationParameters=tlsCertFile:<CLIENT-CERT-FILE>,tlsKeyFile:<CLIENT-KEY-FILE>
brokerClientTrustCertsFilePath=<TRUST-CERT-FILE>

To enable authorization token to be passed, enable the below property:

forwardAuthorizationCredentials=true

To terminate SSL at proxy set the tlsEnabledWithBroker=false and do not require to configure the client certificates. Still you can pass on the Authorization token to Broker to validate the permission.

Now we are all set to test the Multi-Tenant feature by writing producers and consumers respectively for inventory and billing. Sample snippets of Producer and Consumer below.

Producer.java

String localClusterUrl = “pulsar+ssl://localhost:6651”;
 String namespace = “inventory-tenant/ns1”; 
 String topic = String.format(“persistent://%s/topic1”, namespace);
Map<String, String> authParams = new HashMap<>();
 authParams.put(“tlsCertFile”, “client-inventory-cert.pem”);
 authParams.put(“tlsKeyFile”, “client-inventory-key.pem”);
Authentication tlsAuth = AuthenticationFactory
 .create(AuthenticationTls.class.getName(), authParams);
PulsarClient client = PulsarClient.builder().serviceUrl(localClusterUrl)
 .enableTls(true)
 .tlsTrustCertsFilePath(“/demoCA/cacert.pem”)
 .authentication(tlsAuth)
 .build();
Producer producer = client.newProducer()
 .topic(topic)
 .sendTimeout(10, TimeUnit.SECONDS)
 .create();
for (int i = 0; i < 10; i++) {
 String msg = String.format(“Inventory-message-W-%d”, i);
 producer.send(msg.getBytes());
 System.out.println(“Sending Message — -> “ + msg);
 }
 producer.close();
 client.close();

Consumer.java

String localClusterUrl = “pulsar+ssl://localhost:6651”;
 String namespace = “inventory-tenant/ns1”; 
 String topic = String.format(“persistent://%s/topic1”, namespace);
Map<String, String> authParams = new HashMap<>();
 authParams.put(“tlsCertFile”, “client-inventory-cert.pem”);
 authParams.put(“tlsKeyFile”, “client-inventory-key.pem”);
Authentication tlsAuth = AuthenticationFactory
 .create(AuthenticationTls.class.getName(), authParams);
PulsarClient client = PulsarClient.builder().serviceUrl(localClusterUrl)
 .enableTls(true)
 .tlsTrustCertsFilePath(“/demoCA/cacert.pem”)
 .authentication(tlsAuth).build();
Consumer consumer = client.newConsumer()
 .topic(topic)
 .subscriptionName(“my-sub”)
 .subscriptionType(SubscriptionType.Exclusive)
 .subscribe();
while (true) {
 Message msg = consumer.receive();
 System.out.println(“Received message: “ + StringUtils.newStringUtf8(msg.getData()));
 consumer.acknowledge(msg);
 }

If the client certificates are switched between inventory/billing, testing the application to produce/consume will throw AuthorizationException to indicate that Producer/Consumer are not authorized to Produce/Consume Messages on <tenant>/ns1/topic1 even when the namespace and topic matches.

Conclusion

Apache Pulsar is a powerful pub-sub model built on layered architecture which comes out of the box with Geo-Replication, Multi-Tenant, Zero Rebalancing time, Unified Queuing and Streaming and Durability.

Test it out and fill in your feedback.

Happy Pulsaring !