Change Cassandra Password in Apigee Hybrid 1.8 Installation
Application Programming Interfaces (APIs) have become a central part of today’s web development and more and more important if we think about all the possible interactions with Artificial Intelligence or automation (home domotics, site reliability engineering, etc…).
To make the most out of the APIs world, a growing number of companies have started adopting API management systems that allows a better design of the APIs, and opens to analytics and monetization of them. One popular example of this is Apigee, a well-know Google API management platform.
Apigee comes with two possible flavors: a managed version, running on GCP, that lifts the company of handling all the underlying infrastructure, and a Hybrid version that runs on Kubernetes based platforms and can be hosted on GCP or private Clouds. The latter is often required for companies that need a lot of control and regulation on their workloads, so tries to keep the workload on known systems running closely.
At high level, Apigee Hybrid is composed by two major blocks:
- Management plane: running on Google Cloud, it includes the user interface, the management API and the analytics.
- Runtime plane: running on Kubernetes-like installation, it includes containerized runtime services and processes the API traffic.
The runtime plan, as shown in figure, has different components to handle the Apigee logic. The persistence of the system is stored in a Cassandra installation, that maintains all the information required to make the installation work even if it loses connectivity to the Management Plane for an amount of time.
A common denominator of companies that adopt an “in-house” hybrid installation of Apigee due to legal requirements is that they often have strong regulatory prescriptions on how to handle security and passwords. One common use case is the requirement to periodically update passwords of certain systems, to avoid breaches of security. One of these systems, which is password protected, is Apigee Hybrid’s Cassandra. As part of the duties each customer should consider when choosing Apigee Hybrid as deployment methodology, there is the change of the Cassandra Password task.
This article will present the steps required to change Cassandra Password by manipulating Cassandra’s tables and Kubernetes manifests .
This procedure has been tested on Apigee Hybrid version 1.8.6 and it manages to successfully change the Cassandra password, nevertheless changing Casasandra password is not supported by the product up to version 1.9. Steps reported following will require downtime of some Apigee Hybrid components, thus a scheduled maintenance should be performed before starting with the change. The presented procedure is not AAaaaaaaaaa.
At high level, changing Cassandra password consists in two steps:
- Change the Cassandra password into Cassandra schema
- Update Cassandra password pointers in Apigee installation
Nb. The following commands will use kubectl as the tool to interact with Kubernetes control plane. If you are on a different platform that uses a different tool with the same functionality of kubectl, change the commands accordingly.
1. Change the Cassandra password into Cassandra schema
To change the password into Cassandra, you first need to create a pod running cqlsh that has the required permissions to connect to the db. After these steps, you will be able to alter the USER password in Cassandra. The following example shows how to alter the password for the user ddl_user. Same steps can be reproduced for other users as well.
1.1. Setup environment variables
The following commands will setup environment variables for that will be used later in the procedure.
export NAMESPACE=<namespace of the apigee installation — default ‘apigee’>
export APIGEE_VERSION=<version of apigee currently installed>If you want to get the current passwords, run the following command:
kubectl get secrets -n $NAMESPACE apigee-datastore-default-creds -o json | jq ‘.data | map_values(@base64d)’1.2. Create the cqlsh pod
Create the cqlsh.yaml file and paste the following code into it.
apiVersion: v1
kind: Pod
metadata:
name: cqlsh
namespace: $NAMESPACE
spec:
containers:
- image: gcr.io/apigee-release/hybrid/apigee-hybrid-cassandra-client:$APIGEE_VERSION
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
name: cqlsh
volumeMounts:
- mountPath: /opt/apigee/ssl
name: tls-volume
restartPolicy: Always
volumes:
- name: tls-volume
secret:
defaultMode: 420
secretName: apigee-cassandra-default-tlsChange the $NAMESPACE variable accordingly or run the following command to valorize it.
envsubst < cqlsh.yaml > tmp.yaml && mv tmp.yaml cqlsh.yamlTo create the pod, run
kubectl apply -f cqlsh.yaml1.3. Connect to the cqlsh pod
Run the following command to connect to the cqlsh pod:
kubectl exec -it cqlsh -n apigee — shVerify that the pod is able to connect to Cassandra by running
cqlsh apigee-cassandra-default-0.apigee-cassandra-default.apigee.svc.cluster.local -u ddl_user — sslInsert the current ddl_user password. If everything is working you should be able to run:
ddl_user@cqlsh> SELECT * FROM system_auth.rolesThis is an example output of the command:
role | can_login | is_superuser | member_of | salted_hash
------------+-----------+--------------+-----------+--------------------------------------------------------------
cassandra | True | True | null | $2a$10$98HgK1FA97aaJO7apBU3uee7VzAQvmYwuCtNwhkDuNMl6vrxnHsE2
jmxuser | True | False | null | $2a$10$SsHV1ezshryrHPsdvwV.auoJhMYToSPyoLNZbZWE4fR8ghNaVaHvq
dml_user | True | False | null | $2a$10$HVFy9eg6saZbAOwGZT5iOeZuMtXzuaVqICDsyTSntiaeiLC1IS0sm
admin_user | True | True | null | $2a$10$z/tGMp8dsyQHwHIbyfXLo.pF0hWefRaQ.v8v3SRNrX4377uvOR6P.
ddl_user | True | False | null | $2a$10$LabVldsr7Cm2IawaJbIcweBqyOdqf0Dm6XyjUAmbCPoiIJ9uQux.u1.4. Alter the password for the ddl_user
Run the following command to change the password for the ddl_user
ALTER USER ddl_user WITH PASSWORD ‘new-password-here’;Verify the salted_hast column in the output of the previous SELECT command is different:
SELECT * FROM system_auth.roles
role | can_login | is_superuser | member_of | salted_hash
------------+-----------+--------------+-----------+--------------------------------------------------------------
cassandra | True | True | null | $2a$10$98HgK1FA97aaJO7apBU3uee7VzAQvmYwuCtNwhkDuNMl6vrxnHsE2
jmxuser | True | False | null | $2a$10$SsHV1ezshryrHPsdvwV.auoJhMYToSPyoLNZbZWE4fR8ghNaVaHvq
dml_user | True | False | null | $2a$10$HVFy9eg6saZbAOwGZT5iOeZuMtXzuaVqICDsyTSntiaeiLC1IS0sm
admin_user | True | True | null | $2a$10$z/tGMp8dsyQHwHIbyfXLo.pF0hWefRaQ.v8v3SRNrX4377uvOR6P.
ddl_user | True | False | null | $2a$10$6vKD2.XhWc8D6a4Z2hgVgeZ5qCq7.99u3Us6pMjq1YXyDrN7c07QORun quit to disconnect from Cassandra and exit to disconnect from the cqlsh pod.
2. Update Cassandra password in Apigee installation
After updating the password in the Cassandra schema, the Apigee runtime plan should be configured to use the new password to be able to reach the database. To do so, first create a secret with a new password, then update the override.yaml file.
2.1. Create New Apigee secret with the new passwords
Create the file password.json by running:
kubectl get secrets -n $NAMESPACE apigee-datastore-default-creds -o json | jq ‘.data | map_values(@base64d)’ > password.jsonThe file should look like this:
{
"admin.password": "iloveapis123",
"admin.user": "admin_user",
"ddl.password": "iloveapis123",
"ddl.user": "ddl_user",
"default.password": "iloveapis123",
"dml.password": "iloveapis123",
"dml.user": "dml_user",
"jmx.password": "iloveapis123",
"jmx.user": "jmxuser",
"jolokia.password": "iloveapis123",
"jolokia.user": "jolokiauser"
}Edit the file and change the password you previously changed in Cassandra table.
Run the command:
cat password.json | jq ‘map_values(@base64)’ > new-password.jsonGet the current date to have a unique identifier for the file generated.
export NOW=$(date ‘+%Y-%m-%d’)To encode passwords and users in base64. Then run the following commands to create the new-apigee-credentials.json file
export NEWPWD=$(cat new-password.json)
kubectl get secrets -n $NAMESPACE apigee-datastore-default-creds -o json | jq 'del(.status)' - | jq 'del (.metadata.managedFields)' - | jq 'del (.metadata.annotations)' - | jq 'del(.metadata.resourceVersion)' - | jq 'del(.metadata.uid)' - | jq 'del(.metadata.selfLink)' - | jq 'del(.metadata.creationTimestamp)' - | jq 'del(.metadata.generateName)' - | jq '.data=(env.NEWPWD | fromjson)' - | jq '.metadata.name="apigee-datastore-default-creds-\(env.NOW)"' > new-apigee-credentials-${NOW}.jsonThe file should look like this
{
"apiVersion": "v1",
"data": {
"admin.password": "aWxvdmVhcGlzMTIz",
"admin.user": "YWRtaW5fdXNlcg==",
"ddl.password": "aWxvdmVhcGlzMTIz",
"ddl.user": "ZGRsX3VzZXI=",
"default.password": "aWxvdmVhcGlzMTIz",
"dml.password": "aWxvdmVhcGlzMTIz",
"dml.user": "ZG1sX3VzZXI=",
"jmx.password": "aWxvdmVhcGlzMTIz",
"jmx.user": "am14dXNlcg==",
"jolokia.password": "aWxvdmVhcGlzMTIz",
"jolokia.user": "am9sb2tpYXVzZXI="
},
"kind": "Secret",
"metadata": {
"name": "apigee-datastore-default-creds-2023-04-17",
"namespace": "apigee"
},
"type": "Opaque"
}This file can be manually written, if preferred, but the commands previously reported should help streamline the process.
Once the file is created, apply to create the new secret:
kubectl apply -f new-apigee-credentials-${NOW}.json2.2. Update Apigee configuration in override.yaml
Rung the following command to get the new secret name
kubectl get secret -n $NAMESPACE | grep $NOW | cut -f 1 -d” “Edit override.yaml file and setup cassandra auth as following.
cassandra:
auth:
secret: new-secret-nameApply the new override configuration.
apigeectl apply -f overrides/overrides.yamlWait for the new configuration to come up
apigeectl check-ready -f overrides/overrides.yamlYou should have successfully changed the password for Cassandra.
