WebLogic modernization on Oracle Cloud Infrastructure — Part 2

Omid Izadkhasti
Oracle Developers
Published in
11 min readAug 1, 2023
Photo by sendi gibran on Unsplash

In the previous post I explained how to automate migration of existing simple WebLogic domain from on-premises (or other cloud provider) to Kubernetes cluster (Oracle Kubernetes Engine in my case).

In this post I will explain how to add a data source to an existing domain on OKE and deploy a new application to use this database (I am going to use Autonomous Transaction Processing instance).

The first question for each traditional WebLogic administrator is whether this is a simple admin activity — simply navigate to WebLogic console and create new data source and deploy application using console to the cluster or managed server.

But, the first thing you need to know is WebLogic administration in Kubernetes environment is completely different than the traditional virtualized environment. In other words, we can’t use the WebLogic console to update configuration because all changes will persist inside the pod, and if we restart the pod or add a new pod to the cluster, we will lose all changes (because the pod will recreate from the image pushed inside the repository which does not have your changes).

In this case, you should use Kubernetes features to dynamically update the domain or go through same process we did in the previous post to update the image and push new image to the OCIR and roll out changes to the cluster.

Architecture

Here is the high level architecture for this post.

In this architecture, I assume we have provisioned an Autonomous Transaction Processing (ATP) instance with private connectivity (has VNIC in private subnet and only accessible through this VNIC) and created the following schema and database table inside the ATP instance (we can extend our terraform script to include provisioning of ATP instance, network resources and download ATP instance wallet file).

variable "db_name" {  default = "APPATP" }
variable "db_display_name" { default = "APPATP"}
variable "db_autoscaling" { default = "false" }
variable "db_cpu_core_count" { default = "1" }
variable "db_storage_size" { default = "1" }
variable "db_version" { default = "19c" }
variable "db_license_model" { default = "LICENSE_INCLUDED" }
variable "db_admin_credential" { default = "<Password>" }

resource "oci_database_autonomous_database" "atp_instance" {
admin_password = var.db_admin_credential
compartment_id = var.compartment_ocid
cpu_core_count = var.db_cpu_core_count
data_storage_size_in_tbs = var.db_storage_size
db_name = var.db_name

#Optional
db_version = data.oci_database_autonomous_db_versions.autonomous_db_versions.autonomous_db_versions[0].version
db_workload = var.autonomous_database_db_workload
display_name = var.db_display_name
is_auto_scaling_enabled = var.db_autoscaling
is_auto_scaling_for_storage_enabled = var.db_autoscaling
license_model = var.db_license_model
character_set = "AL32UTF8"
ncharacter_set = "AL16UTF16"
subnet_id = var.db_subnet_ocid
nsg_ids = [var.db_nsg_ocid]
}

data "oci_database_autonomous_db_versions" "autonomous_db_versions" {
compartment_id = var.compartment_ocid
db_workload = var.autonomous_database_db_workload

filter {
name = "version"
values = [var.db_version]
}
}

resource "oci_database_autonomous_database_wallet" "atp_instance_wallet" {
autonomous_database_id = oci_database_autonomous_database.atp_instance.id
password = var.db_admin_credential
base64_encode_content = "true"
}

resource "local_file" "atp_instance_wallet_file" {
content_base64 = oci_database_autonomous_database_wallet.atp_instance_wallet.content
filename = "/tmp/appatp_wallet.zip"
}
CREATE USER UsersDB IDENTIFIED BY "<Schema Credential>";
GRANT CONNECT, RESOURCE TO UsersDB;
GRANT UNLIMITED TABLESPACE TO UsersDB;

CREATE TABLE UsersDB.Users
(
ID NUMBER(10,0) GENERATED BY DEFAULT ON NULL AS IDENTITY,
USERNAME VARCHAR2(50) NOT NULL,
FIRST_NAME VARCHAR2(50) NOT NULL,
MIDDLE_NAME VARCHAR2(50),
LAST_NAME VARCHAR2(50) NOT NULL,
AGE NUMBER(5,0) DEFAULT 0 NOT NULL,
CREATED_ON TIMESTAMP(9) NOT NULL,
CONSTRAINT TEST_PK PRIMARY KEY
(
ID
)
ENABLE
);

Also, we need firewall rules in the database subnet security list of database network security group (NSG) to allow access to database instance on port 1522 from pod and worker node subnets.

Implementation

Now I will explain the automation scripts (Ansible scripts) to add new data source to existing WebLogic domain and deploy our new application into a WebLogic cluster to list records inside the ATP table.

Note: you need wallet to access ATP instance. This wallet is an archive file that you can download from ATP instance (I already included the terraform code to download wallet) and includes all necessary files (sqlnet.ora, tnsname.ora, keystore, etc.) to access ATP instance.

To add a new data source to the existing WebLogic domain (deployed in Kubernetes) we can use two approaches.

  • Dynamic approach (without update auxiliary image).
  • Update auxiliary image and rollout changes into the cluster.

I will implement the second approach in my example and will only describe first approach.

Dynamic approach (without update auxiliary image)

In this approach, instead of updating the base auxiliary image, we will create a configmap inside Kubernetes cluster that includes datasource yaml file and put database credentials in a Kubernetes secret.

Here is the datasource.yaml file and codes to create configmap and secret in Kubernetes cluster.

resources:
JDBCSystemResource:
ATPDS:
Target: '##cluster_name##'
JdbcResource:
DatasourceType: GENERIC
JDBCConnectionPoolParams:
TestTableName: SQL ISVALID
JDBCDataSourceParams:
JNDIName: ##jndi_name##
JDBCDriverParams:
DriverName: oracle.jdbc.OracleDriver
PasswordEncrypted: '@@SECRET:##atp_db_credential_secret##:password@@'
URL: '@@SECRET:##atp_db_credential_secret##:url@@'
Properties:
oracle.net.tns_admin:
Value: '##atp_wallet_path##'
user:
Value: '@@SECRET:##atp_db_credential_secret##:user@@'
oracle.net.wallet_location:
Value: '##atp_wallet_path##'
oracle.jdbc.fanEnabled:
Value: false
oracle.net.ssl_version:
Value: 1.2
oracle.net.ssl_server_dn_match:
Value: true

You can update cluster name, jndi name, wallet path and secret name before creating configmap in Kubernetes.

Here is the code to create configmap and label it with WebLogic domain UID.

kubectl -n <WebLogic domain namespace> create configmap <domain name>-wdt-config-map \
--from-file=<path>/datasource.yaml

kubectl -n <WebLogic domain namespace> label configmap <domain name>-wdt-config-map \
weblogic.domainUID=<domain name>

and the code to create secret that hold ATP user, credential, and connection URL.

kubectl -n <WebLogic domain namespace> create secret generic \
<domain name>-atp-secret \
--from-literal='user=<ATP Schema user>' \
--from-literal='password=<ATP Schema Credential>' \
--from-literal='url=<ATP Connection URL for exmple jdbc:oracle:thin:@appatp_tp>'

kubectl -n <WebLogic domain namespace> label secret \
<domain name>-atp-secret \
weblogic.domainUID=<domain name>

OK, one big issue now: How can I transfer ATP wallet file to WebLogic pods? I am going to use the Kubernetes secret and attach the secret as a volume to server pods in my case.

First, you need to create a secret in your Kubernetes cluster that includes all wallet files content in Base64 format as below.

apiVersion: v1
kind: Secret
metadata:
name: atp-wallet-secret
namespace: <domain name>
type: Opaque
data:
ojdbc.properties: `<Base64 format of ojdbc.properties content>`
tnsnames.ora: `<Base64 format of tnsnames.ora content>`
sqlnet.ora: `<Base64 format of sqlnet.ora content>`
cwallet.sso: `<Base64 format of cwallet.sso content>`
ewallet.p12: `<Base64 format of ewallet.p12 content>`
keystore.jks: `<Base64 format of keystore.jks content>`
truststore.jks: `<Base64 format of truststore.jks content>`

You can use shell script to read wallet files content and convert it to Base64 format, for example:

 #!/bin/bash
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: atp-wallet-secret
namespace: <domain name>
type: Opaque
data:
ojdbc.properties: `cat <path>/ojdbc.properties | base64 -w0`

And finally, you need to update the domain yaml file to include configmap, secret and attach wallet secret as volume to the server pods.

apiVersion: "weblogic.oracle/v9"
kind: Domain
metadata:
name: <domain name>
namespace: <domain namespace>
labels:
weblogic.domainUID: <domain name>

spec:
configuration:
secrets:
- <domain name>-atp-secret
model:
configMap: <domain name>-wdt-config-map
apiVersion: "weblogic.oracle/v1"
kind: Cluster
metadata:
name: <cluster name>
namespace: <domain name>
labels:
weblogic.domainUID: <domain name>

spec:
replicas: 1
clusterName: <vluster name>
serverPod:
volumes:
- name: atp-wallet-secret
secret:
secretName: atp-wallet-secret
volumeMounts:
- mountPath: /u01/shared/atpwallet
name: atp-wallet-secret

and update domain configuration inside the cluster.

kubectl apply -f domain.yaml

Keep in mind, you should use ‘/u01/shared/atpwallet’ path as the wallet location in data source configmap.

Now you need to force the WebLogic operator to roll out these changes to the cluster by changing spec.restartVersion. Update the value of restartVersion and rerun its introspector job to regenerate its configuration and this will pass the configuration changes found by the introspector to each restarted server.

With this command, you can find the current value of restartVersion.

kubectl -n <domain namespace> get domain <domain name> '-o=jsonpath={.spec.restartVersion}'

Update the value using following command.

kubectl -n <domain namespace> patch domain <domain name> -type=json '-p=[{"op": "replace", "path": "/spec/restartVersion", "value": "<new value>" }]'

Update auxiliary image and rollout changes in to the cluster.

I recommend the second approach and automated this process here. In this solution, I will regenerate auxiliary image to include data source (and all dependencies like the secret) and the new application and roll out a new image to the cluster.

These are the steps we need to follow:

  • Update original domain model file and add data source part and new application to deploy.
  • Update domain properties file to include new data source variables.
  • Add ATP wallet and new application archive file to the domain archive file.
  • Create Kubernetes secret that contains ATP user, credential, and database connection URL.
  • Create new auxiliary image using source domain files.
  • Push auxiliary image to OCIR (use new tag version).
  • Roll out changes to Kubernetes cluster to restart WebLogic domain using new image.
  • Update ingress route to include new application path.

We need to update our on-premises domain model file to include data source and new application sections and update Kubernetes domain definition to include ATP database secret. I have used the following python script (using PayYAML library to update yaml files).

import yaml
import sys

from yaml.loader import SafeLoader


def read_yaml(filename):
with open(f'{filename}.yaml','r') as f:
output = list(yaml.load_all(f, Loader=SafeLoader))
return output

def write_yaml(filename, domain):
with open(f'{filename}.yaml','w') as f:
output = yaml.dump_all(domain, f, sort_keys=False)

def updateSourceDomain():
domain = read_yaml('onprem')
ds = read_yaml('datasource')
app = read_yaml('application')

atpDS = ds[0]["resources"]["JDBCSystemResource"]
appConfig = app[0]["appDeployments"]["Application"]

if "resources" in domain[0]:
if "JDBCSystemResource" in domain[0]["resources"]:
domain[0]["resources"]["JDBCSystemResource"].update(atpDS)
else:
domain[0]["resources"].update(ds[0]["resources"]["JDBCSystemResource"])
else:
domain[0].update(ds[0])


if "appDeployments" in domain[0]:
if "Application" in domain[0]["appDeployments"]:
domain[0]["appDeployments"]["Application"].update(appConfig)
else:
domain[0]["appDeployments"].update(ds[0]["appDeployments"]["Application"])
else:
domain[0].update(app)

write_yaml("onprem", domain)

def addSecretsToWKODomain(secretName):
domain = read_yaml('domain')
i = 0
for item in domain:
if item['kind']=="Domain":
if "spec" in item:
if "configuration" in item["spec"]:
secretExists = False
if "secrets" in item["spec"]["configuration"]:
for it in domain[i]["spec"]["configuration"]["secrets"]:
if it==secretName:
secretExists=True
break
if not secretExists:
domain[i]["spec"]["configuration"]["secrets"].append(secretName)
else:
domain[i]["spec"]["configuration"]["secrets"]=[secretName]
i=i+1
write_yaml("domain",domain)


numArgs = len(sys.argv)
secretName = ""
if numArgs<=1:
print("Command usage: updateSourceDomainDatasourceAndApplication.py [db-secret-name]")
exit()
else:
secretName = sys.argv[1]

updateSourceDomain()
addSecretsToWKODomain(secretName)

This script reads four yaml files from filesystem (on-premises model, data source configuration, new application configuration and Kubernetes domain configuration) and receives the ATP database secret name as a parameter and returns an on-premises model file and Kubernetes domain configuration as output.

You can see the updated part of our on-premises model file and Kubernetes domain configuration after executing the script following (script will execute by Ansible as I will explain later).

apiVersion: weblogic.oracle/v9
kind: Domain
metadata:
name: app-domain
namespace: app-domain
labels:
weblogic.domainUID: app-domain
spec:
configuration:
secrets:
- appatp-db-credential-secret
model:
auxiliaryImages:
appDeployments:
Application:
sample-app:
SourcePath: wlsdeploy/applications/sample-app.war
ModuleType: war
Target: app-cluster
SecurityDDModel: DDOnly
StagingMode: stage
SimpleApp:
SourcePath: wlsdeploy/applications/SimpleApp.war
ModuleType: war
Target: app-cluster
SecurityDDModel: DDOnly
StagingMode: stage
resources:
JDBCSystemResource:
atpds:
Target: app-cluster
JdbcResource:
DatasourceType: GENERIC
JDBCConnectionPoolParams:
TestTableName: SQL ISVALID
JDBCDataSourceParams:
JNDIName: jdbc/atpds
JDBCDriverParams:
DriverName: oracle.jdbc.OracleDriver
PasswordEncrypted: '@@SECRET:appatp-db-credential-secret:password@@'
URL: '@@SECRET:appatp-db-credential-secret:url@@'
Properties:
oracle.net.tns_admin:
Value: wlsdeploy/dbWallets/appatp
user:
Value: '@@SECRET:appatp-db-credential-secret:user@@'
oracle.net.wallet_location:
Value: wlsdeploy/dbWallets/appatp
oracle.jdbc.fanEnabled:
Value: false
oracle.net.ssl_version:
Value: 1.2

Finally, I am using the following Ansible code to create a Kubernetes secret, update domain archive (add new application artifact and ATP wallet files), update on-premises domain model, update Kubernetes domain configuration, generate a new auxiliary image and push it to OCIR and finally update domain in Kubernetes cluster.

#Copy datasource.yaml file to the server
- name: Copy datasource.yaml file to the server
copy:
src: ../files/datasource.yaml
dest: "/home/{{user}}/"
owner: "opc"
mode: '0755'

#Copy application.yaml file to the server
- name: Copy application.yaml file to the server
copy:
src: ../files/application.yaml
dest: "/home/{{user}}/"
owner: "opc"
mode: '0755'

- name: Copy onprem domain files to the server
copy:
src: "{{ item }}"
dest: "/home/{{user}}/"
owner: "opc"
mode: '0755'
with_fileglob:
- "/tmp/onprem*"

- name: Copy ATP Wallet archive to the server
copy:
src: "../files//{{atp_wallet_file}}"
dest: "/home/{{user}}/"
owner: "opc"
mode: '0755'

#Create ATP wallet folder in server
- name: Create ATP wallet folder in server
ansible.builtin.file:
path: "/home/{{user}}/{{atp_wallet_folder}}"
owner: "opc"
state: directory
mode: '0755'

#Create wallet folder structure
- name: Create wallet folder structure
file:
path: "/home/{{user}}/dbWallets/{{atp_wallet_folder}}"
owner: "opc"
state: directory
mode: '0755'

#Unzip ATP Wallet archive on the server
- name: Unzip ATP Wallet archive on the server
unarchive:
src: "/home/{{user}}/{{atp_wallet_file}}"
dest: "/home/{{user}}/dbWallets/{{atp_wallet_folder}}"
remote_src: yes

#Unzip domain archive on the server
- name: Unzip domain archive on the server
unarchive:
src: "/home/{{user}}/{{onprem_domain_archive_file}}"
dest: "/home/{{user}}/"
remote_src: yes

#Copy new application archive to server
- name: "Copy new application archive to server"
copy:
src: "../files//{{app_archive}}"
dest: "/home/{{user}}/wlsdeploy/applications"
owner: "opc"
mode: "0755"

#Copy wallet folder to archive structure
- name: "Copy wallet folder to archive structure"
copy:
src: "/home/{{user}}/dbWallets"
dest: "/home/{{user}}/wlsdeploy/"
owner: "opc"
mode: "0755"
remote_src: true

#Archive onprem artifacts
- name: "Archive onprem artifacts"
archive:
path: "/home/{{user}}/wlsdeploy"
dest: "/home/{{user}}/{{onprem_domain_archive_file}}"
force_archive: true
format: "zip"

#Copy wallet-secret.sh to server
- name: "Copy wallet-secret.sh file to the server"
copy:
src: "../files//wallet-secret.sh"
dest: "/home/{{user}}"
owner: "opc"
mode: "0755"

#Update wallet-secret.sh variables
- name: Update wallet-secret.sh variables in wallet-secret.sh
ansible.builtin.replace:
path: "/home/{{user}}/wallet-secret.sh"
regexp: '##{{item.key}}##'
replace: '{{item.value}}'
with_dict:
- "{{wallet_secret_variables}}"

#Execute wallet-secret.sh on the server
- name: Execute wallet-secret.sh on the server
shell: "/home/{{user}}/wallet-secret.sh"
environment:
OCI_CLI_AUTH: "{{ oci_auth }}"

#Update datasource.yaml variables
- name: Update data source variables in datasource.yaml
ansible.builtin.replace:
path: "/home/{{user}}/datasource.yaml"
regexp: '##{{item.key}}##'
replace: '{{item.value}}'
with_dict:
- "{{datasource_variables}}"

#Copy Python script to server
- name: Copy Python script to update in-premises domain model file (update datasource and application)
copy:
src: "../files/{{ python_script }}"
dest: "/home/{{user}}/"
owner: "opc"
mode: '0755'

#Update application.yaml variables
- name: Update application variables in application.yaml
ansible.builtin.replace:
path: "/home/{{user}}/application.yaml"
regexp: '##{{item.key}}##'
replace: '{{item.value}}'
with_dict:
- "{{ application_variables }}"

#Create ATP datasource secret
- name: "Create ATP datasource secret"
shell: "kubectl create secret generic \
{{item.atp_db_credential_secret}} \
--from-literal='user={{item.atp_db_user}}' \
--from-literal='password={{item.atp_db_credential}}' \
--from-literal='url={{item.atp_url}}' \
-n {{domain_namespace}}"
ignore_errors: true
loop:
- "{{datasource_variables }}"
environment:
OCI_CLI_AUTH: "{{ oci_auth }}"

#Lable ATP datasource secret with weblogic-operator=enabled
- name: "Lable ATP datasource secret with weblogic-operator=enabled"
shell: "kubectl label secret {{item.atp_db_credential_secret}} -n {{ domain_namespace }} weblogic.domainUID={{ domain_namespace }}"
ignore_errors: true
loop:
- "{{ datasource_variables }}"
environment:
OCI_CLI_AUTH: "{{ oci_auth }}"

#Update image version in domain.yaml file
- name: Update image version in domain.yaml file
ansible.builtin.replace:
path: "/home/{{user}}/domain.yaml"
regexp: '{{ocir_url}}\/{{tenancy}}\/{{ocir_repository}}\/{{domain_name}}:.+'
replace: '{{ocir_url}}/{{tenancy}}/{{ocir_repository}}/{{domain_name}}:{{image_version}}'

#Update onprem domain model file (add datasource and new application and add server to WKO domain)
- name: Update onprem domian model
command:
argv:
- python3
- "/home/{{user}}/{{ python_script }}"
- "{{datasource_variables.atp_db_credential_secret}}"

#Update cluster name in source domain
- name: Update cluster name in source domain
ansible.builtin.replace:
path: "/home/{{user}}/onprem.yaml"
regexp: '{{source_cluster_name}}'
replace: '{{cluster_name}}'

#Add weblogic-deploy.zip to image tool cache
- name: Add weblogic-deploy.zip to image tool cache
shell: /home/{{user}}/imagetool/bin/imagetool.sh cache addInstaller --type wdt --version latest --path "/home/{{user}}/weblogic-deploy.zip"
args:
executable: /bin/bash
environment:
JAVA_HOME: "/home/{{user}}/jdk{{ jdk_version }}"

#Create Auxiliary Image Image from onprem domain
- name: Create Auxiliary Image from onprem domain
shell: /home/{{user}}/imagetool/bin/imagetool.sh createAuxImage --tag "{{domain_name}}:{{image_version}}" --wdtModel "/home/{{user}}/onprem.yaml" --wdtArchive "/home/{{user}}/onprem.zip" --wdtVariables "/home/{{user}}/onprem.properties" --wdtHome "/auxiliary" --wdtModelHome "/auxiliary/models" --wdtVersion "latest"
args:
executable: /bin/bash
environment:
JAVA_HOME: "/home/{{user}}/jdk{{ jdk_version }}"

#Login to OCIR
- name: Login to OCIR
shell: docker login -u "{{ocir_user}}" -p "{{ocir_password}}" "{{ocir_url}}"
args:
executable: /bin/bash

#tag Image with OCIR
- name: tag image
shell: docker tag "localhost/{{domain_name}}:{{image_version}}" "{{ocir_url}}/{{tenancy}}/{{ocir_repository}}/{{domain_name}}:{{image_version}}"
args:
executable: /bin/bash

#Puch image to OCIR
- name: Push image to OCIR
shell: docker push "{{ocir_url}}/{{tenancy}}/{{ocir_repository}}/{{domain_name}}:{{image_version}}"
args:
executable: /bin/bash

#Deploy WebLogic domain in OKE
- name: "Deploy WebLogic domain in OKE"
shell: "kubectl apply -f /home/{{user}}/domain.yaml"
environment:
OCI_CLI_AUTH: "{{ oci_auth }}"

and here are the Ansible variables.

jdk_filename: jdk-8u371-linux-x64.tar.gz
python_script: updateSourceDomainDatasourceAndApplication.py
jdk_version: 1.8.0_371
user: opc
atp_wallet_file: appatp_wallet.zip
atp_wallet_folder: appatp

onprem_domain_archive_file: onprem.zip
app_archive: SimpleApp.war

oci_auth: instance_principal

ocir_url: <region code>.ocir.io
ocir_user: <OCIR Urer>
ocir_password: <User auth token>
ocir_repository: <OCIR Repo name>
ocir_secret: ocirsecret
tenancy: <Tenancy Name>

domain_namespace: app-domain
domain_name: app-domain
domain_home: /u01/data/domains/app-domain
cluster_name: app-cluster
source_cluster_name: apps_cluster
image_version: v3

datasource_variables:
cluster_name: "app-cluster"
jndi_name: "jdbc/atpds"
atp_db_credential_secret: "appatp-db-credential-secret"
atp_url: "<ATP Database connection string>"
atp_wallet_path: "wlsdeploy/dbWallets/appatp"
atp_db_user: "<ATP database schema name>"
atp_db_credential: "<ATP database schema credential>"

application_variables:
app_archive_file: "SimpleApp.war"
cluster_name: "app-cluster"

wallet_secret_variables:
atp_wallet_secret: "atp-wallet-secret"
domain_namespace: "app-domain"
user: "opc"
walet_folder: "appatp"

Conclusion

In this blog post, I explained how to update an existing WebLogic domain deployed in a Kubernetes environment by adding a new data source and deploying a new application to the WebLogic cluster. In next post I will explain how to enable SSL and monitor our environment (create a monitoring dashboard and view logs). Stay tuned!

References

WebLogic Operator: https://oracle.github.io/weblogic-kubernetes-operator/

PyYAML: https://pyyaml.org/wiki/PyYAMLDocumentation

OCI Autonomous Transaction Processing database: https://docs.oracle.com/en/cloud/paas/atp-cloud/index.html

--

--

Omid Izadkhasti
Oracle Developers

Principal Cloud Solution Architect @Oracle. The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.