Open in app

Sign in

Write

Sign in

Software project: Creating a real-life transactional microservices application on Kubernetes

Mika Rinne, Oracle EMEA
9 min readJan 24, 2024

--

Part 7: Building and deploying to Kubernetes

During our previous parts we’ve finally reached the point when we can start deploying Gymapp to the cloud on Kubernetes with VS Code.

To deploy Gymapp to Kubernetes on Oracle Cloud Infrastructure (OCI) first let’s prepare the Oracle Kubernetes Engine (OKE) cluster with MicroTX for it. This is going to be a test cluster installation, not 100% ready for production to simplify the installation a bit and will explain how as we go.

I already mentioned the MicroTX installer in the Part 3 and now we are going to use it. The prereqs for this are:

  • Have a OKE cluster with 2 nodes at least up and running with kubectl and Helm access from the local dev; this requires a working oci cli connection to the OCI tenancy from local dev as well
  • Have Istio installed in the OKE cluster per the MicroTX documentation
  • SSL cert for a domain; I am using gymapp.devrocks.io domain that I have a SSL cert for
  • Downloaded Oracle MicroTX 23.4.1 on the local dev with JDK 21 and Maven working

This is how the Gymapp OKE env should look like for production:

Gymapp on Istio service mesh on Kubernetes with MicroTX and IDCS

This is basically the same picture as in the Part 6 describing the JWT setup on Istio and OKE, this time adding the MicroTX to it, too. As we can see the MicroTX calls inside the cluster are also protected by the JWT that can be setup with IDCS using the MicroTX installer Helm configuration.

However, since this is a test env I’m skipping this for simplicity and disabling the JWT from MicroTX calls in Helidon and in the MicroTX Helm installer configuration as follows.

Hence my MicroTX LRA config in application.yaml for the test env is as follows, note the default headers propagation commented out to omit the LRA authorization headers:

mp.lra:
  coordinator.url: http://otmm-tcs:9000/api/v1/lra-coordinator
  propagation.active: true
  participant.url: http://gymuser:8080
  coordinator.headers-propagation.prefix: ["x-b3-", "oracle-tmm-", "refresh-"]
  #coordinator.headers-propagation.prefix: ["x-b3-", "oracle-tmm-", "authorization", "refresh-"]

Actually I have separate config file from my local dev for this at src/main/resources/application-oke.yaml for the OKE env. There I also have the IDCS config for the test IDCS confidental app similar to what I covered in the part 6 but with empty values that for which the actual values are be passed using Kubernetes YAML when doing the deployment to OKE.

Here are the MicroTX Helm install settings in qs-oke-values.yaml to disable LRA authorization below, and hence also not needing to configure the IDCS into it to simplify the setup for OKE test env:

#Authorization settings
  authorization:
    enabled: "false"
    #This indicates the Authorization (Access) and Refresh-Token propagation is enabled/disabled.
    authTokenPropagationEnabled: "false"
    # Provide the identity provider details under identityProvider configuration

...

# Authentication using JWT token. Provide JWT token issuer url
authentication:
  requestsWithNoJWT: ALLOW # ALLOW or DENY
  # Provide the jwksUri and the issuer details under identityProvider configuration

For the domain gymapp.devrocks.io config I have configured it as the “hosts” in qs-oke-values.yaml:

istioIngressGateway:
  name: ingressgateway
  tlsEnabled: "true"
  # Kubernetes secret that contains the SSL and key certificate 
  # Run the below command to create the secret,
  # kubectl create -n istio-system secret tls tls-credential --key=<ssl key file path> --cert=<ssl cert file path>
  # example: kubectl create -n istio-system secret tls tls-credential --key=example.dev.key --cert=example.dev.crt
  credentialName: tls-credential
  hosts: [ "gymapp.devrocks.io" ]

Before running the MicroTX runme.sh installer against my OKE cluster I changed the istio_url to match my domain name gymapp.devrocks.io.

if [ $environment == 3 ]; then
    printf "${GREEN}\nCreating a self signed certificate to enable TLS connection to transaction coordinator."
    printf "${GREEN}\nEnter the system/keytool password when prompted. This is required to connect to transaction coordinator running on TLS and hence the certificates should be trusted."
    printf "${GREEN}\nPreceding command requires sudo access. On password prompt, please enter sudo user password.\n"
    #delete_certificates
    create_certificates
    istio_url="https://gym.devrocks.io:443"
fi

And since I have my own SSL cert I can comment out the delete_certificates for self-signed demo certs from above and replace the demo self-signed certs creation with the following kubectl command to use my own SSL cert instead:

#chmod +x $runkeytool
#sudo $runkeytool
#rm $runkeytool
kubectl create -n istio-system secret tls tls-credential --key=/Users/MRINNE/Downloads/gymapp.devrocks.io/private.key --cert=/Users/MRINNE/Downloads/gymapp.devrocks.io/certificate.crt

With these modifications the installer should be able to create a working MicroTX setup for the Gymapp on OKE by choosing the option 3 from it:

sh runme.sh
java version "11.0.21" 2023-10-17 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.21+9-LTS-193)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.21+9-LTS-193, mixed mode)
*************************

 Use this script to run microservices and get started with Oracle Transaction Manager for Microservices.

*************************

Enter a number (1-3) to specify the platform on which you want to run the microservice
1) Docker - only the transaction coordinator will run in a docker container and the sample microservices will run in the local environment
2) Minikube
3) Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE)
#?

Running thru the OKE install it will create MicroTX “otmm-tcs” pod into the “otmm” namespace in the OKE cluster with Istio below.

BTW, install the MicroTx with the Kiali enabled to be able to use it in the next part to configure the Istio Ingress routes for the Gymapp.

get pods -n otmm
NAME                            READY   STATUS    RESTARTS      AGE
otmm-tcs-0                      2/2     Running   0             49d

As part of the Istio install to the OKE cluster an OCI Load Balancer was created for the Istio ingress with a public IP (89.168.122.197 below):

kubectl get services -n istio-system
NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                                      AGE
istio-ingressgateway   LoadBalancer   10.96.157.172   89.168.122.197   15021:32482/TCP,80:31386/TCP,443:30556/TCP   56d

The next step is to run the build from VS Code using the OCI DevOps extension (VSIX) for which I covered the installation in the Part 1.

To create OCI DevOps project in OCI for Gymapp microservices click the OCI DEVOPS panel’s button “Create OCI DevOps Project” that will launch the setup Wizard in VS Code:

Wizards start by asking the OCI target compartment and then selecting the OKE cluster from selected compartment, project name and few other settings for the DevOps project to be created.

Doing this for both Gymapp microservices VS Code projects, Gymuser and Gyminstructor, separately and going thru all the steps for both in the Wizard two OCI DevOps projects are then created to OCI.

After this VS Code OCI DEVOPS panel shows two options, one for building the microservice and one for deploying it to OKE:

Clicking the “Build” button the OCI DevOps build pipeline is run:

Build and deployment (to OKE) pipelines can be fully customized, an automatic build trigger be added to build and deploy automatically after commit in VS Code etc., so that we are not limited to the default functionality that is created by the VSIX extension and it gives a very nice fast-forward to CI/CD with OCI DevOps.

Before building Gymapp there are a few things to do:

  • Modify the VSIX generated pipeline docker_jvmbuild_spec.yaml for Helidon 4 (by default VSIX supports Springboot and Micronaut, but Helidon 4 support is not there yet)
  • Modify the Dockerfile for Helidon 4

In the VSIX generated docker_jvmbuild_spec.yaml add the “JAVA_VERSION” “21” and build arg “LIBS” as follows:

env:
  variables:
    JAVA_VERSION : "21"
...
  docker build -f ./.devops/Dockerfile.jvm \
    --build-arg JAR_FILE=target/gymuser-helidon.jar \
    --build-arg LIBS=target/libs \
    -t gymuser-helidon-dev-jvm:${DOCKER_TAG} -t gymuser-helidon-dev-jvm:latest .

I have then modified the generated Dockerfile.jvm to add the “COPY $LIBS ./libs” and to use “openjdk:21-jdk” as the base image:

FROM openjdk:21-jdk

ARG JAR_FILE
ARG LIBS
EXPOSE 8080

COPY $JAR_FILE app.jar
COPY $LIBS ./libs

ENTRYPOINT ["java","-jar","/app.jar"]

The database connection for the XE database shown below as covered in the part 2 and is the same as in my local dev in src/resources/main/META-INF/microprofile-config.properties:

# Datasource properties
javax.sql.DataSource.gyumsers.dataSourceClassName=oracle.jdbc.pool.OracleDataSource
javax.sql.DataSource.gyumsers.dataSource.url=jdbc:oracle:thin:@localhost:1521:XE
javax.sql.DataSource.gyumsers.dataSource.user=sys as sysdba
javax.sql.DataSource.gyumsers.dataSource.password=WelcomeFolks123##

After making the adjustments above the OCI DevOps build pipeline runs for both projects and Gymapp microservices Docker images are created and pushed to OCIR (using the Frankfurt region) with tags “latest”:

fra.ocir.io/<tenancy namespace>/gymuser-helidon-dev-jvm:latest
fra.ocir.io/<tenancy namespace>/gyminstructor-helidon-dev-jvm:latest

Finally to deploy to OKE test env using OCI DevOps deployment pipeline I have modified the OKE Kubernetes manifest YAML artifact OCI DevOps project of both projects Gymuser and Gyminstructor, here’s the YAML for Gyumser Kubernetes deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gymuser
  namespace: otmm
  labels:
    app: gymuser
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gymuser
  template:
    metadata:
      labels:
        app: gymuser
    spec:
      containers:
      - name: gymuser-db
        imagePullPolicy: Always
        image: fra.ocir.io/<replace with tenancy namespace>/gym-db-xe:latest
        ports:
          - containerPort: 1521
        env:
          - name: ORACLE_PWD
            value: "WelcomeFolks123##"
      - name: gymuser-helidon
        imagePullPolicy: Always
        image: fra.ocir.io/<replace with tenancy namespace>/gymuser-helidon-dev-jvm:latest
        ports:
          - containerPort: 8080
        env:
          - name: security.properties.idcs-uri
            value: "https://idcs-b491bb4...161665.identity.oraclecloud.com:443"
          - name: security.properties.idcs-client-id
            value: "45b...59d2"
          - name: security.properties.idcs-client-secret
            value: "e1425...3b744"
          - name: idcs.scope-audience
            value: "https://gymapp.devrocks.io"
          - name: idcs.frontend-uri
            value: "https://gymapp.devrocks.io"
      imagePullSecrets:
      - name: gymuser-helidon-vscode-generated-ocirsecret

Using this YAML Gymapp microservices pods are deployed with the XE database image as a sidecar with the Istio sidecar:

kubectl get pods -n otmm
NAME                            READY   STATUS    RESTARTS        AGE
gyminstructor-7fbd58948-mmmmn   3/3     Running   1 (10m ago)     10m
gymuser-c89994f55-fghg6         3/3     Running   1 (9m46s ago)   10m
otmm-tcs-0                      2/2     Running   0               50d

To see logs for the Gymuser pod and it’s microservice container above:

kubectl logs gymuser-c89994f55-fghg6 -c gymuser-helidon -n otmm 
2024.01.24 10:44:02 INFO org.jboss.weld.Version Thread[#1,main,5,main]: WELD-000900: 5.1.1 (SP2)
2024.01.24 10:44:06 INFO io.helidon.microprofile.lra.LraCdiExtension Thread[#1,main,5,main]: Adding Jandex index at jar:file:/app.jar!/META-INF/jandex.idx
2024.01.24 10:44:06 INFO io.helidon.microprofile.lra.LraCdiExtension Thread[#1,main,5,main]: Adding Jandex index at jar:file:/libs/helidon-integrations-cdi-hibernate-4.0.0.jar!/META-INF/jandex.idx
2024.01.24 10:44:06 INFO io.helidon.microprofile.lra.LraCdiExtension Thread[#1,main,5,main]: Adding Jandex index at jar:file:/libs/helidon-integrations-cdi-jta-4.0.0.jar!/META-INF/jandex.idx
2024.01.24 10:44:06 INFO io.helidon.microprofile.lra.LraCdiExtension Thread[#1,main,5,main]: Adding Jandex index at jar:file:/libs/narayana-jta-7.0.0.Final.jar!/META-INF/jandex.idx
2024.01.24 10:44:06 INFO io.helidon.microprofile.lra.LraCdiExtension Thread[#1,main,5,main]: Adding Jandex index at jar:file:/libs/helidon-integrations-cdi-jpa-4.0.0.jar!/META-INF/jandex.idx
2024.01.24 10:44:06 INFO org.jboss.weld.Bootstrap Thread[#1,main,5,main]: WELD-ENV-000020: Using jandex for bean discovery
2024.01.24 10:44:09 INFO org.jboss.weld.Event Thread[#1,main,5,main]: WELD-000411: Observer method [BackedAnnotatedMethod] public org.glassfish.jersey.ext.cdi1x.internal.ProcessAllAnnotatedTypes.processAnnotatedType(@Observes ProcessAnnotatedType<?>, BeanManager) receives events for all annotated types. Consider restricting events using @WithAnnotations or a generic type with bounds.
2024.01.24 10:44:13 INFO org.hibernate.jpa.internal.util.LogHelper Thread[#1,main,5,main]: HHH000204: Processing PersistenceUnitInfo [name: gymusers]
2024.01.24 10:44:13 INFO org.hibernate.Version Thread[#1,main,5,main]: HHH000412: Hibernate ORM core version 6.3.1.Final
...
2024.01.24 10:44:24 INFO io.helidon.webserver.ServerListener VirtualThread[#33,start @default (/0.0.0.0:8080)]/runnable@ForkJoinPool-1-worker-1: [0x57ca5353] http://0.0.0.0:8080 bound for socket '@default'
2024.01.24 10:44:24 INFO io.helidon.webserver.LoomServer Thread[#1,main,5,main]: Started all channels in 11 milliseconds. 24011 milliseconds since JVM startup. Java 21+35-2513
2024.01.24 10:44:24 INFO io.helidon.microprofile.server.ServerCdiExtension Thread[#1,main,5,main]: Server started on http://localhost:8080 (and all other host addresses) in 24016 milliseconds (since JVM startup).
2024.01.24 10:44:24 INFO io.helidon.common.features.HelidonFeatures Thread[#36,features-thread,5,main]: Helidon MP 4.0.0 features: [CDI, Config, Health, JPA, JTA, Long Running Actions, Metrics, Security, Server]

To get the logs for the pod’s database sidecar (not the complete log):

kubectl logs gymuser-c89994f55-fghg6 -c gymuser-db -n otmm
Starting Oracle Net Listener.
Oracle Net Listener started.
Starting Oracle Database instance XE.
Oracle Database instance XE started.
The Oracle base remains unchanged with value /opt/oracle
SQL*Plus: Release 21.0.0.0.0 - Production on Wed Jan 24 10:43:59 2024
Version 21.3.0.0.0
Copyright (c) 1982, 2021, Oracle.  All rights reserved.
Connected to:
Oracle Database 21c Express Edition Release 21.0.0.0.0 - Production
Version 21.3.0.0.0

SQL> 
User altered.
...
Table created.
..
Sequence created.
..
1 row created.
1 row created.
1 row created.
...
DONE: Executing user defined scripts
The following output is now a tail of the alert.log:
...
XEPDB1(3):TABLE AUDSYS.AUD$UNIFIED: ADDED INTERVAL PARTITION SYS_P348 (3495) VALUES LESS THAN (TIMESTAMP' 2024-01-25 00:00:00')
2024-01-24T10:44:11.265121+00:00
===========================================================
Dumping current patch information
===========================================================
No patches have been applied
===========================================================
2024-01-24T10:44:12.180312+00:00
XEPDB1(3):Resize operation completed for file# 10, fname /opt/oracle/oradata/XE/XEPDB1/sysaux01.dbf, old size 337920K, new size 358400K
2024-01-24T11:03:39.903930+00:00
Resize operation completed for file# 3, fname /opt/oracle/oradata/XE/sysaux01.dbf, old size 573440K, new size 593920K

Before the Gymapp can be accessed using URL https://gymapp.devrocks.io/gymuser the OCI DNS needs to be set to use the OCI Load Balancer that was created as Istio ingress for the MicroTX during it’s installation and set the Gymapp routes for it.

Let’s do that in the Part 8 and also some basic performance testing for the Gymapp.

DevOps
Helidon
Vsix
Oci
Oke

--

--

Mika Rinne, Oracle EMEA

Written by Mika Rinne, Oracle EMEA

37 Followers

Appdev, Cloud Native and DevOps Specialist at Oracle

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams