Oracle Coherence
Published in

Oracle Coherence

Hello Coherence, Part 3

Deploy Coherence application to a Kubernetes cluster, and then scale, monitor and manage that cluster

Figure 1. A user-friendly interface for the sample to-do list application

Creating a Docker image

<profiles>
<profile>
<id>docker</id>
<build>
<plugins>
<plugin>
<groupId>com.google.cloud.tools</groupId>
<artifactId>jib-maven-plugin</artifactId>
<version>2.7.1</version>
<configuration>
<from>
<image>
gcr.io/distroless/java:11
</image>
</from>
<to>
<image>${project.artifactId}</image>
<tags>
<tag>${project.version}</tag>
</tags>
</to>
<container>
<ports>
<port>1408</port>
<port>7001</port>
</ports>
</container>
<containerizingMode>packaged</containerizingMode>
</configuration>
<executions>
<execution>
<goals>
<goal>dockerBuild</goal>
</goals>
<phase>package</phase>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
[INFO] --- jib-maven-plugin:2.7.1:dockerBuild (default) @ todo-list-server ---
[INFO]
[INFO] Containerizing application to Docker daemon as todo-list-server, todo-list-server:20.12...
[INFO] Using base image with digest: sha256:b25c7a4f771209c2899b6c8a24fda89612b5e55200ab14aa10428f60fd5ef1d1
[INFO]
[INFO] Container entrypoint set to [java, -cp, /app/classpath/*:/app/libs/*, io.helidon.microprofile.cdi.Main]
[INFO]
[INFO] Built image to Docker daemon as todo-list-server, todo-list-server:20.12

Deploying the application to Kubernetes

$ kubectl get deploymentsNAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
coherence-operator-controller-manager 1/1 1 1 31d
apiVersion: coherence.oracle.com/v1
kind: Coherence
metadata:
name: todo-list
spec:
replicas: 1
image: todo-list-server:latest
application:
type: helidon
jvm:
memory:
heapSize: 2g
ports:
- name: grpc
port: 1408
- name: http
port: 7001
serviceMonitor:
enabled: true
  • The values for the apiVersion and kind attributes reference a custom CRD that was installed by Coherence Operator and trigger the processing of the Coherence deployment resource above by Coherence Operator.
  • The metadata/name attribute is the only other required piece of information, and it is used as a Kubernetes identifier for this Coherence deployment. It is also used as the name of the Coherence cluster, unless it is overridden via the spec/cluster attribute.
  • The replicas attribute defines the number of Coherence cluster members to start.
  • The image attribute tells Kubernetes to use the Docker image created earlier (and should be modified accordingly if you tagged the image and pushed it to a Docker repo).
  • The jvm/memory/heapSize attribute tells Coherence Operator to set the initial and maximum heap size for each cluster member to 2 GB.
  • The application/type attribute tells Coherence Operator that this is a Helidon application, which results in the default Helidon main class being used to start each cluster member, instead of the DefaultCacheServer class that would normally be used in a standalone Coherence deployment.
  • Finally, the resource exposes ports 1408 and 7001 to serve the gRPC and HTTP/REST endpoints, respectively, via Kubernetes services todo-list-grpc and todo-list-http, based on the default service naming convention of appending the port name to the Coherence deployment name, and the resource tells Prometheus to scrape the metrics from each cluster member over the HTTP port by enabling the service monitor.
$ kubectl get coherenceNAME        CLUSTER     ROLE        REPLICAS   READY   PHASE
todo-list todo-list todo-list 1 1 Ready
$ kubectl get allNAME                          READY   STATUS    RESTARTS   AGE
pod/todo-list-0 1/1 Running 0 8m52s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
service/todo-list-grpc ClusterIP 10.96.13.213 <none> 1408/TCP
service/todo-list-http ClusterIP 10.96.89.213 <none> 7001/TCP
service/todo-list-sts ClusterIP None <none> 7/TCP
service/todo-list-wka ClusterIP None <none> 7/TCP
NAME READY AGE
statefulset.apps/todo-list 1/1 8m52s

Scaling the Coherence cluster

$ kubectl get coherenceNAME        CLUSTER     ROLE        REPLICAS   READY   PHASE
todo-list todo-list todo-list 10 10 Ready
$ kubectl get allNAME READY STATUS RESTARTS AGE
pod/todo-list-0 1/1 Running 0 46m
pod/todo-list-1 1/1 Running 0 2m56s
pod/todo-list-2 1/1 Running 0 2m55s
pod/todo-list-3 1/1 Running 0 2m55s
pod/todo-list-4 1/1 Running 0 2m55s
pod/todo-list-5 1/1 Running 0 2m55s
pod/todo-list-6 1/1 Running 0 2m55s
pod/todo-list-7 1/1 Running 0 2m55s
pod/todo-list-8 1/1 Running 0 2m55s
pod/todo-list-9 1/1 Running 0 2m55s
NAME READY AGE
statefulset.apps/todo-list 10/10 46m

Accessing the application

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: http-ingress
spec:
rules:
- host: tasks.seovic.com
http:
paths:
- backend:
serviceName: todo-list-http
servicePort: 7001
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: grpc-ingress
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
spec:
rules:
- host: grpc.tasks.seovic.com
http:
paths:
- backend:
serviceName: todo-list-grpc
servicePort: 1408

Configuring data persistence across restarts

  • Write data automatically to some kind of persistent datastore, such as a relational database or key-value store, and load data from it when necessary. This is certainly possible via the Coherence cache store mechanism, but it is less than ideal for this application because it requires setting up an external datastore. That negates some of the benefits and simplicity of the stateful application you created.
  • Enable Coherence persistence and configure the application to attach a persistent disk volume to each pod within the stateful set backing your deployment. This way Coherence itself will persist all the data to a disk volume that is managed by the cloud provider and will be reattached to the correct pod upon restart. This is the option that’s best for this project, and that’s what I’ll discuss next.
spec:
# insert before jvm section
coherence:
persistence:
mode: active
persistentVolumeClaim:
storageClassName: "oci-bv"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
$ kubectl delete -f java/server/src/main/k8s/app.yamlcoherence.coherence.oracle.com "todo-list" deleted$ kubectl apply -f java/server/src/main/k8s/app.yamlcoherence.coherence.oracle.com/todo-list created
$ kubectl get pvcNAME                             STATUS   VOLUME  CAPACITY  STORAGECLASS
persistence-volume-todo-list-0 Bound csi-… 50Gi oci-bv
$ k scale coherence todo-list --replicas 10coherence.coherence.oracle.com/todo-list scaled$ kubectl get pvcNAME                             STATUS   VOLUME  CAPACITY  STORAGECLASS
persistence-volume-todo-list-0 Bound csi-… 50Gi oci-bv
persistence-volume-todo-list-1 Bound csi-… 50Gi oci-bv
persistence-volume-todo-list-2 Bound csi-… 50Gi oci-bv
persistence-volume-todo-list-3 Bound csi-… 50Gi oci-bv
persistence-volume-todo-list-4 Bound csi-… 50Gi oci-bv
persistence-volume-todo-list-5 Bound csi-… 50Gi oci-bv
persistence-volume-todo-list-6 Bound csi-… 50Gi oci-bv
persistence-volume-todo-list-7 Bound csi-… 50Gi oci-bv
persistence-volume-todo-list-8 Bound csi-… 50Gi oci-bv
persistence-volume-todo-list-9 Bound csi-… 50Gi oci-bv
persistence-volume:
Type: PersistentVolumeClaim
ClaimName: persistence-volume-todo-list-0
ReadOnly: false

Configuring observability and tracing

  • Monitoring metrics, exposed via Java Management Extensions and an OpenMetrics-compliant HTTP endpoint that can be consumed by tools such as Prometheus
  • Built-in Grafana dashboards that can be used to visualize those metrics
  • Tracing information via OpenTracing that allows you to better understand the flow of individual requests through the system, which can help you understand and fix performance bottlenecks
<dependency>
<groupId>io.helidon.tracing</groupId>
<artifactId>helidon-tracing-jaeger</artifactId>
</dependency>
spec:
# insert before 'application' section
env:
- name: TRACING_HOST
value: "jaeger-collector"
- name: JAEGER_SAMPLER_TYPE
value: "const"
- name: JAEGER_SAMPLER_PARAM
value: "1"
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: jaeger-ingress
spec:
rules:
- host: jaeger.seovic.com
http:
paths:
- backend:
serviceName: jaeger-query
servicePort: 16686
Figure 2. Querying for all tasks
Figure 3. Creating a task
Figure 4. Updating a task

Monitoring with Prometheus and Grafana

Figure 5. Grafana’s cluster overview dashboard
Figure 6. The Member Details dashboard for todo-list-0
Figure 7. The Service Summary dashboard
Figure 8. The Service Details dashboard
Figure 9. The Cache Details dashboard
Figure 10. The Persistence Summary dashboard

Conclusion

Figure 11. A 1,000-member Coherence cluster

--

--

Oracle Coherence is the industry leading in-memory data grid solution that enables organizations to predictably scale stateful workloads. Often immitated, but never duplicated, it is now available for everyone to use free of charge.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Aleks Seovic

Father of three, husband; Coherence Architect @ Oracle; decent tennis player, average golfer; sailor at heart, trapped in a power boat