Google Managed Prometheus for JVM
前面幾篇[Ref1, Ref2, Ref3] 我們介紹了幾種GMP的部署方法，這裡我們來看一下該如何可以使用GMP來監控JVM的Memory使用狀態，讓Java開發者可以更了解目前記憶體的相關分配狀況。
本人對於JVM的記憶體分配，也是一知半解，所以爬文學習了一下，大概了解分為Heap & No Heap兩類，也有蠻多有趣的名稱像EDEN, Survivor, Tenured for Heap, 以及Metaspace for non-heap. 詳情請見 Understanding Java Memory Model and JVM Technology。
開發環境介紹：
坊間有許多的Java WebServer，這裡我們使用捧由提問的Wildfly (formerly known as Jboss) 作為開發Servlet。完整的Dockerfile如下，主要分為兩大部分：
# Multi-Stage Build: Compile Helloworld
FROM openjdk:17.0.2-jdk as builder
# Set the WILDFLY_VERSION env variable
ENV PATH /usr/local/bin/apache-maven-3.9.1/bin:${PATH}
WORKDIR /workspace
RUN curl -o apache-maven.tar.gz https://dlcdn.apache.org/maven/maven-3/3.9.1/binaries/apache-maven-3.9.1-bin.tar.gz && \
tar zxvf apache-maven.tar.gz -C /usr/local/bin/
ADD . /workspace
RUN mvn package
# JMX_Exporter Agent with Wildfly Standalone mode
FROM quay.io/wildfly/wildfly:27.0.0.Final-jdk17
WORKDIR /opt/jboss/wildfly
RUN bin/add-user.sh -m -u admin -p password
USER jboss
COPY jmx_agent.jar /opt/jboss/wildfly/bin/jmx_agent.jar
COPY wildfly-10.yaml /opt/jboss/wildfly/standalone/configuration/wildfly-10.yaml
COPY standalone.conf /opt/jboss/wildfly/bin/
COPY --from=builder /workspace/target/helloworld.war /opt/jboss/wildfly/standalone/deployments/helloworld.war
CMD ["bin/standalone.sh", "-b", "0.0.0.0", "-bmanagement", "0.0.0.0"]HelloWorld with Wildfly:
由Wildfly Github的Quickstart下載HelloWorld的範例，詳細說明見此。在我們提供的專案中，選擇了
(1) 使用Wildfly Standalone模式，並設定登入Wildfly Management Server帳密admin/password
(2) 使用WebServelet Annotation定義了Wildfly API的Subpath為Wildfly。
(3) MultiStage Built，將編譯出的War File拷貝到Wildfly的Base Image中。
為了加上GMP可以收集的Wildfly Metrics，我們使用JMX_Exporter，發佈於Wildfly Standalone的啟動設定中：
## -*- shell-script -*- ######################################################
## ##
## WildFly bootstrap Script Configuration ##
## ##
##############################################################################
if [ "x$JBOSS_MODULES_SYSTEM_PKGS" = "x" ]; then
JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,org.jboss.logmanager"
fi
#
# Specify options to pass to the Java VM.
#
if [ "x$JBOSS_JAVA_SIZING" = "x" ]; then
JBOSS_JAVA_SIZING="-Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m"
fi
if [ "x$JAVA_OPTS" = "x" ]; then
# logmanager jar (update jar version as needed)
JBOSS_LOG_MANAGER_LIB="$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.1.19.Final.jar"
# wildfly-common jar (update jar version as needed)
WILDFLY_COMMON_LIB="$JBOSS_HOME/modules/system/layers/base/org/wildfly/common/main/wildfly-common-1.6.0.Final.jar"
JAVA_OPTS="$JBOSS_JAVA_SIZING -Djava.net.preferIPv4Stack=true"
JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/a:$JBOSS_LOG_MANAGER_LIB:$WILDFLY_COMMON_LIB -Dsun.util.logging.disableCallerCheck=true -Dorg.wildfly.logging.skipLogManagerCheck=true"
JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true"
# set jmx_agent to report metrics at port 9991
JAVA_OPTS="$JAVA_OPTS -javaagent:$JBOSS_HOME/bin/jmx_agent.jar=9991:$JBOSS_HOME/standalone/configuration/wildfly-10.yaml"
else
echo "JAVA_OPTS already set in environment; overriding default settings with values: $JAVA_OPTS"
fi(1) 通過JAVA_OPTS的設定，以-javaagent通過premain的方式，在啟動主程式main 之前發布Metric服務
(2) 將Metrics發佈於port 9991
(3) 使用Wildfly社群提供的預設值，發布Metrics格式(wildfly-10.yaml)
編譯與部署服務：
我們通過cloudbuild.yaml 完成一站式的CI/CD流程。整個專案請見本連結。使用時，只用一行指令就可以將Wildfly HelloWorld + Metrics發布於指定的Anthos Cluster (底下變數：_CLUSTER)中，安裝Anthos Cluster的IaC文件請見本篇。
gcloud builds submit . --substitutions=_IMAGE_TAG="v1.00"steps:
# Build the container image
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'asia-east1-docker.pkg.dev/${_PROJECT}/gcf-artifacts/wildfly-hello:${_IMAGE_TAG}', '.']
# Push the container image to Container Registry
- name: 'gcr.io/cloud-builders/docker'
args: ['push', 'asia-east1-docker.pkg.dev/${_PROJECT}/gcf-artifacts/wildfly-hello:${_IMAGE_TAG}']
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: bash
id: Deploy to Anthos BM cluster
args:
- '-c'
- |
set -x && \
sed -i "s/_IMAGE_TAG/${_IMAGE_TAG}/g" kubernetes-manifests/hello-deployment.yaml && \
export KUBECONFIG="$(pwd)/gateway-kubeconfig" && \
gcloud container fleet memberships get-credentials ${_CLUSTER} && \
kubectl version && \
kubectl --kubeconfig gateway-kubeconfig apply -f kubernetes-manifests/
substitutions:
_IMAGE_TAG: latest # default value
_CLUSTER: cluster1
_PROJECT: shawnho-demo-2023
images:
- asia-east1-docker.pkg.dev/${_PROJECT}/gcf-artifacts/wildfly-hello:${_IMAGE_TAG}驗證結果：
我們通過Laptop來驗證Wildfly Standalone Server已經正常被執行。同時也可以通過K8s LoadBalancer取得網頁的回應與JMX_Exporter Metrics的相關數據。
視覺化量測結果(GMP):
這裡我們使用基於Kube-Prometheus所安裝出的GMP，我們可以通過設定ServiceMonitor or PodMonitor命令Prometheus定時對我們做出的服務進行數據採集。
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: wildfly-demo
spec:
namespaceSelector:
matchNames:
- default
endpoints:
- port: web-metrics
path: /
selector:
matchLabels:
k8s-app: hello-wildfly
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: wildfly-monitor
spec:
namespaceSelector:
matchNames:
- default
selector:
matchLabels:
app: hello-wildfly
podMetricsEndpoints:
- path: /
port: metrics
interval: 15s同時我們引用Grafana現有針對Jmx_exporter
- JBoss JMX：這個直接匯入Grafana就可以使用，無需任何修改。
- JMX Prometheus Exporter: 原始監控的JVM Memory Pool在JVM 9 & JVM 11，Pool的名稱有所改變（e.g. PS_EDEN_SPACE變成 EDEN_SPACE, CODE_CACHE甚至一個變成3個）。名稱的錯誤會造成Dashboard Query不到數據。依照修正掉這個之後，就可以正常看到Dashboard。我也將修改後的Dashboard提供給讀者使用。
