View OpenShift Router’s Log

Zhimin Wen
Oct 25 · 3 min read

To troubleshoot the Router of the OpenShift, the logs are essential, especially the HAProxy logs on how the traffic is handled and forwarded. However, the HAProxy pumping the log to a rsyslog server by default, the Pod log of the Router is not able to see it.

Instead of setting up a rsyslog server outside of the cluster, this paper explores a “cloud-native” way of checking the HAproxy logs for debugging purposes.

Identify the container

The image of rsyslog/syslog_appliance_alpine seems like a good fit.

Customize the config file, rsyslog.conf, as below

global(processInternalMessages="on")module(load="imrelp")
module(load="imptcp")
module(load="imudp" TimeRequery="500")
module(load="omstdout")
module(load="omelasticsearch")
module(load="mmjsonparse")
module(load="mmutf8fix")
input(type="imptcp" port="514")
input(type="imudp" port="514")
input(type="imrelp" port="1601")
syslog.* :omstdout:include(file="/config/droprules.conf" mode="optional") # this permits the user to easily drop unwanted messages
action(name="main_utf8fix" type="mmutf8fix" replacementChar="?")
# dump the haproxy log to console
local1.* :omstdout:

For debugging purposes, we emit the haproxy log to the stdout only.

Create a ConfigMap in the OpenShift cluster to host the config file.

oc new-project rsyslog
oc create cm cm-rsyslog-config --from-file=rsyslog.conf=rsyslog.conf

Create the deployment

Create the following deployment, apply it in the rsyslog namespace

apiVersion: apps/v1
kind: Deployment
metadata:
name: rsyslog
labels:
app: rsyslog
spec:
selector:
matchLabels:
app: rsyslog
template:
metadata:
labels:
app: rsyslog
spec:
securityContext:
runAsUser: 0
containers:
- name: rsyslog
image: rsyslog/syslog_appliance_alpine
ports:
- containerPort: 514
env:
- name: RSYSLOG_CONF
value: /myconfig/rsyslog.conf
volumeMounts:
- mountPath: /myconfig
name: rsyslog-config
- mountPath: /config
name: config
- mountPath: /work
name: work
- mountPath: /logs
name: logs
volumes:
- name: rsyslog-config
configMap:
name: cm-rsyslog-config
- name: config
emptyDir: {}
- name: work
emptyDir: {}
- name: logs
emptyDir: {}

Mount the volume for /config, /work, /logs with emptyDir. Mount the config file from the configMap. Use the environment variable to point to the config.

As it will listen on port 514, we need to assign the pod to run as root. Set the security context in the pod. In addition, assign the anyuid SCC (Security Context Constraints) to the default service account.

oc adm policy add-scc-to-user anyuid -z default

The deployment pod is running.

Expose service

Expose the rsyslog service within the cluster by using the default ClusterIP. Apply the following yaml file,

apiVersion: v1
kind: Service
metadata:
name: rsyslog
spec:
selector:
app: rsyslog
ports:
- name: udp
protocol: UDP
port: 514
targetPort: 514

Now the rsyslog will be available in the cluster at the address of rsyslog.rsyslog.svc

View Router’s log

Update the router’s environment setting,

oc -n default set env dc/router ROUTER_SYSLOG_ADDRESS=rsyslog.rsyslog.svc ROUTER_LOG_LEVEL=debug

When the router pod is restarted, the HAproxy’s log is available by check the rsyslog pod stdout,

oc -n rsyslog log rsyslog-675c46ff6f-hnrww -f

A sample is shown below,

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade