Configuring Tomcat logging for a read-only file system in Kubernetes.
From this article, I will discuss about how to configure Tomcat server logging for a read-only file system in a container.
Recently I got the chance to work with Kubernetes deployment which will deploy a Tomcat web server, while deploying this application I faced an issue that is related to Tomcat server logging….Let me describe the issue to you. Before deploying the application I configured the container which will run the Tomcat web server as a read-only file system. Now, this configuration leads me to an error, which was Tomcat internally publishing its own logs to the /logs directory. Since the file system was configured as a read-only one, Tomcat was not able to publish the logs to the /logs directory, because of that my Tomcat deployment was failed. Luckily I found an answer to this problem.
From this article, we are going to discuss the following topics.
1. How Tomcat default loggings are configured.
2. How to redirect tomcat logs to console
3. Configuring the emptyDir volume mount
How Tomcat default loggings are configured.
In default mode tomcat loggings are configured as write to log files in {TOMCAT_HOME}/logs directory and to the console. The following file shows the default tomcat logging configuration defined in {TOMCAT_HOME}/conf/logging.properties file.
handlers = 1catalina.org.apache.juli.AsyncFileHandler, 2localhost.org.apache.juli.AsyncFileHandler, 3manager.org.apache.juli.AsyncFileHandler, 4host-manager.org.apache.juli.AsyncFileHandler, java.util.logging.ConsoleHandler.handlers = 1catalina.org.apache.juli.AsyncFileHandler, java.util.logging.ConsoleHandler############################################################
# Handler specific properties.
# Describes specific configuration info for Handlers.
############################################################1catalina.org.apache.juli.AsyncFileHandler.level = FINE
1catalina.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
1catalina.org.apache.juli.AsyncFileHandler.prefix = catalina.
1catalina.org.apache.juli.AsyncFileHandler.maxDays = 90
1catalina.org.apache.juli.AsyncFileHandler.encoding = UTF-82localhost.org.apache.juli.AsyncFileHandler.level = FINE
2localhost.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
2localhost.org.apache.juli.AsyncFileHandler.prefix = localhost.
2localhost.org.apache.juli.AsyncFileHandler.maxDays = 90
2localhost.org.apache.juli.AsyncFileHandler.encoding = UTF-83manager.org.apache.juli.AsyncFileHandler.level = FINE
3manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
3manager.org.apache.juli.AsyncFileHandler.prefix = manager.
3manager.org.apache.juli.AsyncFileHandler.maxDays = 90
3manager.org.apache.juli.AsyncFileHandler.encoding = UTF-84host-manager.org.apache.juli.AsyncFileHandler.level = FINE
4host-manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
4host-manager.org.apache.juli.AsyncFileHandler.prefix = host-manager.
4host-manager.org.apache.juli.AsyncFileHandler.maxDays = 90
4host-manager.org.apache.juli.AsyncFileHandler.encoding = UTF-8java.util.logging.ConsoleHandler.level = FINE
java.util.logging.ConsoleHandler.formatter = org.apache.juli.OneLineFormatter
java.util.logging.ConsoleHandler.encoding = UTF-8############################################################
# Facility specific properties.
# Provides extra control for each logger.
############################################################org.apache.catalina.core.ContainerBase.[Catalina].[localhost].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].handlers = 2localhost.org.apache.juli.AsyncFileHandlerorg.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].handlers = 3manager.org.apache.juli.AsyncFileHandlerorg.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager].handlers = 4host-manager.org.apache.juli.AsyncFileHandler# For example, set the org.apache.catalina.util.LifecycleBase logger to log
# each component that extends LifecycleBase changing state:
#org.apache.catalina.util.LifecycleBase.level = FINE# To see debug messages in TldLocationsCache, uncomment the following line:
#org.apache.jasper.compiler.TldLocationsCache.level = FINE# To see debug messages for HTTP/2 handling, uncomment the following line:
#org.apache.coyote.http2.level = FINE# To see debug messages for WebSocket handling, uncomment the following line:
#org.apache.tomcat.websocket.level = FINE
Tomcat is publishing logs to four major components, which are Catalina, localhost, manager, and host-manager. As you can see tomcat uses the org.apache.juli.AsyncFileHandler to write each component log to a file. This juli is implemented as apache's own implementation of several key elements of java.util.logging API. This will enable properties files to support extended constructs which allow more freedom for defining handlers and assigning them to loggers. A handler’s log level threshold is INFO by default and can be set using SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST, or ALL. You can also target specific packages to collect logging from and specify a level.
So..What could be the issue here?
So far everything is perfect and nice, but when it comes to containerized environments there could be some issues. As I mentioned before I was working with a Kubernetes deployment recently, In this deployment I was simply tried to deploy the docker container which will run the tomcat server, but the file system of this container is set as a read-only file system. Once I started my deployment it started to get failed, when I checked the logs of my failed container I saw that tomcat was unable to publish its logs to a file in the /logs directory. So we can understand that tomcat was not able to write into this file because of the read-only file system configuration. Tomcat faced this issue because its logs are defined as write to a file, So the solution is we have the set these logs to redirect to the console.
How to redirect tomcat logs to console
In the default configuration file simply removing the file handlers will stop publishing the logs to a file. But there should be a proper way to redirect the logs to the console. To do this I was able to come up with a new logging configuration.
handlers = 1catalina.java.util.logging.ConsoleHandler, \
2localhost.java.util.logging.ConsoleHandler, \
3manager.java.util.logging.ConsoleHandler, \
4host-manager.java.util.logging.ConsoleHandler.handlers = 1catalina.java.util.logging.ConsoleHandler############################################################
# console Handler specific properties.
# Describes specific configuration info for Handlers.
############################################################1catalina.java.util.logging.ConsoleHandler.level = INFO
1catalina.java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
1catalina.java.util.logging.SimpleFormatter.format=[%1$tc] [:Catalina:] %4$s %3$s %5$s %n
1catalina.java.util.logging.ConsoleHandler.encoding = UTF-82localhost.java.util.logging.ConsoleHandler.level = INFO
2localhost.java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
2localhost.java.util.logging.SimpleFormatter.format=[%1$tc] [:localhost:] %4$s %3$s %5$s %n
2localhost.java.util.logging.ConsoleHandler.encoding = UTF-83manager.java.util.logging.ConsoleHandler.level = INFO
3manager.java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
3manager.java.util.logging.SimpleFormatter.format=[%1$tc] [:manager:] %4$s %3$s %5$s %n
3manager.java.util.logging.ConsoleHandler.encoding = UTF-84host-manager.java.util.logging.ConsoleHandler.level = INFO
4host-manager.java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
4host-manager.java.util.logging.SimpleFormatter.format= [%1$tc] [:host-manager:] %4$s %3$s %5$s %n
4host-manager.java.util.logging.ConsoleHandler.encoding = UTF-8############################################################
# Facility specific properties.
# Provides extra control for each logger.
############################################################org.apache.catalina.core.ContainerBase.[Catalina].[localhost].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].handlers = 2localhost.java.util.logging.ConsoleHandlerorg.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].handlers = 3manager.java.util.logging.ConsoleHandlerorg.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager].handlers = 4host-manager.java.util.logging.ConsoleHandler# For example, set the org.apache.catalina.util.LifecycleBase logger to log
# each component that extends LifecycleBase changing state:
#org.apache.catalina.util.LifecycleBase.level = FINE# To see debug messages in TldLocationsCache, uncomment the following line:
#org.apache.jasper.compiler.TldLocationsCache.level = FINE# To see debug messages for HTTP/2 handling, uncomment the following line:
#org.apache.coyote.http2.level = FINE# To see debug messages for WebSocket handling, uncomment the following line:
#org.apache.tomcat.websocket.level = FINE
You can see that I have introduced new console handlers to redirect tomcat logs to the console using java.util.logging.ConsoleHandler.
- <handler-name>.level specifies the default level for the
Handler(defaults toLevel.INFO). - <handler-name>.filter specifies the name of a
Filterclass to use (defaults to noFilter). - <handler-name>.formatter specifies the name of a
Formatterclass to use (defaults tojava.util.logging.SimpleFormatter). - <handler-name>.encoding the name of the character set encoding to use (defaults to the default platform encoding).
But the problem is still there, With this configuration, I was not able to redirect the tomcat access logs to the console. Because of that, my deployment started to fail again due to the read-only file system configuration in my deployment. Currently, you can not redirect the tomcat access logs to the console because access logs configurations are defined in the server.xml file in order to provide a new configuration for access logs you have to come up with a new custom valve implementation. So what I did to resolve this issue was mounting an emptyDir volume mount to the /logs directory in tomcat.
Configuring the emptyDir volume mount
To resolve this issue I tried to mount an emptyDir volume to the default logging directory in tomcat which is “/logs” directory. emptyDir volume is initially empty, so when you mounting it please make sure that you don’t have any files in that directory because when you mount an emptyDir it will initially create an empty directory in the respective location, which may remove your other files in that directory also. the best thing about emptyDir is that all containers in a pod can read and write the same files in the emptyDir. So this will resolve our issue by letting the tomcat write the logs to the emptyDir volume mount which is mounted to the “/logs” directory.
volumes:
emptyDir: {}
- name: tomcat-logs-dir Once you define the emptyDir you have to mount it to the tomcat “/logs” directory.
volumeMounts:
- name: tomcat-logs-dir
mountPath: /usr/local/tomcat/logsOnce I did this change I was able to lunch my tomcat server with a read-only file system configuration for my deployment. You can log into the container and check the logs and you should be able to see the logs in the console except for access logs, to check the access logs you can go to the /logs directory check the access log file there. So I hope this article will help you configuring the apache logs in your deployment :).
