How to create server/machine disk usage dashboard and trigger “delete logs” DevOps action automatically?

Author: Haydar Seyfi and Salih Ozdemir

“Oracle WebLogic Server has quite a number of log files that are generated which naturally tend to grow. Without any type of log rotation, archiving, or purging set up, your file system can be filled up rather quickly and eventually pose stability issues to your application environment.”

Fortunately, WLSDM can create “Generic DevOps Mbeans” to prevent this scenario. It is also extremely simple.

INFO: We are going to create a dashboard in WLSDM console and add Generic DevOps MBean Metrics.

1. Creating Generic DevOps Mbean

1.1 Go to WLSDM console.

1.2 Go to “Configuration > Monitoring & Diagnostics” page.

Script Types and Usage

(OS Executables: sh, bash, bat, cmd)

Key / value pairs are the expected output format for the OS executable scripts. Sample output(echo) is listed below.

attributeName=attributeValue  
TotalCompletedOrders=2017  
TotalWaitingOrders=35

Note: MBean attribute names cannot contain non alphanumeric characters except underscore ( _ ). Non-alphanumeric characters must be replaced.

Sample Shell Script — 1 : Disk usage of a Linux server

df -H | grep -vE '^Filesystem|tmpfs|lv|dev' | awk '{ print $5"="$4}' | sed 's/%//' | sed '1s/\//ROOT/'

Sample Shell Script — 1 Output :

ROOT=85  
data=67  
u01=76

1.3 Type “Script” as the following highlighted script.

echo DISK_DATA=$(df /data | grep /data | awk '{print $5}' | tr '%' ' ') ; echo DISK_ROOT=$(df / | grep / | awk '{print $5}' | tr '%' ' ')

This script will generate below attributes; values comes from domain disk usage.

DISK_DATA=57  
DISK_ROOT=4

1.4 Click “Transform: Run & Get MBean Attributes” button. Wizard is going to generate JMX MBean Attributes then click “Next” button.

1.5 MBean Values can be visualize on WLSDM Console Smart Dashboards.

1.6 Click the Smart Dashboards > Custom Dashboard in the main menu.

If you are able to see below dashboard on your WLSDM console, it means you have successfully created Generic Disk Usage Mbean.

2. Adding Action to Generic DevOps Mbean

INFO: We are going to add an action in Generic DevOps MBean.

2.1 Go to “Configuration > Monitoring & Diagnostics” page.

2.2 Click the “New User Defined Action / Script” button from the right corner in WLSDM console.

2.3 Type action name ( i.e. DeleteLogFilesAction )

2.4 Choose Type as “Embedded Script” from “Script Source Type” dropdown list.

2.5 Type “Executable Script” as the following highlighted script(Specify your WebLogic path and log type)

find /data/weblogic/domains/$1/servers/. -type f -mtime +3 | grep -E 'server-2019*|access-2019*' | awk '$1' | xargs rm -rf

This script going to delete your rotate log files (as starting with ‘server-2019’, ‘access-2019’ logs) older than 3 days. You can specify your domain path and log type.
‘$1’ parameters will came from WLSDM “System Argument/Parameters” section if you choose one of them.

Sample scripts;

find /data/wlslogs/$1/. -type -f -mtime +5 | grep -E '*.log0*|*.out0*|*.err0*' | awk '$1' | xargs rm -rf  
find /data/wlslogs/sample-dmn/. -type -f -mtime +5 | grep -E '*.log0*|*.out0*|*.err0*' | awk '$1' | xargs rm -rf

2.6 Select Parameters as “$DOMAIN_NAME” from “System Arguments/Parameters” dropdown list, then click “Save”.

2.7 Now go to “Smart Dashboards > Custom Dashboard” page.

2.8 Click “Metric & Chart Options” button.

2.9 Click “Action” button and select “DeleteLogFilesAction” action then click save.

2.10 Set the alarm “ON”, choose Operator as “>=” then give Threshold value; finally save your configuration.

Let WLSDM reduce your disk usage and optimize your domain stability.

2.11 Check your domain disk usage with “df -h” command.

Our Threshold value is : 50
Data disk usage is : 57

DiskUsage Mbean will trigger “DeleteLogFiles” action and domain disk usage will be reduced.
If you are able to see changes like below dashboard, it means you have successfully completed this Generic Disk Usage Mbean Tutorial.
That’s it!
That’s how WLSDM organize your domains…

Watch disk usage DevOps MBean youtube tutorial video: