Capture and analyse WebLogic logs in OCI

Published in

Oracle Developers

8 min readJun 8, 2021

Introduction

After migrate WebLogic domain to Oracle Cloud Infrastructure (using JCS, WebLogic marketplace or custom WebLogic domain on compute instance), operation team needs to access WebLogic log files to investigate ongoing issues.

We can centralise all managed servers log files using OCI logging service and use OCI log analytics to find any specific pattern in the logs.

In this article I am going to explain how to configure OCI logging for WebLogic domain and ingest logs in OCI log analytics for analytics purpose.

I just want to highlight that I am going to only cover instances inside OCI in this article. If you want to ingest logs from any host inside on-premises or other cloud provider you need to use management agent that I will cover in future article.

Architecture

Here is high level architecture diagram:

In this architecture, OCI Unified Monitoring Agent collect and pars (if needed) custom logs in compute instances and send them to OCI logging service. you can do simple search inside log entries in OCI logging, but if you need enterprise grade log analytics solution, you should transfer log files to OCI log analytics using OCI Service connector hub.

In next part I will explain in detail to how implement this solution inside OCI.

Implementation

Here are the steps that you need to follow to implement this solution inside your OCI tenancy (I assume you already provision networking resources and install and deploy your WebLogic cluster on OCI compute instances).

Agent Installation

To ingest events from your applications into your custom log, you can install the Oracle fluentd-based agent. This agent allows you to control exactly which logs you want to collect, how to parse them, and more.

Oracle Cloud Infrastructure Logging provides an easy mechanism (Agent Configurations) to enable and manage the agent for a set of supported operating systems. Agent Configurations give you a central experience to easily configure what custom logs you want to ingest across your fleet of hosts.

depending of type of your instance (new OCI instance, existing OCI instance or instances created from custom image) you should follow different steps to install and configure monitoring agent (please follow Agent Management document).

after install and configure the agent on compute instance you can check status of instance using following command.

sudo systemctl status unified-monitoring-agent

You can enable monitoring agent when provision new compute instance. simply select “Show Advanced Options” in “Create Compute Instance” screen in OCI console and select “Oracle Cloud Agent” tab. then you can enable “Custom Logs Monitoring” plugin.

Also, you can enable “Custom Logs Monitoring” plugin for existing compute instance by navigating to the instance details page (in OCI console) and select “Oracle Cloud Agent” tab (this plugin enabled by default for compute instances).

Keep in mind, the monitoring agent is more than log collector agent. it uses for collect performance metrics, install OS updates, and perform other instance management tasks.

2. Agent Configuration and logging service configuration

After agent installation, we should create agent configuration and select following information:

Which hosts you want to collect logs from.
Exactly which logs you want to ingest from those hosts.
A log group/log destination.

Log groups are logical containers for organizing logs. Logs must always be inside log groups. You must create a log group to enable a log.

Before create agent configuration, you should create a OCI dynamic group that include all OCI instances which you want to collect logs from.

To create dynamic group, navigate to Identity & Security -> Dynamic Groups inside OCI console and create new dynamic group with following matching rule:

ANY {instance.id = 'ocid1.instance.<region>.<location>.<unique_ID>', 
instance.compartment.id = 'ocid1.compartment.<region>..<unique_ID>'}

You can specify instance OCID and compartment OCID of all WebLogic instances in the matching rule.

All {instance.compartment.id = 'ocid1.compartment.<region>..<unique_ID>'}

or simply specify compartment OCID that include all WebLogic instances (if this compartment only include WebLogic instances).

Also, you should create a policy to grant permission to the dynamic group that you created earlier to use log-content in the tenancy.

To create the policy, navigate to Identity & Security -> Policies inside OCI console and create new policy with following statement:

allow dynamic-group <dynamic_group_name> to use log-content in tenancy

after creating the policy and dynamic group, you should create the log group and the custom log. you should navigate to Observability & Management -> Log Groups inside OCI console and create new log group and populate compartment, name and description for the log group:

then select logs menu item and create new custom log and populate compartment, custom log name and log group:

Now we can create agent configuration. you should navigate to Observability & Management -> Agent Configuration inside OCI console and create new Agent Configuration in specific compartment and populate agent name, description, compartment, dynamic group that you create in previous step and WebLogic log location (you can use wildcard in the path):

WebLogic server logs by default write inside {Domain_Home}/servers/{Server_Name}/logs/{Server_Name}.log file. but you can change this path in WebLogic console. I suppose we are using default file path in this article.

Finally, you should login to all WebLogic compute instance and restart the agent using following command:

sudo systemctl stop unified-monitoring-agent
sudo systemctl start unified-monitoring-agent

After few minutes, open custom log that you have created in above step, and you can see WebLogic logs in the explore log screen.

3. Configure connector hub

Service Connector Hub orchestrates data movement between services in Oracle Cloud Infrastructure.

Data is moved using service connectors. A service connector specifies the source service that contains the data to be moved, optional tasks, and the target service for delivery of data when tasks are complete. An optional task might be a function task to process data from the source or a log filter task to filter log data from the source.

In this step, we will create a service connector to send data from OCI logging service to OCI Logging analytics.

To create a service connector, navigate to Observability & Management -> logging -> Service Connectors inside OCI console and create new Service Connector and populate name, description, compartment, source and target (in our scenario source is logging service and target is Logging Analytics)

also, you should provide log group and custom log for logging service and log group for logging analytics (we should create this log group first that I will explain how to do it in next section).

We also can add log filter task (to filter log files from source) and configuration function task to process data before send to target (I will ignore these configurations for our scenario).

4. Configure Logging Analytics

The Oracle Cloud Infrastructure (OCI) Logging Analytics service is a highly scalable, reliable, and real-time log analysis solution. Logging Analytics automates the collection of historic and real-time logs from any on-premises or cloud resource.

In last part, we will configure logging analytics and create simple report on WebLogic logs in Log Explorer.

First, you should enable log analytics service in your region. navigate to Observability & Management -> Logging Analytics and select “Start Using Logging Analytics”.

If you are not OCI Administrator, you need few permissions to use logging analytics service (please refer to Logging Analytics documentation).

Now you can navigate to log explorer page. first, we need to create log group for Logging Analytics service. Select Administration menu and then select log groups in Administration page and create new log group for logging analytics in specific compartment (you need to use this log group when you create service connector in step 3).

Finally, navigate to Log Explorer select log source in Field’s tab and select “OCI Unified Schema Logs”.

In visualisations tab select “Records” visualisation. also, you can add and remove display fields.

Now you can see WebLogic logs in log explorer page.

You can search in the log content or create cluster visualisation (Clustering uses machine learning to identify the pattern of log records, and then to group the logs that have a similar pattern) to remove similar log entries.