Understanding Logging and Monitoring in Google Cloud Platform (GCP)

In the dynamic world of cloud computing, effective logging and monitoring are indispensable for maintaining the health, performance, and security of your applications and infrastructure. Google Cloud Platform (GCP) offers robust logging and monitoring solutions that empower developers and administrators to gain insights into their systems, detect issues proactively, and optimize performance. In this article, we’ll delve into the concepts of logging and monitoring in GCP, followed by a practical lab to demonstrate their implementation.

Understanding Logging and Monitoring in GCP:

Logging in GCP: Logging in GCP enables you to capture, store, and analyze logs generated by your applications and services running on the platform. GCP’s logging solution is powered by Cloud Logging, which seamlessly integrates with other GCP services to collect logs, providing real-time visibility into your infrastructure.

Key features of Cloud Logging include:

1. Unified Logging: Cloud Logging consolidates logs from various GCP services, including Compute Engine, App Engine, Cloud Functions, and more, into a single, centralized platform.
2. Real-time Insights: It offers real-time log ingestion and analysis, allowing you to monitor events and troubleshoot issues promptly.
3. Scalability: Cloud Logging is designed to handle massive volumes of logs efficiently, ensuring that you can capture and retain logs without worrying about scalability limitations.
4. Advanced Filtering and Searching: With powerful filtering capabilities, you can easily search and filter logs based on criteria such as severity level, timestamp, resource type, and custom labels.

Monitoring in GCP: Monitoring in GCP provides comprehensive visibility into the performance, availability, and health of your cloud resources. Powered by Cloud Monitoring, this solution helps you proactively identify and resolve issues before they impact your applications or users.

Key features of Cloud Monitoring include:

1. Dashboards and Metrics: Cloud Monitoring offers customizable dashboards and a vast array of pre-defined metrics, enabling you to monitor the performance of your resources in real-time.
2. Alerting and Notifications: You can set up alerts based on specific conditions or thresholds, ensuring timely notifications when issues occur. Integration with various notification channels, such as email, SMS, and PagerDuty, allows for seamless incident management.
3. Auto-scaling Policies: Cloud Monitoring can automatically adjust the scaling of your resources based on predefined conditions, optimizing resource utilization and cost efficiency.
4. Integration with Logging: Cloud Monitoring seamlessly integrates with Cloud Logging, allowing you to correlate monitoring data with log entries for comprehensive analysis and troubleshooting.

Practice Lab: Logging and Monitoring in GCP

Objective: In this lab, we’ll set up logging and monitoring for a sample application deployed on Google Compute Engine.

Prerequisites:

1. Google Cloud Platform account
2. Access to the GCP Console

Steps:

1. Create a Compute Engine Instance:

• Navigate to the Compute Engine section in the GCP Console.
• Click on “Create Instance” to launch a new virtual machine.
• Configure the instance with the desired specifications and select a suitable region.
• Click “Create” to provision the instance.

2. Enable Logging and Monitoring APIs:

• Go to the APIs & Services section in the GCP Console.

• Search for “Cloud Logging API” and “Cloud Monitoring API”.
• Enable both APIs for your project.

Cloud Monitoring API has been enabled

Method One. Views Monitoring

Method Two. Views Monitoring

Monitoring and Logs shows

3.Install Ops Agent

Not All things such as Memory ins monitorable yet (disabled) until we install ops Agent into the compute engine.

Some metrics need Ops Agent Installed

The Ops Agent collects metrics, logs, and traces from your Compute Engine instances and workloads for troubleshooting, performance monitoring, and alerting.

• Automatically collect host metrics such as CPU, GPU, memory, and process metrics.
• Automatically collect system logs such as syslog from Linux VMs and Windows Event Log from Windows VMs.
• Observe your applications with 3rd party application integrations, application logs, Prometheus metrics, and OpenTelemetry Protocol (OTLP) metrics and traces.

4.Install Monitoring Agent:

• SSH into the Compute Engine instance created earlier.
• Download Agent Installation Script

curl -sSO https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh
sudo bash add-google-cloud-ops-agent-repo.sh --also-install

Installing Ops Agent

After it is installed, the agent is started automatically.

5. Set Up Monitoring Dashboard:

• Go to the Monitoring section in the GCP Console.
• Create a new dashboard and add widgets to monitor CPU usage, disk I/O, and network traffic of the Compute Engine instance.

• Customize the dashboard layout and configure alerts for critical thresholds.

Adding Widget

Create Alerting Policy

Create Alert Policy

Final dashboard

4. Create Sample Application with Logging:

For this lab, we’ll deploy a simple web application written in Python using Flask framework. The application will serve a basic web page and generate log entries for each HTTP request

1. create A VM named vm_app, choose small for machine type.

And make sure before creating to tick allow http and https below

2. As we will code using python. We need to install python.

Enter the VM terminal with ssh.

Klik SSH

Terminal is open

Install python by running following

sudo apt update
sudo apt install python3
python3 --version

Python installation

Next we need to install pip by running the following

sudo apt update
sudo apt install python3-pip

Install pip

Create directory named code in you home dir. Copy the follwing sample application code to a Python file (e.g., app.py) on the instance.

cloud_user_p_dd52da32@vm-app:~$ ls
cloud_user_p_dd52da32@vm-app:~$ mkdir code
cloud_user_p_dd52da32@vm-app:~$ cd code

Open vi editor, or any editor to copy the code into file named app.py.

If using vi editor , you can type i then copy pasted the code.

cloud_user_p_dd52da32@vm-app:~/code$ vi app.py

from flask import Flask
import logging
 
app = Flask(__name__)
@app.route('/')

def hello_world():
    app.logger.info('Hello, World request received!')
    return 'Hello, World!'

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

To exit vi editor you can press ESC then type :wq! as the screenshot below.

in the same folder, create requirements.txt contains this code

Flask==2.0.1
Werkzeug==2.0.1
# Add other dependencies as needed
gunicorn==21.2.0

install all the package needed listed in the requirements.txt by running

pip install -r requirements.txt --break-system-packages

• If you have python installed, you can test by running the application using the command python app.py or python3 app.py

you can test by visiting above url from within the linux using curl.

5.Visit the Web App from internet

If you want to test from outside network you need to refer its external/public ip.

Find the Public IP

And need to configure your firewall in VPC to allow traffic in on port 8080 as follows

Klik the default VPC

Open the firewall tab

Klik “Add Firewall rule”. Set the following fields

Click create on above

Now you can test in on browser:

6.View Logs in Cloud Logging:

• Navigate to the Logging section in the GCP Console.
• Explore the logs generated by the Compute Engine instance.
• Apply filters and search queries to analyze specific log entries.

7. Analyze Metrics and Logs:

• Correlate the metrics displayed on the monitoring dashboard with the corresponding log entries in Cloud Logging.
• Use advanced filtering and searching capabilities to troubleshoot issues or analyze performance bottlenecks.

Conclusion: Logging and monitoring are essential components of any cloud infrastructure strategy, enabling organizations to maintain operational excellence and deliver superior user experiences. With Google Cloud Platform’s robust logging and monitoring solutions, you can gain deep insights into your applications and infrastructure, ensuring reliability, performance, and security. By following the practice lab outlined in this article, you can familiarize yourself with the logging and monitoring capabilities of GCP and leverage them effectively in your projects.