PHP Application Logging in Google App Engine — Analyze Them in Smart Way
All software engineers are familiar about their application logging features. It’s always a good practice to log as much as we can to figure out issues, bugs, compliance purpose or activity tracking, etc. To log anything from inside PHP application is easy but there are lots of way to analyze them, visualize them and get analytical data from those logs. But sometimes it’s hard to manage millions or billions of logs in old-fashioned way I mean with text based log handler or database driver log handler.
Let’s assume, you have a To-Do List application hosted in Google App Engine and you have 500 users at least who use their application everyday. And your job is to make sure that your application is running properly for all users in all possible way. How you can do that? Log every possible facts that’s happening when any of your user is doing anything inside your application. How you analyze data and how fast you can see any errors and take action? Of course, you download the log file or query your log storage system like MySQL or any other database. The question is how much log you can manage and handle with scale? If your log data size is terabytes and billions or millions of rows. And what specific system you integrated with your Google App Engine App?
Here is my own concept to manage any size or data in smart way. If your application is running on Google App Engine, you obviously know about StackDriver logging. It’s one of the most powerful feature GCP (Google Cloud Platform) has brought for it’s users. And mostly it’s Log Sink feature is awesome and useful. You can sync your logs in real-time to any remote storage like BigQuery, Google Cloud Storage bucket, Pub-Sub messaging, etc.
For PHP developers, Monolog logger is much popular among other logging library. I use it in all of my PHP application. It requires little bit of tweaking to perfectly use it for your Google App Engine application.
Assuming, your application is using composer for using all third-party library. Just add monolog package in your application with following command.
composer require monolog/monolog
This will add Monolog logging library to your application. Now you need to configure it. Remember, Google App Engine won’t let you write any files when your app is deployed. Because their entire PHP55 runtime work with read-only filesystem. So for logging you have to use syslog. All log you will send to it’s system’s log and it will be stored in Stackdriver logging system. Now let’s setup monolog with syslog handler.
Now we configured out logger. And from your application you can log anything with -
Great! So you started taking log from your application. Try to log as much as possible. Now it’s time to see those logs from StackDriver logging interface from GCP console.
Click on that “Logs” link and you will get the following logging interface
Here it is. Your application log is visible in StackDriver logging interface. Now our next step is to create sink and send these logs to BigQuery so we can work with lots of data and later we will see how to visualize and analyze our logs for future purpose.
Syncing StackDriver Logging to BigQuery
Click on exports and let’s create a new sink for our BigQuery. From the right side, provide a sink name, select Big Query as destination and create a BigQuery datasets and create sink.
Wait couple mins and go to your BigQuery homepage to see the datasets and from now on every log from your Google App Engine will be stored in this datasets. Let’s see how it looks like in BigQuery.
Querying Those Logs and Get Results
Let’s query some data (first time, it will take couple mins or hours for your logs to be available in BigQuery, after that it’s nearly real-time)
Run this query and you will see how your PHP application log by Monolog is being logged in StackDriver Logging and syncing with your BigQuery datasets. Here is the output of the query
Out log messages are in JSON format. You will understand why we configured our Monolog syslog handler with JsonFormatter(). Now let’s figure out how we can extract every JSON key value from those full_log_message. We will use BigQuery JSON_EXTRACT(expr…) to extract data from that JSON data. Let’s make another query.
And here is the output.
So we extracted all data from JSON and displaying in separate column here.
It’s pretty simple. We just need to design the architecture by connecting all App Engine + StackDriver Logging + Log Sink to BQ = Get log insights
Visualize Data with DataStudio
Google DataStudio is an awesome tools where you can easily visualize your data and get in-depth data insights without almost any technical knowledge. So here, we have our BigQuery datasets where all of our application logs are being stored and now we want to use DataStudio to get our insights from application Log.
First, you need to create a data source in DataStudio.
And start visualizing your data. Spend couple hours and save your future time. Build your reports as you need them.
Note: Share your thoughts and the way you collect/store and analyze PHP application log from Google App Engine app.