Cloudfront Realtime logs limitation; workarounds, and possible solutions
CloudFront Realtime logs are essential for monitoring and troubleshooting requests in real-time, particularly during migration or high-volume scenarios.
Today, we will explore how to handle real-time logs in CloudFront, discuss the obstacles we encountered, and detail how we overcame them.
Before we dig into the nitty-gritty and challenges, it’s crucial to note that having real-time logs is essential when migrating to CloudFront from another CDN provider. We must be able to monitor requests in real time to ensure nothing breaks during the migration.
We’re dealing with a huge number of requests every second, from different clients, so it’s crucial to be able to securely and instantly look into their requests.
If you’ve had experience dealing with real-time logs, you’re aware that they often come with fixed settings and don’t easily adapt to include important metadata like specific headers and unique business processes.
Today, I’ll guide you through our process, in PayU api.paymentsos.com
Let’s start by listing out what we need:
- Logs need to be visible in real time.
- There shouldn’t be any confidential information included.
- They should integrate personalized business rules, like sending particular headers.
A Lambda function transformation is a must.
When we began working on this, we realized that the live logs were being sent in simple text format without any identifiers no keys there and this stuff should be done by our end in order to be able to understand what is going on. This meant that we had to employ a Lambda function to convert the log lines to JSON format and add relevant keys as you can see in my example:
function transformRecord(event, context, callback) {
const { records } = event;
console.log("events: " + JSON.stringify(event))
const transformedRecords = records.map((record) => {
const data = Buffer.from(record.data, 'base64').toString('utf-8');
console.log("record before transform: " + data);
const fields = data.split('\t');
const mapFields = [
'datetime',
'c_ip',
'time_to_first_byte',
'sc_status',
'sc_bytes',
'cs_method',
'cs_protocol',
'cs_host',
'cs_uri_stem',
'cs_bytes',
'x_edge_location',
'x_edge_request_id',
'x_host_header',
'time_taken',
'cs_protocol_version',
'c_ip_version',
'cs_user_agent',
'cs_referer',
'cs_cookie',
'cs_uri_query',
'x_edge_response_result_type',
'x_forwarded_for',
'ssl_protocol',
'ssl_cipher',
'x_edge_result_type',
'fle_encrypted_fields',
'fle_status',
'sc_content_type',
'sc_content_len',
'sc_range_start',
'sc_range_end',
'c_port',
'x_edge_detailed_result_type',
'c_country',
'cs_accept_encoding',
'cs_accept',
'cache_behavior_path_pattern',
'cs_header_names',
'cs_headers_count',
'primary_distribution_id',
'primary_distribution_dns_name',
'origin_fbl',
'origin_lbl',
'asn'
];
const transformedData = mapFields.reduce((obj, key, index) => {
obj[key] = fields[index].replace('\n','');
return obj;
}, {});
transformedData.datetime = transformedData.datetime.split('.')[0];
const transformedRecord = {
recordId: record.recordId,
result: 'Ok',
data: Buffer.from(JSON.stringify(transformedData)).toString('base64'),
};
return transformedRecord;
});
console.log("returned object:" + JSON.stringify(transformedRecords));
callback(null, { records: transformedRecords });
}
module.exports = { transformRecord };You have no control over which headers to send.
We encountered another significant challenge as we were unable to select specific headers to send in our logs, despite having the option to include headers. It posed a problem as we required visibility of certain headers such as APP_ID and CUSTOMER_ID — some of our custom headers — for quick issue resolution by our team.
By associat a new Lambda function at the CloudFront level, to relevant behavior as Origin response , we were able to capture and log the specific headers and request information needed. Once set up, retrieving these detailed logs from the appropriate CloudWatch log group and sending them to the correct S3 bucket significantly improved our logging process.
exports.handler = (event, context, callback) => {
let record = event.Records[0].cf;
let requestHeader = record.request.headers;
let responseHeader = record.response.headers;
let requestDetails = {
"datetime": Math.floor(new Date().getTime() / 1000),
"cf_request_id": record.config.requestId,
"distribution_id": record.config.distributionId,
"client_ip": record.request.clientIp,
"method": record.request.method,
"origin": record.request.origin.custom.domainName,
"uri":record.request.uri,
"status_code":record.response.status,
"status_description": record.response.statusDescription
};
if(responseHeader["x-zooz-request-id"] !== undefined && responseHeader["x-zooz-request-id"][0].value !== ""){
requestDetails['x_zooz_request_id'] = responseHeader["x-zooz-request-id"][0].value;
}else{
requestDetails['x_zooz_request_id'] = null
}
........
console.log(JSON.stringify(requestDetails));
callback(null, record.response);
};Issue: Missing Logs, Investigation and Solution:
After connecting the real-time logs with the proxy logs, we discovered a significant issue: a large number of logs were missing, we compared internal logs that we have with cloudfront logs and the numbers was pretty different. We had assumed that combining the two would suffice, but this was not the case. Upon investigation, we identified two crucial oversights. Firstly, we had failed to collect logs from relevant regions. As CloudFront triggers logs based on the viewer’s location, it is essential to gather data from every region for a comprehensive view. Secondly, we had overlooked implementing pagination when reading logs from CloudWatch. Without pagination, our ability to see all log entries was hindered by the substantial volume of data. These oversights led to us missing important log information.
const AWS = require('aws-sdk');
const zlib = require('zlib');
const s3 = new AWS.S3();
// Define all AWS regions as of now
const regions = [
'us-east-1', 'us-east-2', 'us-west-1', 'us-west-2',
'af-south-1', 'ap-east-1', 'ap-southeast-3', 'ap-south-1',
'ap-northeast-3', 'ap-northeast-2', 'ap-southeast-1',
'ap-southeast-2', 'ap-northeast-1', 'ca-central-1',
'eu-central-1', 'eu-west-1', 'eu-west-2', 'eu-south-1',
'eu-west-3', 'eu-north-1', 'me-south-1', 'sa-east-1'
];
exports.handler = async (event) => {
const oneHourAgo = new Date();
oneHourAgo.setHours(oneHourAgo.getHours() - 1);
let logMessages = '';
let logLines = 0;
const logFetchPromises = regions.map(async region => {
let params = {
logGroupName: process.env.response_proxy_cloudwatch_logGroupName,
startTime: oneHourAgo.getTime(),
endTime: new Date().getTime(),
filterPattern: '"cf_request_id"'
};
let cloudWatchLogs = new AWS.CloudWatchLogs({region: region});
try {
let eventData = await fetchAllLogEvents(cloudWatchLogs, params,region);
console.log(`logData in region ${region}: ${eventData.length}`);
eventData.forEach(event => {
if (event.message.indexOf('cf_request_id') > -1) {
let cloudfrontLogLine = event.message.split('\tINFO\t')[1];
logMessages = logMessages + cloudfrontLogLine;
logLines++;
}
});
} catch (error) {
console.log(`No logs found or access denied in ${region}:`, error.message);
}
await writeToS3(logMessages,logLines,region);
});
await Promise.all(logFetchPromises);
};
async function writeToS3(logMessages,logLines,region) {
if (logMessages !== undefined && logMessages !== '') {
const compressedData = zlib.gzipSync(logMessages);
try {
const s3Params = {
Bucket: process.env.response_proxy_s3_bucket_name,
Key: `/logs_${region}_${Date.now()}.gz`,
Body: compressedData,
ContentType: 'application/gzip'
};
let response = await s3.putObject(s3Params).promise();
console.log(`Response from s3: ${JSON.stringify(response)}`);
console.log(`Logs written to s3: ${s3Params.Key}`);
console.log(`Number of log lines written: ${logLines}`)
}catch (error) {
console.log(`Error writing logs to s3: ${error.message}`);
throw error;
}
} else {
console.log("No new logs to write to s3");
}
}
async function fetchAllLogEvents(cloudWatchLogs,params,region) {
let allLogEvents = [];
let fetchedPages = 0;
while (true) {
const logData = await cloudWatchLogs.filterLogEvents(params).promise();
allLogEvents = allLogEvents.concat(logData.events);
console.log(`region:${region}, Fetched page ${++fetchedPages}: ${logData.events.length} events`);
if (logData.nextToken) {
params.nextToken = logData.nextToken;
} else {
break;
}
}
return allLogEvents;
}Now let’s describe the Final design:
The First Flow:
- Configure realtime logs to be sent to Kinesis and then forwarded to FIREHOSE.
- Implement a Transform function to convert plain text to JSON.
- Send the logs from the FIREHOSE to an S3 bucket.
- Read logs from an S3 bucket and transfer them to a log vendor using a lambda function
- Read the logs from the S3 bucket and send them to Snowflake in the “realtime log table”.
The Second Flow:
- Attach the Proxy Logs Lambda function to the relevant CloudFront behavior under origin response. This will automatically write the logs to CloudWatch.
- Read logs from CloudWatch in multiple regions and then send them to an S3 bucket.
- Read logs from the S3 bucket and send them to Snowflake in the “proxy log table”
Then you just need to create a join between the two tables, and you will have all the necessary information about any incoming request.
