How to implement Node.js to format data sent by Kinesis Firehose with Lambda

Tomokazu Kozuma
Jul 22, 2017 · 2 min read

Log formatting with Lambda

It is assumed that Kinesis Firehose forwards the log to S3, receives the log at Lambda, and formats the log. I will explain how to write Lambda processing at that time in Node.js.

Source Code

exports.handler = (event, context, callback) => {
const output = event.records.map(record => {
const data = Buffer.from(record.data, ‘base64’).toString(‘utf8’);
const parsedData = JSON.parse(data);
parsedData.text = “add text”return {
recordId: record.recordId,
result: ‘Ok’,
data: Buffer.from(parsedData, ‘utf8’).toString(‘base64’)
};
});
callback(null, {records: output});
};

This is the process from receiving data until returning. Since it is written with ES 6, please upload source code after trans-piling with Babel. The function name is handler, but whatever name you use is okay.
Please refer to the following article for Transpail.

How to receive data

It can be obtained from event.records of the first argument. The data received from Kinesis Firehose will be multiple.

Since the base64 encoded data is passed, it is converted to utf8. Also, if you are sending in JSON you have to do JSON.parse.
In summary, the flow of this data conversion is below.

Binary (base64) -> String (UTF8) -> JSON

How to return data

When returning data, overwrite the changed data and set the character string ‘Ok’ to the result key. Is converted into binary data in base64 encoded to return to the same state as when it was last received is complete it will be rewarded by passing the second argument of the callback.

Tomokazu Kozuma

Written by

Writing About Node.js and Socket.IO. CyberAgent,Inc. Tokyo Institute of Technology github: https://github.com/tomokazukozuma

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade