Use AWS Lambda to Append Daily Data to CSV File in S3 Bucket

I had a task to track the number of subscribers for mental health subreddit groups on a daily basis. When I was doing that, I found a lot of tutorials online incomplete and outdated. In this article, I will share my end-to-end solution to this problem.

There are four major steps in this solution.

1. Download your S3 csv file in /tmp folder. ‘tmp’ is lambda local folder. You have to download the file in this folder and then you can manipulate data.
2. Read the data from /tmp/test.csv, convert it to the list, and insert the data. Write the data to /tmp/test.csv file row by row. (I use test.csv as an example)
3. Upload from /tmp/test.csv to S3 key.

The template code will look like this:

import boto3
import csv 
# call s3 bucket
s3 = boto3.resource('s3')
bucket = s3.Bucket(BUCKET_NAME) # Enter your bucket name, e.g 'Data'
# key path, e.g.'customer_profile/Reddit_Historical_Data.csv'
key = 'KEY_PATH'
# lambda function
def lambda_handler(event,context):
    # download s3 csv file to lambda tmp folder
    local_file_name = '/tmp/test.csv' #
    s3.Bucket(BUCKET_NAME).download_file(key,local_file_name)
    
    # list you want to append
    lists = ['5/20/2020',123,321,31,'A'] 
    # write the data into '/tmp' folder
    with open('/tmp/test.csv','r') as infile:
        reader = list(csv.reader(infile))
        reader = reader[::-1] # the date is ascending order in file
        reader.insert(0,lists)
    
    with open('/tmp/test.csv', 'w', newline='') as outfile:
        writer = csv.writer(outfile)
        for line in reversed(reader): # reverse order
            writer.writerow(line)
    
    # upload file from tmp to s3 key
    bucket.upload_file('/tmp/test.csv', key)
    
    return {
        'message': 'success!!'
    }

If you have a better approach to this problem, please leave your comments below.