Simple AWS S3 Logging in Python3 Using Boto3
Simple way of implementing S3 logging in Python3
One of the Key aspects of a software framework is to log necessary information and persist them. Using boto3 and native python logger its easy to persist logs into AWS S3 within a python program.
Logs would be helpful to debug, monitor, audit and understand the behaviour of the framework at any given point of time. When it comes to applications in cloud, we don’t store important information in local machine, instead we would store it in cloud storage service like AWS S3.
Recently i’ve been working on a framework to automate Big Data ETL deployments using python. The requirement was to persist the logs to S3 for future reference, without using complex code or much code changes. The requirement was not to have the logs in realtime in S3.
My intention was to leverage existing Python logger that i’ve been using all along for all python codes and Boto3 SDK to interact with AWS S3. Here is a simple approach of storing logs generated by Python logger into S3.
Approach
Create String I/O Logger:
Python logger allows us to add different kinds of handlers. We can leverage this to add a String I/O Stream handler which is basically a string buffer that would store all the logs generated by logger.info or logger.error function.
Put Content to S3:
Boto3 is a python SDK to interact with AWS Services. Here i have created a function put_content_to_s3() to put given content in specified s3 path. This function also takes backup key as parameter and if in case the file already exists in the specified S3 location, it will backup the old content in either backup folder or backup file based on specified backup strategy. This backup strategy can be customised as required based on the requirement.
The instance on which the code is running should have read and write access to specified S3 path.
How to use S3 logging:
The main function of the code will create a String I/O buffer and string I/O logger. The code will have a normal flow with logger.info and logger.error.
In the finally block put_content_to_s3() function is called to put the logs content from String I/O buffer to S3.
This is one of the simple and easy way of implementing S3 logging in Python3 using existing python logger and Boto3. There are few libraries available in open source for this purpose, but here the intention was not to use any untrusted python libraries and not to over complicate things.
Happy Coding!
