Botocore is awful, so I wrote a better Python client for AWS S3
If you’ve ever been unfortunate enough to have had to work with botocore, Amazon Web Services’ Python API, you know that it’s awful. There are dozens of ways to accomplish any given task, and the differences between each are unclear at best. I recently found myself working with botocore while trying to build some S3 functionality into codelighthouse, and I got really frustrated with it, really quickly.
AWS S3 (Simple Storage Service) is not complicated — it’s object storage. You can GET, PUT, DELETE and COPY objects, with a few other functionalities. Simple, right? Yet for some reason, if you were to print botocore's documentation for the S3 service, you'd come out to about 525 printed pages.
I chose to use the Object API, which is the highest-level API provided by the S3 resource in botocore, and it was still a headache. For example, the Object API doesn’t throw different types of exceptions — it throws one type of exception, which has numerous properties that you have to programmatically analyze to determine what actually went wrong.
There are a few open-source packages out there already, but I found that most of them left a lot to be desired — some had you writing XML, and others were just more complicated than they needed to be.
To save myself from going mad trying to decipher the docs, I wrote a custom high-level driver that consumes that low-level botocore API to perform most basic botocore functionalities.
To save other developers from the same fate I narrowly avoided, I open-sourced the code and published it on PyPi so you can easily use it in all of your projects.
Let’s Get Started
Installing my custom AWS S3 Client
Since my client code is hosted via PyPi, it’s super easy to install:
pip install s3-bucketConfiguring the S3 Client
To access your S3 buckets, you’re going to need an AWS secret access key ID and the AWS secret access key. I wrote a method that you can pass these to in order to configure the client so that you can use your buckets. I strongly suggest not hard-coding these values in your code, since doing so can create security vulnerabilities and is bad practice. Instead, I recommend storing them in environment variables and using the os module to fetch them:
Using the S3 Client
I designed the S3 Client’s API to be logically similar to how AWS structures S3 buckets. Instead of messing around with botocore’s Client, Resource, Session, and Object APIs, there is one, simple API: the Bucket API.
The Bucket API
The Bucket API is simple and provides most of the basic methods you'd want to use for an S3 bucket. Once you've initialized the S3 client with the keys as described in the previous section, you can initialize an Bucket object by passing it a bucket name:
Once you’ve done that, it’s smooth sailing — you can use any of the following methods:
Custom Exceptions
As I mentioned earlier, the way that botocore raises exceptions is somewhat arcane. Instead of raising different types of exceptions to indicate different types of problems, it throws one type of exception that contains properties that you must use to determine what went wrong. It’s really obtuse, and a bad design pattern.
Instead of relying on your client code to decipher botocore’s exceptions, I wrote custom exception classes that you can use to handle the most common types of S3 errors.
To use these exceptions, you can do the following:
Examples
Below I’ve provided an example of a couple of use cases for the S3 Client.
Uploading and downloading files
This example shows how to upload and download files to/from your S3 bucket.
Storing and retrieving large blobs of text or data
Conclusion
I hope that you find this as useful as I did! Let me know what you think in the comments below.
