AWS S3 ‘Access Denied’

*Note: Not a medium member? Use this link for the full article.
**Note: If you want to generate or validate data, take a look at Data Caterer (Github repo here).

Permissions required for Spark to push a file into S3

— Why
— Setup
— Testing
— — PutObject
— — GetObject
— — ListBucket
— — DeleteObject
— Checklist
— Conclusion

Why

AWS S3 is one of the main infrastructure components that is the foundation for many Data Lake designs. As a result, being a Data Engineer, you will most likely come across this being used in some way, shape or form have to interact with it to push or pull data. Now, throughout my time, I have run in various issues with accessing data, especially relating to Access Denied. At first, you think its simple. I don’t have the permission to access the required resource. This is true. But the number of things that have to be in place before you can access said resource is not always clear from a developers perspective.

At the bottom, there is a checklist that I have compiled over time as I have run into issues that I hope can be helpful to others facing similar issues but the focus on this article will be the policy permissions. There is this resource from AWS itself which goes…