Using S3 Just Like a Local File System in Python
There’s a cool Python module called s3fs which can “mount” S3, so you can use POSIX operations to files.
Why would you care about POSIX operations at all? Because python also implements them. So if you happen to currently run a python app an write things to a local file via:
with open(path, “w”) as f:
write_to(f)
you can write this to S3 simply by replacing it by:
with s3.open(bucket + path, “w”) as f:
write_to(f)
. Of course S3 has good python integration with boto3, so why care to wrap a POSIX like module around it?
Why would you want to do that?
For me, the question is what kind of model you want to have. s3fs supports a simpler implementation, as you usually move from storing local, to storing in the cloud. This change doesn’t require you to change how you think about your app and your data.
If you happen to hit the limits of s3fs it of course makes sense to either sync data in batch to S3 or use a different form of loading your persistent data.
My prime use case for this is machine learning. Loading and saving models in evaluation stages and loading and saving data in various stages…
