Upload files larger than 32MB to App Engine and Cloud Run with this Streamlit Component
TL;DR
Cloud Run and App Engine are Google Cloud services for deploying and managing applications. Developers using these services face a 32MB size limit for large file uploads, which is a significant hurdle for applications requiring large data uploads, such as video processing or large-scale data analysis. Methods like chunking or compression are complex and affect user experience. A solution involves using cloud storage signed URLs for file uploads, bypassing the size limit. Although this process can complicate user experience, we have developed a Streamlit uploader component that simplifies it, hiding complexity and ensuring smooth operation. The component is available as a pip package with comprehensive documentation, allowing users to upload any file regardless of size or content type.
Size Limit of Inbound Requests
In both App Engine and Cloud Run, developers encounter a significant challenge with the strict 32MB limit imposed on incoming requests. This restriction prevents the smooth handling of larger files, triggering a “413: Request Entity Too Large” error message. Real-world applications often need to upload files much larger than this limit. For instance, video editing platforms require users to upload raw video files, which can easily exceed several gigabytes. Similarly, data processing applications often deal with large datasets, such as high-resolution images for analysis or extensive CSV files for machine learning. Despite attempts to overcome this limitation through methods like chunking or compression, these solutions often add complexity and compromise user experience. Consequently, developers are hindered in delivering features that require handling files exceeding the 32MB threshold. Given these constraints, finding a reliable solution becomes imperative for developers aiming to create robust and versatile applications on these platforms.
Solution Design
To illustrate the solution, let’s consider a scenario where we aim to develop a web application enabling users to upload CSV files containing data for processing and generating metrics. Initially, the upload process follows this flow:
However, this approach fails when file sizes exceed 32MB. To overcome this limitation, we leverage cloud storage signed URLs. Instead of directly sending files to App Engine via a POST request, we generate a signed URL. This URL allows users to upload files directly to cloud storage, bypassing the size limit. Subsequently, App Engine accesses the file from cloud storage. This is illustrated in the schema below:
Streamlit signed URL uploader
This alternative design introduces a two-step process for users, involving generating the signed URL and then uploading the file, which could compromise user experience. Fortunately, we have developed a Streamlit uploader component to streamline this process, eliminating the need for separate requests. This Streamlit component allows users to upload files to Google Cloud Storage via signed URLs. It behaves like a normal uploader, hiding all the complexity and ensuring a seamless user experience. It is available as a pip package with detailed documentation for easy integration.
Integration with App Engine and Cloud Run
Integrating this Streamlit component into your web application has two main requirements:
- Set up the bucket where you want to store the uploaded files. You can either use App Engine’s default bucket or create a new bucket, and you need to set up CORS configuration on this bucket.
- Grant the default service account of App Engine or Cloud Run the role “roles/iam.serviceAccountTokenCreator”, which will give it the necessary permissions to generate signed URLs for file uploads. You can do this either manually using the Google Cloud Console or with Terraform.
Below is an example of Terraform code for App Engine.
Note: This example is not production-ready. Customize the CORS configuration and network settings according to your application’s security requirements. It is recommended to restrict the origins, methods, and headers allowed in the CORS configuration to only those necessary for your application. Consider implementing additional security measures and conducting thorough testing before deploying to production.
You can also test the component locally by using a service account and granting it the Token Creator role.
Conclusion
Here is a video demo of the component in action :
Managing large file uploads is a critical challenge for developers using App Engine and Cloud Run due to the 32MB size limit on inbound requests. By leveraging cloud storage signed URLs and our Streamlit uploader component, developers can bypass these constraints and provide a seamless user experience.
Thank you Benoit Goujon and Guillaume Quinquet for reviewing this article.
