Bypass the Cloud Run 32mb Limit with Cloud Storage
Cloud Run imposes a limit of 32mb for HTTP/1¹ requests and responses. Google Cloud Storage, however, allows for objects up to 5 TiB. You might wonder, how can I work with objects larger than 32mb when using Cloud Run? Using Cloud Storage with Cloud Run requires some techniques to bypass this limit. This blog post details our recommendations for performing uploads and downloads larger than the 32mb limit.
All the below recommendations leverage signed requests. Signed requests are authenticated endpoints (URLs) to which you can make requests directly to Cloud Storage. They remove the need for your Cloud Run application to proxy requests to Cloud Storage objects. This may reduce the load on your application if your application is media heavy (for example, if it contains large videos). You’ll need to generate a new URL for each file you want to upload or download. These endpoints will expire after a certain amount of time, which can be specified on URL creation. See the Cloud Storage documentation on signed requests for more information.
Uploads
The recommended option to bypass the Cloud Run limit for uploads is signed policy URLs.
Using signed policy URLs for upload
Using signed policy URLs allow your website to serve a form which can be used to upload files (via POST request) larger than the 32mb limit.
See Upload an object with HTTP forms to find more information and examples in other languages.
Downloads
There are two recommended download options for bypassing the Cloud Run limit: signed URLs and streams.
Using signed URLs for download
If your Cloud Run application presents end-users with a website, you can use signed URLs to download large files directly from Cloud Storage. To perform a GET request with a signed URL, its action must be set to read. (Note this is different than uploads, which require write.) For example:
Now that you have a read signed URL, you can use it to make requests directly with Cloud Storage, bypassing Cloud Run entirely. Below is an example of using this URL in your Express.js web application.
Note that by modifying the Content-Disposition on your file, you will be able to change the behavior of the GET request. Visiting the generated URL with Content-Disposition not set will result in the video rendering on the web page with playback controls. If you’d like the video to be downloaded instead, you can set the Cloud Storage object metadata field Content-Disposition to attachment; filename=”video.mp4". This will result in the file being downloaded as “video.mp4”.
For more information about the Content-Disposition field and how it can be modified on your file hosted on Google Cloud Storage, refer to the Cloud Storage documentation.
Using streams for download
If your Cloud Run application is processing data stored in Cloud Storage, we recommend downloading that data through a stream. Streams are a programming construct used by major programming languages to send and receive chunked data. Streams allow you to break a file into parts that are smaller than 32mb. The example below creates a stream that reads the contents of the file from Google Cloud Storage and pipes it to res to be displayed on your web page.
View the full code for these download samples in this gist.
Learn More
Now that you’ve read this blog post, you can leverage the full power of Cloud Storage with Cloud Run to bypass the upload and download size limit. 🎉
For more information on uploading and downloading data with Google Cloud Storage, see the Cloud Storage documentation on uploads and downloads. For next steps (like deploying your app to Cloud Run), see the Cloud Run documentation.
[1] There is no imposed size limit on HTTP/2 requests
