Using signed URLs to simplify data uploads
Signed URLs are a mechanism that allow users of your system to upload directly to the cloud. In this blog post, we’ll explore the core architectural problem of data uploads, and how leveraging signed URLs can benefit both performance and operating costs by minimizing upload latencies and network bandwidth through your systems.
However, before diving into signed URLs and their trade-offs, we need to share a cursory understanding of the problem they solve and how this problem has been traditionally approached.
The core design tension of data uploads
Allowing users to introduce data into your system for storage and/or processing is an extremely common requirement. For example, at Miro, we strive to supercharge the ability of our users to collaborate. We do this in large part through enabling them to upload, visualize, and enhance many forms of data on interactive whiteboards. We also make heavy use of data uploads within the different internal systems that make up Miro, and within the tools we use to build and maintain it.
However, despite how common this requirement is, there’s a core challenge with data uploads that every system needs to consider: keeping data secure.
The single most important responsibility we have to our users is to protect the information they entrust to us.
This introduces tension in our architecture between giving users the ability to introduce new data into our system, and preventing unwanted access of other data in our storage.
Securing uploads behind a dedicated API
The design tension between the ability to mutate a datastore while keeping it secure long predates cloud storage and signed URLs. So how has this tension traditionally been resolved?
One approach commonly used by many systems, including those at Miro, is to have a dedicated API responsible for exposing limited, secure actions, which external users can invoke to upload data.
This pattern solves our design problem by helping to ensure that only specific internal systems are allowed to interact directly with storage, and that any incoming requests that may affect storage are first validated.
The main strength of this approach is also its main weakness: all uploads need to flow through the API. This can introduce large amounts of latency, especially if the API is hosted in a different location to your storage. Depending on how the API is implemented, this approach may also need to scale to handle the full bandwidth of uploads, which is heavy on both computational and networking resources.
With the mass adoption of cloud storage, an alternative architectural pattern has become viable which moves your system out of the way of uploads by leveraging a common capability of cloud providers: signed URLs.
Secure, low-latency uploads using signed URLs
A signed URL is a URL to a specific item in cloud storage that contains a signature made using your credentials for that cloud service provider. For example, if you use AWS Cloud, you would use your AWS credentials to sign the URL.
The signature allows a specific action to be executed on that cloud-resource URL for a specific timeframe. For example, a resource can be uploaded by performing a one-time POST to a signed URL with a signature that enables this POST for 30 seconds. The action is further bounded by the permissions of the credentials used to sign the URL. For example, a URL signed by read-only credentials cannot be used to upload data.
The key takeaway is that signed URLs decouple the location where permissions are granted from the location where the permissions are used.
A sub-takeaway is that signed URLs can be created without needing to make calls against a cloud provider.
Now that we know what signed URLs are, let’s dive into a couple architectural patterns that leverage them.
Architectural example: upload from browser client
In our first example, let’s enable a web-app user to upload an image from their browser.
In this setup, signed URLs are generated by the backend and returned to the client, which uses the URL to upload large files directly to the cloud. Access to the storage remains secure through the dual layers of the initial API call to request the URL which still enforces user authentication, and the scope of the URL which limits the upload to a specific action on a specific object.
Architectural example: upload from internal services
A second example is in the context of a service-oriented-architecture where a service — the ReportService — has the responsibility of generating a report composed of data from other services.
This architecture uses signed URLs to enable multiple services to upload data to a central storage without those services needing to know anything about that storage.
Trade-offs to using signed URLs
As with all technological choices, there are trade-offs to consider when deciding whether signed URLs are the right choice for you. Here are few, but this list is by no means exhaustive.
Signed URLs can be used by anyone while they are valid. If a signed URL is leaked to a bad actor, they will act on the item during the lifetime of the URL. It’s important to minimize the time-to-live of the URLs you create, and it’s important to control access to actions which create signed URLs.
Signed URLs do not perform data validation. You can upload essentially whatever you want using a signed URL. This can open the door for abuse of uploading large objects. This can also cause problems if the data uploaded has an unexpected format. To mitigate this you would need other mechanisms, for example limits on storage buckets, or data validation when processing uploaded data for the first time.
Using signed URLs does not remove the need to have an authority for storage access. You still need a component in your architecture that is responsible for generating your signed URLs and it needs to be protected to the same extent as other approaches. The main difference with using signed URLs is that this component doesn’t perform the upload itself.
Authorization for a signed URL occurs when it is used, not when it is created. This could result in unexpected behavior if the permissions of the credentials used to generate a signed URL change between when the URL is signed and when it is used. It’s also important to note that successfully generating a signed URL doesn’t guarantee that the request to the URL will succeed.
Implementing signed URLs
If you’re curious about playing with signed URLs yourself, all major cloud providers offer signed URLs for their storage solutions. I encourage you to have a read through your provider’s documentation.
There are also many online resources that step you through creating and using signed URLs in multiple languages and platforms which are within reach of a quick search.
Related topic: signed cookies
If you’re building a browser experience, I highly recommend reading up on a conceptually-similar mechanism called signed cookies. Unlike signed URLs that give the ability to interact with one resource, signed cookies can be used to authorize multiple requests. This can be particularly useful when interacting with CDNs, where there’s often a need to read or upload batches of resources.
Signed cookies are not as supported amongst cloud providers as signed URLs, and they have their own nuances and trade-offs which are beyond the scope of this article. I encourage you to reference your cloud provider’s documentation for details around support and usage of signed cookies.
Conclusion
There you have it. We’ve done a cursory exploration of how signed URLs can help simplify data uploads, explained their trade-offs, and shared a couple examples that illustrate how they can be incorporated into the architecture of our systems.
This, however, is only scratching the surface. Have you used signed URLs to solve interesting problems? Do you have ideas or a different approach to using them? Let us know in the comments — we’re interested in what you have to say!
Oh, and we’re hiring! Check out our open positions if you’re interested in joining the Miro team.
