Just-in-time VOD packaging in an AWS environment with role-based credentials
TL;DR: The nginx+kaltura JIT VOD packager can’t perform role-based request signing for S3 to access mp4 assets. You can set up s3-helper by configuring the S3 region and bucket, and it takes care of the rest. Point nginx to use s3-helper instead of proxying directly to s3. Done.
The longer story…
AWS role-based IAM credentials is a popular way of fencing off services in AWS, reducing the need for servers to implement inter-server authentication and access controls. Instead, somewhat simplified, instances are assigned a role when spun up that determines what services they can access: other instances, virtual private clouds (VPCs), S3, etc. Only backend services which need to implement ingress and egress through fences need to implement inter-server authentication; all other services running as an AWS guest only need to sign requests using the role-based credentials provided by the VM host; the AWS SDKs do this automatically.
We use the Kaltura VOD packager here at Ellation to serve video for our two properties: Crunchyroll and VRV. The vast majority of our video content is played via AWS CloudFront which validates URL tokens (signed URLs) and then fetch objects from the VOD packagers. The vod packagers are autoscaled behind an elastic load balancer, so as our load changes additional packaging nodes are automatically added or removed as needed. Our video content lives in an AWS S3 bucket that’s only accessible to our backend services. The Kaltura VOD packager runs as a module inside nginx, effectively acting as an nginx gateway.
To fulfill a request for HLS or DASH playlists, segments, keys, or other bits and pieces needed for playback, the VOD packager generates HTTP range requests to fetch sections of mp4 files. These get proxied to S3, which responds with the range requested. The problem here is that the free version of nginx doesn’t know how to obtain the role credentials from the instance’s hypervisor-managed metadata to properly identify itself as having access rights. Theoretically, this could be added as say LUA scripting within nginx; but the stock LUA isn’t thread-safe and this calls for concurrent access to a single set of credentials. There is a thread safe LUA, but we didn’t want to add yet another scripting language, especially one nobody involved in this project has any experience with.
So we created s3-helper. This looks and tastes like an S3 instance on a loopback interface, except it doesn’t require a signature. Instead, it simply takes the request and signs it using the instance credentials and proxies the request to S3. S3 sees a request properly identifying itself as originating from a specific instance role, checks that the role has read permissions to the object, and happily returns the result. The s3-helper cleans up the headers removing everything AWS specific, and the VOD packager inside nginx can naively continue on its merry business. We only grant read permission to all playback related roles and services since they never need write access. We do however grant listing access so we get proper 404 errors if something doesn’t exist; otherwise S3 will return 403 (access denied). For development s3-helper can be replaced with another nginx instance, at least until ready to test on a live S3 bucket. In addition, the helper can be used in a similar role to solve the same issue for other software components that don’t know how to sign AWS requests using role based credentials. In addition, it makes the already complex nginx configuration entirely credentials-agnostic and moves this into a different microservice component that can be separately tested and reused. It also simplifies development setups since they can use the same nginx configuration as production systems (only with a few minor strings changed). This in turn reduces the likelihood of running into complex environment-specific configuration problems.
When explicitly configured to listen on 127.0.0.1 only services on the same instance will be able to reach it, and only by explicitly connecting to 127.0.0.1 or localhost.
s3-helper is written in go, with support for statsd/collectd and newrelic (request based only).
Special thanks to the broader Ellation team that worked on the video delivery project including @jimyoul architectural work and for Ellation’s support in open sourcing this s3-helper project. Be on the lookout for a future posts that dive into this delivery system architecture.
