Leveraging Hybrid Cloud Architectures to Increase Velocity
AWS is transforming the way businesses do business. In his re:Invent 2019 keynote speech, Andy Jassy effectively laid down the gauntlet for businesses who have not yet migrated to the cloud. Jassy dismantled many common cloud migration obstacles , and unveiled the general release of AWS Outposts. With AWS Outposts you can use native AWS tooling on-premises, as well as take advantage of tools like AWS Storage Gateway and AWS Data Sync for sharing assets between on-prem and cloud native applications. In this article we are going to examine how we can apply some of these AWS services to create hybrid architectures that fuel modern web to print stacks.
Scalable Rendition Services
All web to print stacks require some sort of rendition service to create a print ready PDF that accurately reflects what the user created in the web application. The premier technology that provides high fidelity between web and print services is Adobe’s InDesign Server (IDS). Specialized developers will create scripts that can be executed using IDS to produce print ready PDFs with stunning quality. There are some limitation with IDS though, not the least of which is its ability to scale to meet demand. Most printers have irregular workloads that tend to peak at certain times of the day, and during a specific times of the year. This makes capacity planning a challenge, and often printers pay for peak capacity 24/7 instead of scaling workloads up to meet demand. Adobe also does not provide a load balanced job queue for InDesign Server. This further complicates things as IDS is a single threaded application that will often crash if it’s overloaded with too many requests.
To solve these limitations a process manager for IDS is created. These process managers run on a shared EC2 instance, with each process managing exactly one IDS server instance. The job of the process manager is to feed its IDS instance rendering jobs, and to restart the IDS process should it become unhealthy. This architecture has the additional advantage of placing IDS inside the VPC, and removes the need for the rendering server to be publicly accessible. This is due to the fact that the job queue is now the primary means of driving rendering jobs. Below is a diagram of this architecture.
The number of processes to run per server is configured using an environment variable. When the server starts up a preconfigured number of processes are spawned, along with their corresponding managed IDS server instances. In this way IDS can be vertically scaled on the EC2 instance.
While this architecture scales really well vertically, there are times when horizontal scaling is required. To scale horizontally we can leverage a job manager that periodically checks the health of the job queue. If it detects a backlog, the job manager will spin up additional EC2 instances using an Amazon Machine Image (AMI). When the new instance spins up a start up script licenses IDS on the server. The job manager itself can be a scheduled Lambda, or a container service launched using Elastic Container Services (ECS). Using Lambda or ECS ensures the job manager is not a single point of failure for the application. Below is the complete architecture of the solution showing both the running rendition servers, and the job manager as a schedule Lambda.
It’s important to note that while this appears to be a cloud native solution, AWS Outposts (described below) allows you to run this solution on-prem with the same cloud native tooling.
On of the most exciting services to go into general availability this year is AWS Outpost. Outposts are physical rack mounted hardware preconfigured to your own specifications that integrate natively into your existing AWS control plane (AWS Console). Outposts can be deployed to existing VPCs and subnets, and look and feel like any other EC2 instance. They also come preloaded with existing AWS services. In addition outposts can support the same CloudFormation templates and automation scripts you’re using on AWS. This is a game changer for companies looking to deploy hybrid applications. For example 3D modeling and print rendition services often have separate automation stacks and code pipelines due to this limitation. But with AWS Outposts you can get the same low latency benefits while leveraging a single IOC code base and deployment pipeline. And because Outposts look and feel like native EC2 instances, there is absolutely nothing you need to change about your cloud application architecture.
There are some network architecture and security considerations though. To connect your AWS outpost to your VPC you will need to configure a site to site hardware VPN, or leverage AWS Direct Connect. In addition you are responsible to securing the physical servers. A sample network architecture for AWS Outposts is below.
One last consideration is asset storage. Most web to print applications that run in the cloud today leverage S3 for assets storage. Outposts does not support S3 yet. To ensure your dependent assets are accessible on-prem with the same low latency you expect leverage a hybrid storage solution (describe below).
Hybrid Storage Solutions
AWS Storage Gateway is the storage solution printers have been waiting for. It allows you to sync files between S3 and your local file system, providing low latency access for your composition and rendition services. By leveraging the power of S3 you can maintain limitless cloud storage (with multiple access tiers), and reduce your use of local storage to only active files. Storage Gateway maintains a local cache for “hot” files. You can tune this cache, or opt for a fully redundant copy. Storage Gateway has several modes for specific use case:
- File Gateway: “File gateway offers SMB or NFS-based access to data in Amazon S3 with local caching. It can be used for on-premises applications, and for Amazon EC2-resident applications that need file storage in S3 for object based workloads.”
- Tape Gateway: “AWS Storage Gateway offers IT organizations a seamless way to transfer backup jobs from tape or Virtual Tape Library systems to the cloud — while keeping trusted backup tools and processes in place.”
- Volume Gateway: “The Volume Gateway provides either a local cache or full volumes on-premises while also storing full copies of your volumes in the AWS cloud. Volume Gateway also provides Amazon EBS Snapshots of your data for backup, disaster recovery, and migration.”
- Hardware Appliance: “You can deploy the AWS Storage Gateway with an on-premises hardware appliance. If you need to connect local applications to cloud storage, but have limited virtualized infrastructure or IT infrastructure staff, the hardware appliance provides a simple procurement, installation, and management experience.”
The use cases I see most often would be supported by running Storage Gateway in File Gateway mode. Printers often only need access to a small number of files for production runs, then archive them for future access. This is the ideal use case for Storage Gateway. One of the biggest headaches for printers in maintaining scalable durable storage. Amazon has made it virtually impossible for printers to ignore Storage gateway as it provides eleven nines of durability with the same low latency benefits at a faction of the costs.
If you are not preparing to move to the cloud, you should to ask yourself how your business will compete with a someone that does. If the competition doesn’t scare you, then profits should. By not leveraging the cloud your business is likely leaving money on the table in the form of features that were never shipped due to resources that we diverted to supporting on-prem infrastructure. AWS can save you time so your business can do more for its customers. In addition AWS can make it easier to scale your business by leveraging AWS certified engineers. Many home brew data centers are held back by knowledge sharing and documentation as much or more than raw costs. Make 2020 the year you embrace AWS!