Slamming Together Stacks and How it Reduced our AWS Costs by 75%

Shane Fast
Aug 27, 2017 · 7 min read
Squish

Remember moving out of your parents house and having to begin paying for your own shit? We had a similar experience moving out of Cybera into AWS. Our production was beginning to generate revenue, white label sales started covering chunks of our burn and after enjoying far beyond our initial allowance (thanks a bunch for that Cybera, can’t express that enough!) we decided to do the honorable thing and offload our production nodes to AWS to free up free resources for new hustlers and researchers.

Its one thing to ball park an estimate for server cost, quite another thing to get the first bill. We quickly realized that our sloppy habits had to come to an end. No more leaving nodes on for “experimentation”, storing an unreasonable number of images, or using the larger servers just in case we get a surge of traffic. In seeking ways to reduce costs we were lucky to come across some sage advice which made the largest dent very quickly. Simply put — combine all of our stacks into a single stack and use host headers to differentiate which environment variables and database connection to use.

For context, we erected several stacks as white label deployments extremely rapidly in order to take advantage of time sensitive business opportunities. We deemed it as an acceptable source of technical debt in order to further our business prospects. While containerized, some configurations still needed to be setup manually along with updating deployment pipelines and branding customization. The initial setup for each new stack took 2–3 hours and probably 2 additional hours a week in maintenance for each new stack (propagating infrastructure changes, manual tasks, swapping feature flags, etc…).

So what did it take to make the switch?

A metaphor I like to use when combining our stacks is moving home owners into an apartment complex. Before each home owner had their own utilities, their own storage space, and they can assume any visitors arriving are meant for them. On the other hand, once moved into an apartment complex each tenant shares the utilities, storage space, and there needs to be a mechanism for visitors to get to the correct tenant.

Sharing Utilities

This includes everything from node modules to third party resources to server installations. It really helps to have a good inventory of what utilities get affected when making these changes. To be fair, something will likely slip through the cracks, so ensure that a thorough QA process is followed as well.

For example, we use Auth0 and Stripe in each one of our deployments. We needed to consider whether or not to create new clients for each tenant or to simply allow each tenant to use the same clients for these services. By default we decided to have each tenant use the same Auth0 client, but use different Stripe clients unless requested otherwise by our customers.

Shared Storage:

A concern that white-label customers and apartment dwellers have in common is whether their things in storage is safe and won’t be intermingled with other people’s stuff. We all know that most apartments have dedicated storage behind lock and key only available to the tenant and the building manager/owner. This is contained in the same building and Tenants can rest assured that no one else will be sifting through their stuff (Hopefully). Much in the same way each of our customers can rest assured that their data is safe behind stringent security practices even through its stored with other customer’s data within the same infrastructure.

In our system we simply add a new database user with access to a new database for each new tenant. On the application side of things we needed to initialize connections to each database when the application starts up. This turned out to be the most challenging part of the build for us because we took the wrong approach at first. Our first attempt was to create a new database connection when ever a request was made (DO NOT DO THIS!).

Gaius Baltar knows what I mean

You will need to manually close connections and this significantly harms performance since you are initializing a network connection with each API call.

The better way is to initialize each connection only once at the startup of the application and simply direct the endpoint to which connection to use based on the host header:

  1. Create a new database connection file
var Env = require('../env.js'); //File Pulling in Environment Variables
var mongoose = require('mongoose');
// DB Authentication Information
var dbUser = <database username>; //example_dot_com_user
var dbPass = Env.<name of password variable>; //Keep this value protected
// DB Location Information
var dbHost = <tenant URL>; //example.com
var dbPath = <name of tenant's database>; //example_dot_coms_db
var dbPort = Env.dbPort; //usually 27017
// DB Authentication Methods
var dbAuthMech = "SCRAM-SHA-1";
var dbAuthSrc = <name of admin database>; //"admin" by default
var dbAuth = "authMechanism=" + dbAuthMech + "&authSource=" + dbAuthSrc;// Build Connection String
var dbConnection = "mongodb://" + dbUser + ":" + dbPass + "@" + dbHost + "/" + dbPath + "?" + dbAuth;
// Connection to DB
module.exports = mongoose.creatConnection(dbConnection)

Notice that mongoose.createConnection() is used rather than mongoose.connect(). Explore these links to get a better understanding of the difference here.

2. Create a model object file

const PaperLTS = {};
const LTS_connection = require('./path/to/connection_file');
const Model1 = require('./path/to/model1');
const Model2 = require('./path/to/model2');
.
.
.
const ModelN = require('./path/to/modelN');
PaperLTS.Model1 = LTS_connection.model('model1', Model1);
PaperLTS.Model2 = LTS_connection.model('model2', Model2);
.
.
.
PaperLTS.ModelN = LTS_connection.model('modelN', ModelN);
module.exports = PaperLTS;

Note that if you add new objects to your application you will also need to update each model object file as well.

3. Require the new model object file in app.js

const Tenant1Connection = require('./path/to/model1_object_file');
const Tenant2Connection = require('./path/to/model2_object_file');
.
.
.
const PaperLTSConnection = require('./path/to/LTS_model_object_file');

Mechanism to Direct Visitors:

Now here’s where the heart of the implementation is. Once we have created the above configurations our application need to be able to direct users to use the correct database and the correct services. To accomplish this we created a tenant middleware which is called on the beginning of every route:

_tenantMW: (req, res, next) => {
return new Promise((resolveAll, rejectAll) => {
//1. Issue Tenant Variables
TenantLIB._selectTenant(req.headers.host)
//2. Attach Tenant Variables to request
.then((tenantVariables) => resolveAll({ tenantVariables: tenantVariables}))
.catch((err) => rejectAll({err: err, msg: "Error - Obtaining Tenant Variables"}));
})
.then((ret) => {
console.log("URL --------------> ", ret.tenantVariables.paperUrl);
req.tenantVariables = ret.tenantVariables;
next();
})
.catch((err) => {
console.error(err);
res.status(404);
res.json({
type: "ERR",
msg: err.msg,
});
});
}

The function that Issues the Tenant Variables (_selectTenant) is simply a promise wrapped switch case:

//Environment variables
const Env = require('../env.js'); //File Pulling in Environment Variables
const tenantLib = {
_selectTenant: (origin) => {
return new Promise(function(resolve, reject){
let tenantVariables = {};
switch(origin) {
case "https://secure.paperlts.com/static/":
case "https://www.secure.paperlts.com/static/":
case "https://secure.paperlts.com":
case "https://www.secure.paperlts.com":
tenantVariables = {
paperUrl: "https://secure.paperlts.com",
paperDb: Env.<DB name>,
paperDbUser: Env.<DB User Variable name>,
paperDbPass: Env.<DB password variable>,
poweredBy: "paper LTS",
defaultLogoURL: <URL or file path to Logo>,
defaultImagePath: "path/to/image/folder/",
.
.
.
dbConnection: "PaperLTSConnection",
}
break;
case "https://example.com":
.
.
.
default:
reject("Origin Invalid");
}
}
resolve(tenantVariables);
})
},
}
module.exports = tenantLib;

Now we have specific Tenant variables available in the request object for all end points to use. The only thing left to consider is how we invoke mongoose to take actions on specific databases. Notice how the value for dbConnection is the same for the variable called in app.js. This is intentionally done in order to match the connection while calling any mongoose methods.

In our design we centralized all of our mongoose methods into a single library file. Each method is also wrapped in a promise. I highly recommend this structure because it allows your higher level methods to compose mongoose calls. This pattern also happened to make it really easy to implement our host header separation because we only had to rewrite one file:

//Environment variables
const Env = require('../env.js');
//Aggregating Connections
const Tenant1Connection = require('./path/to/model1_object_file');
const Tenant2Connection = require('./path/to/model2_object_file');
.
.
.
const PaperLTSConnection = require('./path/to/LTS_model_object_file');
const connectionList = {
Tenant1Connection: Tenant1Connection,
Tenant2Connection: Tenant2Connection,
.
.
.
PaperLTSConnection: PaperLTSConnection
}
const mongoosePromises = {
// Promise Based Find Method
_dbFindAllPM: (database, collection, request, options) => {
return new Promise(function (resolve, reject) {
connectionList[database][collection].find(request, options, function (err, data) {
if (err) {
console.log("Error Occured");
reject(err);
} else if(!data) {
console.log("No Data Found");
reject(data);
} else {
resolve(data);
}
});
});
},
.
.
.
}module.exports = mongoosePromises;

This just shows one example of a mongoose method wrapped in a promise. The database argument is fed the req.tenantVariable.dbConnection value which connects it to the appropriate database. The connection argument is fed the model to act on. Take the following example from an endpoint:

mongoosePromises._dbFindAllPM("PaperLTSConnection", "Contracts", {_status: "Completed", _creator: "Shane Fast"})

This will fetch all completed contracts for myself in the paper LTS database. “PaperLTSConnections” would be generalized by using req.tenantVariables.dbConnection to fetch from the correct database depending on where the visitor is arriving from.

The Benefits

After implementing this strategy we only have to add 2 new files and edit an additional 2 files for any new white label deployments (plus some third party configurations). This replaces provisioning additional stack of nodes, several manual nginx configurations, hours each week in maintenance and can shut down unused resources (heavily reducing our hosting costs).

Conservatively, I can say this reduces our monthly burn by about 5% overall. For a few days of concentrated effort to implement I definitely recommend to anyone running multiple databases or is running highly similar application servers.

Hopefully this quick run down gives you a general idea about how you can reduce your infrastructure using coding solutions. I do not necessarily argue that coding solutions are always superior to infrastructure solutions, but rather using the solution that is more appropriate to your team’s makeup. For our team this solution will be easier to maintain and add onto moving forward, but might be different for other teams. Use whichever helps your business win!

Athennian Dev Life

Bring cookies or be denied.

Athennian Dev Life

It’s easy to lose sight of what’s important in life. As developers and people deeply interested in technology, we can easily forget why we started this pursuit. These are our projects, discoveries, and stories we want to carry into the future to remember and share with the world.

Shane Fast

Written by

Co-founder of Athennian @athennian. Always interested in hearing from entrepreneurs, colleagues, and self-driven people.

Athennian Dev Life

It’s easy to lose sight of what’s important in life. As developers and people deeply interested in technology, we can easily forget why we started this pursuit. These are our projects, discoveries, and stories we want to carry into the future to remember and share with the world.