Optimizing Technology for Data Processing and Integration
DemystData’s engineering team recently began a substantial redesign of our technical infrastructure that will enable us to release faster, service more clients with unique needs (e.g., custom waterfall attributes, algorithmic attributes), and process more transactions (we add less than one second of overhead across all of our different vendors, with transaction times generally ranging from 0–3 seconds).
We predicated redesigning our infrastructure on achieving three key objectives:
Immutable Infrastructure: “On the fly” changes reduce stability and slow down deployments. With our design, live production infrastructure cannot be changed on the fly by engineers, but must always be committed, tested, and deployed following strict production change protocols.
Deployment Testing: As we modify our product, DemystData employs a “Blue/Green” deployment strategy in which a modified environment (blue) is built and deployed along side the live environment (green). Once the “blue” environment is determined to be stable, the “green” environment is decommissioned.
Scalable Infrastructure: Processing data transactions requires a production environment that may be scaled up and down based on resources and cost-efficiency. Simply put, certain clients require tens of thousands of transactions per week while others only require a few. Technology should be designed to serve a range of client needs.
As a side note, while building a solution to achieve these objectives, our engineering team is making vital contributions to the Linux Container Ecosystem, particularly a recipe for running Apache Marathon to deliver resource-based scheduling on CoreOS Linux (click here to see our technical thought leadership on this topic, shared through github).
The ultimate goal of technology platforms that serve financial services firms should be to make the process easy, fast, and secure, regardless of whether that technology is for asset allocation, data management, data integration (what we do at DemystData), or any one of the hundreds of other use-cases pertaining to financial services firms.
Here at DemystData, we’ve focused on these principles while building a nimble platform that can be optimized for any size client, from bulge bracket banks to alternative lenders.
Next week, we’ll be returning to more traditional “medium data” and “finance” topics, but highlighting the technology behind our products is crucial to understanding how we can truly transform the lending space through data.