Engineering & Data Science @ TravelTriangle — Building Complex and Scalable Holiday Marketplace (Part II)
This is Part II of three-part series discussing generic and scalable framework being developed by our engineering and data science teams as well as to give you a glimpse of engineering guidelines and culture @ TravelTriangle
Tech Vision and Generic Frameworks
Please do read Part I of this series first if you haven’t, as it talked about a few generic frameworks that we have built @ TravelTriangle aligned with our product & tech vision. This part of the series would continue talking about more frameworks that we have or will be building, in the same direction. We’ll also talk about interesting frameworks being developed by our QA and DevOps teams.
Inventory Management System with Third-Party Integrations
To enable our customers to find better packages and to have greater control of the supply we have our own supply network, which includes Hotels, Activities, Sightseeing, cab, Flights, etc. Travel agents can buy all this supply from us while helping travelers plan their trips. Tech architecture was needed to handle multiple travel SKUs as well as maintaining the quality and availability of a supply that is completely fragmented and majorly offline. Further integration of new vendors should be fast and seamless to increase coverage of our supply on our product quickly.
Since the last 1.5 years, we have been proactively rearchitecting our system to handle and manage current as well as future SKUs in general so that business can kick-off any vertical anytime and that too with minimal coding. Any SKU needs to have some metadata, CRUD & search operations, booking flow (including third party integration), booking information, payments, and reconciliation as well as reports for all stakeholders to track and make data-backed decisions. Booking flow needed to take care of online as well as offline inventories.
We have created an Inventory management service(“IMS”) creating abstract entities to be extended as per SKU. For booking offline as well as online SKU, we’ve built a DataSource model wherein offline data source, as well as, as any third-party API gets integrated seamlessly for create, update, search and booking actions.
The next challenge that we faced here is to manage common entities across different inventory providers for which we had to develop a fuzzy match algorithm so as to identify & map common entities across different vendors with one master entity, so as to avoid duplicates.
Future aspects here would be to intelligently crawl and route search and/or booking queries to the right provider(s) based on the best price and/or better commission rate and thereby creating best-bundled packages for our agent(s). This would be topped up by deals/discounts as well as reward point systems (RPS) by integrating through our payment as well as RPS services, mentioned below.
Payment & Accounting Service
With growing business and transactions, handling finances keeps becoming more tedious and complex to manage timely transfers of payments; honor time-bound compliances like taxes, filing, etc.; MIS reportings; payments reconciliations for audit and tally purposes as well as to have the right picture of cash flow on a daily basis. Further, since this is the final true & real metric for any business, there can’t be any scope of error here.
Since there are multiple SKUs and booking flow, we needed to build generic payments and accounting services to take care of payments and reconciliations across all BUs seamlessly.
Payment service also needed to take care of all discounts, coupons, tax calculations, and invoices, etc. as per policies defined for each or group of SKUs, line by line. We used the adaptor pattern over our payment service to integrate any new payment gateway or wallet within 7 days, which use to take around 2 sprints of work earlier.
Further, we needed to build intelligent routing over payment gateways (PG) based on the PG success rate in realtime, PG charges, etc. Further, we also needed to manage the state of each transaction against each payment diligently as it is a matter of someone’s money.
Accounting service, on the other hand, take care of payable and receivables. Entries are immutable here and every change is recorded as credit and debit log (Double-entry bookkeeping). Further, it needed to support multi-currencies for our international agents. Post developing this, reconciliation, as well as finance reportings/cuts, become a breeze for our finance team while maintaining financial data sanity all the time.
Reward Point System (RPS), Wallet & Redemption protocol
How can we create systems and gamification modules that will enable the actors to work in a specific desired way in the marketplace and; also to reward the good and weed out the bad actors in the system?
We created a generic Point-Badge-Leadership (PBL) framework as a separate service, interacting with other micro-services through the event stream. Any event that would be triggered from any of the micro-service will get listened by this service, processed through its rule engine and accordingly, events are emitted to add or subtract reward from the wallet (in double-entry bookkeeping style) within wallet service.
The wallet service is in turn integrated with our payment service to redeem these points through discounts at the time of booking any holiday package or to process it for the agent(s) as incentives over each booking, again as specified in the rule engine.
The business team can create and run campaigns on-the-go using UI interface and tech bandwidth is not needed anywhere now except when more redemption methodologies to be added in the wallet service.
QA Automation — High-Quality Checks with least effort/time
QA efficiency is something most of Engg heads tend to skip and once ignored for long, it comes biting in the back massively and creating ripple effects either in release quality or its turnaround time. Any bandwidth or resources would not be sufficient if we deliver low-quality releases and hence would always be busy just fixing what we have developed. Also until unless our manual work keeps getting automated, the need for engineering bandwidth will keep increasing with each release while release cycle time getting increased as well.
We @ TravelTriangle have been very cognizant of this fact and always devote some of the engineering bandwidth to make sure that we deliver quality work in production (so as to minimize debt) as well as work on automating things in parallel.
DevOps and QA turnaround time are two very essential part of this.
Our QA teams are no lesser than any development team and always keep thinking quite out of the box to automate and optimize their manual testing. They have built in-house frameworks for the below aspects:
Furthermore, to even reduce the time of automated tools & analyzing failures, they have optimized their test-cases execution, as well as, worked on mitigating the brittleness of UI tests.
DevOps Automation — Improving engineering team & release efficiency
As mentioned earlier, this often takes a backseat in terms of prioritization and with increasing product lines, servers and automated testing frameworks, etc., DevOps automation keeps becoming more critical. DevOps team focusses on below aspects to automate things for others accordingly:
Improve the efficiency of operational processes & deployment time:
- Automated a lot of operational processes by building Jenkins CI/CD pipelines. Further, deployments are integrated into a Jenkins CI/CD pipeline using Ansible for around 120+ servers (at peak time) including staging machines.
- We face no downtime during deployments.
- Complete deployment time (on all servers) which was dev team major concern is now ~14–15 min from 35 minutes (which was earlier an hour), while the team is working further to reduce it within 7–8 mins in the coming weeks.
Internal tools for engineering team efficiency:
- DevOps is responsible for setting up monitoring and alerting infra over which engineering team can configure their metrics.
- We have set up ELK stack to stream production logs to central elastic search for teams to quickly search and debug issues, if any, in an almost realtime fashion.
Frequent backups, replica and disaster recovery:
- The team has been responsible to make sure that we have periodic backups and/or replica of all our storages, as well as recovery time, is minimal, in case of disaster.
- Our recovery times for most of the storages except Elastic Search are in minutes now. Elastic Search complete recovery time, though, has been brought down within 1.5 hours from earlier 1 day while the team is working on reducing it even further getting it within 30 minutes, as the next milestone.
Infra Cost optimization:
- Our internal benchmark is to not let infra cost increase more than 1.5x for a 5x scale in requests. Every new topology/server/tool is evaluated under this benchmark.
- The team has done massive cost optimization in the past using Spotinst, Imagekit (replacing AWS CloudFront as CDN + Cloudinary as on-the-fly image processing), smart auto-shut strategies for staging machines, auto-scaling, etc. Our infra cost has just increased 1.2x in the last 1 year, despite our request going up 4x times as well as a lot of new topologies/services getting added up as part of re-architecture.
- We are currently working on moving our applications to Docker and orchestrating dockers through Kubernetes to even utilize our machine(s) more effectively and reducing costs further.
Access control & Audit Trails:
- The team has configured google authentication over all our tools so as to link with the company email account.
- The team has set up a VPN on-premise to access servers only through VPN clients.
- Identity and access management system(FreeIPA) has been set up to manage all our users centrally from a single console, Create custom policies for authorizing users to access specific servers as per requirement, Restrict their command execution rights and track their activities on servers.
Future Tech Frameworks
While we have discussed a lot of frameworks being developed, we still have a lot of miles to travel yet before realizing our tech vision fully, to building a complex and scalable holiday marketplace. Few interesting things, already in development, as of now:
- automated chatbot between travelers and agents (blog coming soon…)
- Working on various data-modeling to prioritize tasks, predictive dialing for our agents to take things even further levels up, suggesting to them what task to perform on what lead and when so as to minimize human-based decisions to only where they are actually needed as well as reducing the effort of our agents massively.
- We are also working on building an AutoML like framework (blog coming soon…) which can run multiple data models in parallel on structured data. The first version has already reduced our final modeling turnaround time (structuring, modeling and then integrating) from 50 days to 15 days while we intend to take it to 7 days by March/Apr’20. Not to mention saving quite our team’s bandwidth as well, but investing just in analyzing the accuracy across the various models and/or analyzing A/B test results across different models.
- Personalization and recommendation systems around packages along with price prediction through a high volume of data acquired over the past many years.
Part I of this series talked about a few generic frameworks that we have built @ TravelTriangle in alignment with our product & tech vision. This part has mentioned even more frameworks that we have or will be building, in the same direction. Part III (coming soon..) will deep dive into aspects around people, guidelines, hiring methodologies and other things that work behind the scene but are founding pillar to everything else.
A lot of interesting things happening @ TravelTriangle. Join us in solving these challenging problems and creating a world-class holiday B2C and B2B ecosystem; email at lead_on@traveltriangle.com
If you know anyone in your network who might be interested in solving these problems, do share this article.
If you like the article, do like and/or clap so that others might stumble upon this article.
Originally published at https://www.linkedin.com.
