BREW : More about Multi-Tenancy
Better Engineering Weeks (BREW)
Something’s… brewing 😉🍲
Since Q3 2022, ShopBack has dedicated two weeks each quarter to our Engineering team for Better Engineering Weeks (BREW). This initiative aims to tackle high-impact projects that cross team boundaries, driving significant progress in engineering investment areas and fostering greater collaboration across the organization.
The Whys :
- Enhancing Engineering Life : We strive to make our infrastructure more stable, operable, and cost-efficient;
- Planning for the Future : We aim to proactively plan for our technological future;
- Building Collaborative Teams : We believe in fostering collaboration by working together on impactful projects.
In this series, ShopBack Engineers will reflect on key BREW projects which have then become core initiatives that have transformed how we work and how the Group operates — and it all started… with a little brew ☕ #OwntheProblem
More about me
Hi, I’m Joyce, a Software Engineer in ShopBack! I have been here for a little over two years now, and am currently in the Merchant team supporting billing and reporting operations.
Throughout my time in ShopBack, I have worked on many projects revamping our merchant onboarding and billing systems to align with the rapid growth of our merchant base; these systems have scaled from originally operating in 2 markets to now serving merchants in 12 markets.
Currently, we are focusing on initiatives to introduce new billing and reporting models, while implementing a more cost-effective way of running the system. These efforts are aimed at enhancing support for ShopBack’s merchants and preparing for future growth and scalability.
Here are some of the ShopBack engineers during Offsite 2022 at Zouk Singapore!
More on Multi-Tenancy and My Role
Last year, I worked on the Multi-Tenancy (MT) project alongside the entire Engineering department. The primary goals for MT were to reduce engineering infrastructure costs and consolidate our technologies for more efficient scaling to new markets.
In addition to migrating to region-based environments from single-country counterparts, the Engineering team also needed to transition to Aurora PostgreSQL-managed databases to further reduce resource costs.
My responsibility in this project was to oversee the MT migration of ShopBack’s merchant platform services with my team. A smooth and bug-free migration of these services was of utmost importance, as they support merchant operations and facilitate ShopBack Pay transactions. Both roles mentioned are critical paths that serve the internal and external users of ShopBack daily, and it was not something we could afford to get wrong or delay.
Biggest Challenge the Team faced
During the planning phase, my team decided to deprecate the legacy Merchant service rather than migrate it to the MT model. We wanted to take this opportunity to rewrite the logic from this service on the newer Merchant Information and Billing platforms. The advantages included fewer services to maintain post-migration, cost reduction from spinning down the service, and faster migration rollout with one less dependency, all of which contributed to the project’s overall success.
The biggest challenge I faced was coordinating the migration of data stored in the legacy Merchant service database to the newer platforms’ databases. As part of the MT rollout, we planned to test the robustness of the endpoints from our migrated services in production before performing the database switch.
We achieved this by using our legacy service to continue pulling data from upstream sources and streaming it over to our migrated database, while endpoint traffic was directed to the corresponding migrated services. Since we were rewriting parts of the legacy Merchant database, streaming posed a risk to data reliability due to significant schema differences between the old and new databases.
After various discussions and analyses, we decided to first stream the data to temporary tables in the new databases, and utilize data transformation cron jobs to sift through and patch this data to fit the new schema. We also had to get various approvals from our stakeholders, since the patching process meant a slight delay to our data being processed during the initial rollout.
Using this method allowed us to isolate the streaming process and data transformation processes, making it easier to debug and experiment flexibly with custom transformation controls. Our goal of not having any data loss was eventually achieved post-migration.
Staying Aligned & Motivated
This project was on a whole other level compared to our usual engineering projects, with challenges aplenty. Since the whole Engineering team in ShopBack was working on the MT migration project, there was communication and coordination overhead.
However, as most teams shared common issues and migration strategies, we established various SOPs and communication channels to facilitate as much collaboration as possible. This resulted in many cross team discussions and experiments to solve common challenges together.
Collaborating across teams was very motivating, as Engineers could rely not only on their direct team for support, but also on ShopBackers from other teams. Lastly, seeing the entire Engineering team hit our migration goals together week by week was truly a strong morale booster.
Growth lies outside one’s Comfort Zone
I am very proud of the Engineering team’s ability in navigating ambiguity throughout this project. At the start, many of us felt daunted by the project’s big goals, as there was no precedent for this in ShopBack’s history. By working together, thinking outside the box at times, and making reasonable trade-offs, the Engineering organization was able to deliver the project successfully within the deadline.
Personally, I was pushed out of my comfort zone frequently as I had to work with many new technologies such as Aurora PostgreSQL, canary deployments, and more. I also had opportunities to lead the solutioning for large-scale problems that I did not (and would not) usually encounter on a day-to-day basis.
Seeing my contributions from Day One come to fruition upon the project’s rollout was an extremely fulfilling experience 💪
❗️ Interested in what else we work on?
Follow us (ShopBack Tech blog | ShopBack LinkedIn page) now to get further insights on what our tech teams do!
❗❗️ Or… interested in us? (definitely, we hope 😂)
Check out here how you might be able to fit into ShopBack — we’re excited to start our journey together!
