• Day 1 — Refactoring
• Day 2 — Code Smells
• Day 3 — Testing Legacy Code
• Day 4 — Simple Design
• Day 5 — Mob Programming

We kicked off each day with a book reading activity. Reading was intertwined with discussion and was followed by a coding kata — an exercise that reinforced learning by practice.

A short retrospective meeting to gather reflections and learnings brought each activity to a close.

Learning Week was held remote-first on Zoom and all activities were group-based —i.e. reading and programming in either one large group or several smaller teams. For these smaller groups, we used Zoom’s breakout rooms feature.

Want to run these activities for your team?

If participants are not familiar with Pair Programming & TDD, consider including relevant introductions. Participants in our case were a budding XP-team with a year of practical experience.

Day 1 — Refactoring

• Group-reading: Design II — Refactoring from Agile Technical Practices Distilled. While reading, participants discovered heuristics such as Object Calisthenics & duplication, and techniques to refactor (Safe Refactoring & Parallel Change).
• Participants were introduced to 5 atomic refactors: (1) Rename, (2) Extract, (3) Inline, (4) Move, (5) Safe Delete. A look-up table was provided with keyboard shortcuts for reference.
• Kata: Yatzy Refactoring Kata

A few thought-provoking excerpts from Chapter 2: Principles in Refactoring of Refactoring: Improving the Design of Existing Code were shared with participants through the day:

“But doesn’t the time I spend on refactoring reduce the speed of development?”
“How to tell a manager about refactoring?”
“Refactoring can be a fantastic tool to help understand a legacy system …”

Day 2 — Code Smells

Participants added Code Smells to their refactoring tool-kit.

• Group-reading: Chapter 3: Bad Smells in Code from Refactoring: Improving the Design of Existing Code, followed by Design III — Code Smells from Agile Technical Practices Distilled. Participants discovered the catalogue of refactoring recipes in Martin Fowler’s book.
• Kata: Smelly Tic-Tac-Toe — participants used the following commit convention: [code smell] message, for e.g. [primitive obsession] wrapped float as Amount.

Day 3 — Testing Legacy Code

Tests were already written for coding exercises on previous days. Day 3 asked the question: What if we don’t have tests?

• Group-reading: Chapter 1: Changing Software and Chapter 2: Working with Feedback from Working Effectively with Legacy Code, followed by Testing Legacy Code from Agile Technical Practices Distilled.
• Kata: Ugly Trivia Game — writing a Golden Master test.

Day 4 — Simple Design

Besides introducing the principles of Simple Design, we set out to build a habit of experimentation & improvement.

“After deleting and starting again enough times, I’ve often heard people say they have a feeling of freedom they’ve not experienced before.” — Corey Haines

• Group-reading: Good design? and 4 Rules of Simple Design from Understanding The Four Rules of Simple Design, followed by Classic TDD I and Classic TDD II from Agile Technical Practices Distilled.
• Kata: 4 sessions of Conway’s Game of Life, 45 minutes each. This followed the Code Retreat format: code was thrown away & pairing partners switched after each session.

Constraints

Participants were given a constraint in each session to break out of habitual code & design practices.

• Session 1 — TDD, driver/navigator rotation in five minutes (or less)
• Session 2 — ≤ 3 lines per method, no variable reassignment
• Session 3 — Evil mute ping/pong. One person is a test-redder, and the other a test-greener. The test-greener does not care about requirements and will get tests to pass any which way. Partners cannot talk to each other.
• Session 4 — No if/else, no loops, no getters (Tell, Don’t Ask principle)

We adapted constraints after each session based on progress. More constraints can be found on coderetreat.org, along with a facilitation guide.

Reflections

• Cyber-Dojo allowed for a low overhead pair programming experience, i.e. no IDE quirks, no git commands and a simple way to throw away code by closing a browser tab. The downside was the lack of refactoring support. We switched to an IDE with git for our last session.
• Groups may require a nudge to follow a constraint or discover an insight during sessions. We started with 1 facilitator and 4 pairs, before shifting to 2 mobs in the last session. The last session yielded better engagement between facilitator and groups.

Day 5 — Mob Programming

Participants chose what to do on the final day: refactoring the Ugly Trivia Game from Day 3.

A single mob formed with five to eight members throughout the day. Driver/navigator roles rotated every five minutes, with a primary-navigator selected to avoid overwhelming the driver. Drivers would be rotated into the primary-navigator role.

We experimented with mob.sh for quick handovers. It turned out to be a nifty tool: keeping time and managing the sequence of drivers.

Code changes were deleted at the end of each rotation if tests were not green, encouraging participants to use safe refactors.

Book references

• Agile Technical Practices Distilled by Pedro M. Santos, Marco Consolaro, Alessandro Di Gioia
• Refactoring: Improving the Design of Existing Code by Martin Fowler
• Working Effectively with Legacy Code by Michael C. Feathers
• Understanding the Four Rules of Simple Design by Corey Haines

A Week of Learning Agile Development was originally published in Tech @ Careem on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Careem, our Care agents resolve thousands of interactions everyday. These interactions involve over 100+ different contact reasons. A customer (here customer refers to a user — a captain, customer or a merchant) can contact us pre, post or in ride and naturally, each of these contact reasons has a different resolution. The wide variety of contact reasons are both different in terms of the investigation as well as the actions taken to resolve the contact.

When I joined Careem, back in 2018, I visited one of our call centres to understand how our care agents resolve a contact. That visit made me realise the number of manual steps that were being done were very high. Some agents had calculators open on their desktops to recalculate the ride fare. New agents had to go through documents to understand how to resolve an issue which they were solving for the first time. Only the highly experienced Care agents were made to receive calls from customers as they had the investigation and resolution steps fully memorised. To escalate an issue to the supply team they had to write an email or create a Jira ticket. At this point, you might be wondering if the handling time was skyrocketing? Well yes, it was.

Being a Care agent is a hard job especially when the other person is reaching when they are most unhappy with a service or situation. They have to make sure the customer is happy by the end of the interaction. They have to make sure that customer satisfaction (CSAT) remains high, handling time remains low and incase of payment adjustments they are just giving the right amount to the customer — enough to keep both customer and Careem happy. By mid-2018, we made a goal to simplify the lives of our Care agents and provide fast and consistent resolution to our customers.

It was not that we didn’t have a portal, we did. That portal allowed them to see customer details, trip details and even to perform payment adjustments on customer accounts. The problem was that it was a generic web-app with all the information that the agent might need. But the steps required to resolve a contact were embedded into the brains of our agents. We wanted to embed those steps into an app which would guide them towards a resolution. We wanted to create a resolution engine.

And so we started our journey to simplify the lives of our agents. We wanted to come up with an MVP and launch it as soon as possible. More importantly we wanted to gather feedback of our agents if the product really helped them and then iterate on it.

We experimented with the Lean Startup methodology. And we wanted to think of the fastest way to empower our agents. The goal was to simplify Care agents life by guiding them on every step.

Lean Product Management

Our team came up with the idea of using Google Forms in an iframe inside our web app. When I first heard of the idea, it sounded bizarre. As a junior software engineer out of university, I was new to the idea of Lean Product Management. I wanted to deliver complete features and as quickly as possible. And so I questioned why are we using Google Forms when we can create flows and forms inside our web app. My PM calmed me down and assured me that we will definitely iterate over it and improve.

Google Forms helped us in policy adherence as well. It ensured that every agent was following the same steps to resolve the same problem and that there was no personal sentiment involved. We gradually started rolling it out. Not only our Care agents were satisfied but our quality & other metrics started to improve. QA agents found a way (via Google Form responses) to validate the steps taken by the Care agents and ensure that the resolution provided was correct and followed Careem expectations. In about three months, we launched 20+ category flows on Google Forms. It was not only Google Forms, we started displaying the relevant trip and customer related relevant information beside it as well.

Google Forms had its limitations

The data filled by our agents was not easily accessible. Trivial questions like trip related information and basic customer information had to be filled manually. Policies change based on cities, car types, customer segmentation, customer LTV and so on. These are all things which our ecosystem (100’s of microservices) already know and so there was no point in asking an agent to fill this trivial information to reach a resolution. There was limited automation possible inside Google Forms. We could auto populate the form sections but we couldn’t skip those forms. And so we decided to move away from Google Forms.

The first iteration allowed us to understand the problem context further. We wanted to create something not only for our agents but also for our business operations team as well. We wanted to give them the ability to change resolution policies on the fly with little to no tech involvement. That sounded really hard to do when we thought about it at first. Embedding business policies in code is easy, if they seldom change. In our case, policies changed very often (1–2 changes every week in one flow). If you consider encoding 20+ different flows then it sure seems like a lot of work to do. Below is an example of how does a contact resolution policy looks like:

It’s a flow chart. One way to embed this policy is by using conditional statements in code. But this was not a viable option as we wanted to minimise tech involvement. Another more dynamic way to implement this is by encoding this policy on a BPM engine. We researched on the available open source BPM’s available. After careful consideration we chose Flowable due to its community support. This time it was an experiment for our tech ops team. We wanted to know if Flowable was the right tool for them and that they were comfortable enough to create and modify business policies in it?

Migrating away from Google Forms

It was challenging at first. Coming from a stable solution like Google Forms to an open source tool seemed like a tough decision, but we took a leap of faith. As a POC we migrated one of our google forms to flowable app. We utilised API’s exposed by flowable SDK’s and built a service on top of it. We then migrated 2–3 more flows. Everything worked fine in the beginning. We were doing about 50k+ transactions on Flowable everyday. As data grew in Flowable database the API’s started getting slower. We suffered a few hiccups on the way. In order to truly reap the benefits of an opensource tool you need to dig deeper and understand its internal working as well. It turned out that there were missing indexes on the database side. We raised it with the Flowable team and got them added in their source code.

Currently we have around 20+ resolution flows on Flowable and we are scaling it as I write this. Now our goal is to introduce automated machine tasks in the BPM process to reduce the average handling time of each resolution and remove those trivial questions that I outlined above.

Reflection

As I look back on the past two-year journey, I realise that we have travelled so far. Not only in-terms of making our agents lives easier but also evolving as a true agile team.

When we started this journey, we had a design sprint to decide on what we are going to build and we created wireframes, prototypes and what not about what out final web-app would look like. It looked great as a prototype, but the thought of implementing that prototype burdened us. Being introduced to lean startup methodology made our lives a lot easier. We have started experimenting a lot more. If we are not sure of something, we make that available to our Beta agents. To validate our hypothesis, we measure the impact using analytical tools like Google Analytics and finally, we iterate over it.

Learnings

• A feature can never be fully complete. There is always room for improvement. Its in the hands of the user to decide which improvement is more important than the other. So reduce the scope, launch soon, gather feedback and then iterate over it.
• To truly understand what your user wants, sit with them while they are working on the problem you want to solve. If possible, shadow and role play as a user as well. This way you understand the problem in users shoes.
• Tech debt keeps growing and growing. New features have some monetary benefits attached to them and so they will always have higher priority. The way we deal with tech debt is by incrementally reducing it with new features. When we have to implement a feature on a piece of code with huge tech debt, we first refactor the code and then implement the feature on it. Incremental refactoring is always helpful and keeps the code clean over the long run.
• Open source tools give you a huge boost and enable you to launch quickly and experiment if the intended functionality is beneficial to the users or not.
• Writing tests increased our confidence on the code we wrote. If we didn’t have enough time, we reduced the scope of the feature but ensured that we wrote quality code.

Transforming the way customer disputes are resolved at Careem was originally published in Tech @ Careem on Medium, where people are continuing the conversation by highlighting and responding to this story.

The ‘Reproducibility Crisis’ is common knowledge in the scientific community. Most businesses, however, haven’t even begun to recognise the problem.

Background

“Science is in a methodological crisis” — this seems to be a common sentiment among academics.

In a survey of 1,500 researchers, 70% of them said they had failed to replicate another scientist’s experiment [1]. In a 2009 study, 14% of scientists admitted to personally knowing someone who had falsified results [2]. A 2019 meta-study of deep learning in recommendation systems suggested that less than 40% of papers were replicable [3]. Countless other studies have gone on to confirm similar findings.

These findings have been extremely demotivating for researchers, especially anyone working in Social Sciences and Psychology at the time. It has weakened the position of Science in the public sphere.

What's not talked about is how incredible this realisation is, and how far ahead the Scientific Community is in identifying these problems. Most of the private sector hasn’t even begun to recognise the issue or understands its relevance.

A lot of analytical work in businesses is bizarre and senseless. Too often, employees deploy subtle scientific tools with a superficial understanding. The result is a vast waste of time, energy, and money.

Introduction

In this article, I'm going to talk about some of the lessons we are learning from academic researchers, and highlight some of the things we can do to improve analysis, experimentation, and A/B testing in our companies.

1. Stop Incentivising Bad Practises

In many companies, your promotion/remuneration/commission is directly or indirectly linked to the results of an experiment you are running. I understand the motivation for this, however, it incentivises false reporting and lazy analysis, and doesn’t reward exploration and risk-takers.

2. Education

Most business leaders don’t know what a significance test is or understand why statistical analysis is important.

Statistical Analysis can drastically change the results of your findings. For example, let's look at this revenue data for an e-commerce site.

> SELECT cohort, AVG(revenue) FROM revenue_table GROUP BY 1 ;
cohort, AVG(revenue) , COUN(custoomer_id)
treatment, 14.01 , 5723
control, 13.75, 5721

On face value, it would appear the treatment group had an increase in revenue.

However, after filtering outliers and plotting confidence intervals you can see, if anything, there is a negative impact for the majority of users.

The variance of your metrics can we very high, especially if you are involved in both B2B and B2C sales.

3. Peer Reviews

Pair programming, replicability studies, and cross-department validation can help you understand if your methods and findings are valid.

4. Understand P-hacking

P-hacking is a collection of ways you can distort your analysis, by collecting or selecting data or statistical analyses until nonsignificant results become significant. Common examples are the early stopping of A/B tests and selective reporting,

For an entertaining discussion on the P-hacking check out Last Week Tonight with John Oliver.

5. Don’t Confuse your Experiment Unit and Observation unit

A common mistake people make when running an A/B test is creating a test split on users but then treating more granular data as your independent observations.

Take a look at this conversion data. Users have been placed into random control and treatment groups.

Often, the analyst’s first instinct is to model this as independent Bernoulli trials.

However, because the split is at a user level, the data is more like a power distribution.

By treating sessions as i.i.d we had tricked ourselves into thinking we had more data and less variance than we do. The ‘real’ confidence intervals are much wider than our initial assumption had us believe.

The results are no longer statistically significant (which is good, as this was a random A/A test with no difference between control and treatment).

6. Make Sure Your Test Has Enough Power

Whether you are doing a Hypothesis test or calculating confidence intervals, you need to know what kind of impact you are able to measure and how likely you are to recognise it.

7. Deduction vs Induction

Deductive reasoning moves from idea to observation, while induction moves from observation to the idea. Both are important types of scientific reasoning.

Always think about your hypothesis and theory behind the observations you are seeing. Often having a valid theory is more important than the observations themselves.

8. Careful Calculating Rates

For example, if your treatment increases the number of users you see, then you might see a drop in revenue per user, even though total revenue is increased.

Try to look at sums and totals when possible.

9. Become a Bayesian

It's likely you are using Bayes Theorem and prior distributions already. But to really make use of the Bayesian paradigm look into Thompson Sampling.

PyData has a helpful introduction to Thompson Sampling https://www.youtube.com/watch?v=wcCSAbcj5Q0.

10. Random is Rarely Enough

A common issue in testing is "Pre-Experiment Bias". This is when, by chance, your control and treatment have a pre-existing difference. This dramatically reduces the statistical power of your tests.

Solutions are strategised sampling and propensity matching. There are more details on the subject in these posts:

• https://engineering.thumbtack.com/repeated-rerandomisation/
• https://blog.twitter.com/engineering/en_us/a/2015/detecting-and-avoiding-bucket-imbalance-in-ab-tests.html
• https://dataorigami.net/blogs/napkin-folding/17987495-the-class-imbalance-problem-in-a-b-testing
• https://eng.uber.com/xp/

However, the conversation around this topic is often ill-defined. Sometimes its talked about as a solution to measuring significance, however, it shouldn’t affect the validity of your significance test, only its power.

11. Don’t Trust Blog Posts

There is a large industry around testing, particularly in A/B tests. It's estimated that the global A/B Testing Software Market to Reach US$1,081M by 2025.

Claims-makers love to point out problems they can provide solutions to. Try to remember that what you are reading is a piece of self-promotion, not academic material.

Much of the language in online significance calculators is vague and doesn’t clarify what your observations are or how the test needs to be constructed, e.g:

• https://neilpatel.com/ab-testing-calculator/
• https://abtestguide.com/calc/
• https://www.surveymonkey.com/mp/ab-testing-significance-calculator/

12. Don’t Filter

As soon as you put a filter on your data (e.g filtering users who didn’t make it past a certain point in the conversion funnel), the test is no longer a randomised experiment.

13. Filter Outliers

Filtering outliers can help improve the power of your significance test. Keep in mind these outliers might need to be analyzed separately.

14. Multi Comparison

If you are doing multi-variant testing you might need to correct your P-values, (e.g .Bonferroni Correction). The more variants you have, the higher the chance of a false positive.

15. Be Aware of Network Effects

If you’re testing in a marketplace or social network, it is likely that the effect on your treatment will have consequences on the control. For this, you will need a network segmentation, a time-based test, or geo-spatial segmentation.

Lyft has some posts on the problem, https://eng.lyft.com/experimentation-in-a-ridesharing-marketplace-b39db027a66e

16. Bootstrapping

Finding the right closed-form solutions for confidence-intervals/hypothesis tests is nuanced and difficult. It can require years of experience and a deep understanding of statistics.

Bootstrapping is an easy and intuitive alternative for calculating a probability distribution and confidence intervals.

17. Increase Transparency

Always share your code, data, and methodology.

Adopt a coding convention when sharing code. ‘Papers With Code’ provide a useful checklist for this, https://medium.com/paperswithcode/ml-code-completeness-checklist-e9127b168501.

18. Be Wary of Time Based Tests

Time-based switching is when your alternate versions of your app/website/etc based on hourly windows.

The number of observations you have is the number of time windows — this is usually very small. You will need very large shifts in metrics in order to see any statistical significance.

19. Check Tracking

Often with new app releases come improvements in tracking. This will affect what kind of metrics you can compare.

Conclusion

This article is far from extensive. Each experiment is its own piece of analysis with 100 different pitfalls each time.

Do not underestimate how difficult experimentation is.

Crisis, Replication and A/B Testing was originally published in Tech @ Careem on Medium, where people are continuing the conversation by highlighting and responding to this story.

And now, building a stronger tech team is more important than ever before. Careem’s 2022 vision is to become the regions everyday super app. To make this vision a reality, Careem has been searching for the best leaders with the right expertise to support the scaling and reliability of ride-hailing.

Careem has appointed three senior tech leaders in its ride-hailing domain over the past few weeks: Moin Moinuddin, Reno Marioni and Mohamad Charafeddine.

Moin Moinuddin will lead engineering for Careem’s ride-hailing domain. Prior to Careem, Moin was the product and engineering leader at Tesla, based in the United States. At Tesla, he was responsible for leading global ecommerce, POS and its payments platform to enable a worldwide frictionless commerce experience for Tesla customers.

Before that, Moin spent close to 20 years in senior tech-leadership positions across Microsoft, Dropbox and Visa. Moin started his professional career as a developer where his first big achievement after graduation was developing a POS system that is still used by Starbucks around the world.

Reno joined Careem as Product Director for Marketplace in Berlin. Before joining Careem, Reno was the Chief Product Officer for Moovel, now known as ReachNow, the Mobility-as-a-Service arm for the Daimler/BMW Mobility joint venture, where he led product and design teams.

Prior to this, he served as the Senior Director of Digital Products & Services at BMW Group, where he led digital service innovation for the BMW Connected Car ecosystem in Munich, Germany. Marioni was influential in spearheading the transformation of BMW into the connected car digital services across all consumer touch points. He also worked at HERE Technologies, Nokia, and founded his own startup.

Mohamad will be heading Product for Trust, Safety and Integrity and Care. He recently relocated from Silicon Valley, where he worked on Core ML for business integrity at Facebook to classify and enforce on all ads content in order to reduce personal, financial, sentiment, and societal harms.

Prior to that, he headed product strategy and execution for the applied AI R&D lab at Samsung SDS Research America, where he led applications of Reinforcement Learning and Deep Learning in multiple domains such as digital marketing, healthcare, and manufacturing. He was part of the founding team for a Stanford spin-off startup, ASSIA, where he built big data mining software to diagnose and optimize entire access networks for telecommunication service providers with 100M households under management.

Careem appoints three tech leaders to help achieve Vision 2022 was originally published in Tech @ Careem on Medium, where people are continuing the conversation by highlighting and responding to this story.

Every day, engineering teams are working tirelessly to solve problems and more often than not, this means choosing between the best of many possible solutions. When we share our approaches to technical problems, we hope to fast track this process for our teams.

Recently, Careem’s DataOPS team was tasked with the migration of a database from one AWS account to another with minimum downtime. Here you will find the specific steps we took to set up a live RDS MySQL replication across AWS accounts.

The same approach can also be applied to set up production account data for testing in another AWS account like QA. When you are testing on live data, sensitive information must be scrubbed by adding database triggers on the QA account.

Note: We have tested this functionality on AWS RDS MySQL 5.7 and it works.

Let’s suppose you have a database named DB1. For the sake of clarity, the accounts will be referred to as “Account A” and “Account B”.

Do the following task on On Account A.

• Create one new exclusive VPC Security group to be used in new read replica. Let's name it sg_share_RDS.
• On “DB1”, run following to make sure binlog files there for at least 24 hours.

call mysql.rds_set_configuration(‘binlog retention hours’,24);

• Create new RDS Subnet Group, with at least two public subnets attached to this subnet group, let’s name it “share_rds_subnet”.
• Create new Read Replica from database “DB1” with port 3316(or any nondefault port i.e. 3306).
• Set Public access to ‘Yes’ and assign Subnet Group “share_rds_subnet” to Read Replica. Also, enable auto backup on this new read replica.
• Assign security group sg_share_RDS to newly created read replica & Allow port 3316 to your VPN Security group to make a connection to this read replica.

Lock every database user in the new read replica, after switch off (read_only=0) in read replica parameter group, we prefer you also have exclusive DB parameter group for newly created read replica. After locking down all users, create one database user that is exclusively used for replication with a tough password. For example

GRANT REPLICATION SLAVE ON *.* TO <your_user>@'%' IDENTIFIED BY <your_password>;

Run below command

SHOW SLAVE STATUS; 

And check seconds_behind_master must be zero. on a newly created read replica. Make sure the newly create read replica is ready and fully synced with the master. Now do the following steps.

Stop replication on read replica by calling

CALL mysql.rds_stop_replication().

• Run following command on stop read replica and save values of (Master_Log_File , Read_Master_Log_Pos) to use it later.

SHOW MASTER STATUS;

• If you are on Mac run dig <your stop read replica endpoint> note down the public IP address.
• Take a snapshot of a read replica in account “A” and share snapshot to other AWS account “B” from AWS console.
• After share snapshot restores RDS snapshot in AWS account (“B”) make sure you also have designated security group and subnet group (with at least two public subnets) on the target AWS account.
• Once a snapshot is restored and available, connect to it and run the following commands. Now run dig <newly created RDS endpoint in account “B”>

Get the public IP address of newly created RDS in account “B” and allow this IP in account “A” ( source DB ) security group to connect on the port which you defined for your read replica. This means only this IP (from the account “B”) can connect to source DB in account “A”.

Now run following in your target DB in account “B”.

CALL mysql.rds_set_external_master(<Public IP Address of stop Read Replica>, <port of read replica>, <username> , <password> , Master_Log_File,Read_Master_Log_Pos, 1);

CALL mysql.rds_start_replication();

Now run following command

SHOW SLAVE STATUS; 

on newly created DB in the account “B” and check if its successfully connect to its master database (The read replica on AWS account “A”).

How to setup RDS Replication across AWS accounts was originally published in Tech @ Careem on Medium, where people are continuing the conversation by highlighting and responding to this story.

Careem is increasingly playing a big part in the region’s digital future, through our technology, infrastructure, and our diverse and talented people. We thought you might like to hear from one of the tech team who is helping make this happen and learn more about what goes on at Careem. Ivan Belfinger is one of our backend software engineers…

What exactly do you do at Careem?
I’m a backend software engineer, which means I work with things that are under the hood. You know when you use the Careem app and you see all those nice things? Well, I work on all of those other things that are going on behind them. I’m part of “New Categories” team, and currently working mostly with a fun problem called pooling; how to efficiently match multiple customers in a single car.

What technologies are you working with specifically?
Java, Python, AWS, Dynamo DB, Redis and some of the other industry standard tools. We’ve just been playing around with OR Tools, this is really fun. We’re using this to solve real-life problems such as vehicle routing issues — how to route the vehicle for as little cost as possible whilst maintaining high customer service. This is why I love working for Careem, we get to work and experiment with the newest and latest technologies and we are not limited to legacy software.

What does your typical day involve?
My day usually involves a lot of coding, reviewing other people’s code and talking to other teams. Our current tasks demand a lot of inter-team collaboration, and that has been super fun. My day is interrupted here and there by a few meetings and interviews. Apply for a job in Careem, and I might be one of the first people from the team that you meet!

What were doing before you joined Careem?
I’m slightly ashamed to say I was working for a very boring company — a bank. The environment there was painfully slow. Something I really love about being part of a tech start-up now, is how different it is.

So what inspired you to join Careem?
Dubai was part of the attraction, for sure, but Careem is the biggest tech company in the region so that was my key motivation — I knew that I would get to work on the latest and best tech innovations in the region.

What motivated to choose this career in the first place?
As a child my dad worked with AutoCAD, one day my dad brought home an old machine his company was throwing away. I was only six years old and my brother was 10 years old, and he gave us a screwdriver and said start playing. That is how both my brother and I are now Software Engineers. It runs in the family.

What drives you at Careem?
What I like about Careem is that we enable low-income individuals to create their own futures and opportunities for themselves and their families. By becoming a Captain with Careem they are changing their lives and we are helping to do this. We are having a huge impact in the region. The culture here is also really good — it’s very multicultural with over 70 nationalities at Careem overall and around 50 in the tech team. Everyone is super open to talking and helping each other. In Careem you are never the smartest person! Sometimes this makes me feel bad, but it’s great that I’m constantly learning every day.

Careem Tech profile: Engineering under the hood was originally published in Tech @ Careem on Medium, where people are continuing the conversation by highlighting and responding to this story.

As Careem enters Chapter 2 and expands into platformisation, we are increasingly playing a big part in the region’s digital future, through our technology, infrastructure, and our diverse and talented people. We thought you might like to hear from some of our tech people who are making this happen to learn what they do and how they got into this line of work.

Asma Alyamani — Lead Product Manager

What do you do at Careem?
I lead product for the customer ride-hailing experience at Careem. We are the voice of the customer. We understand their wants and needs, and build products to simplify their lives. We are obsessed with creating the best experiences for our customers across all touch points in their journey, primarily on our iOS and Android apps.

How big is your team?
We are a team of seven product managers, 40 engineers and three product designers.

And you’re responsible for creating a roadmap for the entire customer-facing product?
Yes, and each product manager on the team owns a different part of the customer journey. For a particular problem space or product, there is a dedicated team of engineers, designer and a product manager who are jointly responsible for defining their priorities and goals, and delivering value for our customers. For example, we have teams dedicated to payments, locations, and acquisition to name a few.

How did you start at Careem?
I joined almost four years ago. Careem was much smaller back then, and the product team was only three or four people. I started as a country product manager for Saudi Arabia, focused on getting strong product-market-fit for Saudi by building a deep understanding of both our customers and captains, figuring out what their biggest needs are and finding ways to solve them.

I did that for about a year, then shifted into the customer product management role where I worked on a few different products before more recently leading it overall.

How is the customer app different from market to market?
Some things are standardised across markets and some are localised. One of the primary features of our product which differs in each market is the set of services available. If you open the app anywhere you still maintain that familiarity with the Careem experience, but the services available in that market are completely localised. In Dubai, for example, we have Dubai Taxi, Economy, Business, First Class and so on. In Karachi, we have Bikes and Rickshaws which aren’t available in Dubai. It’s important to have this flexibility to offer the most suitable products for customers’ needs in each city.

Another feature that’s localised is Careem Rewards, our loyalty program. You earn points for each Careem ride you take, and can then use your points to give back to those in need or take free rides. Depending on where you open the app, you’ll see a different set of rewards that you can redeem. In Dubai, for example, we’ve partnered with Dubai Cares so you can use your points to provide education for underprivileged children, whereas in Riyadh we’ve partnered with the local charity Disabled Children’s Association which you’ll see as a reward if you use the app there. The vision is to keep growing this library of rewards and be highly connected to the local ecosystem in each market.

How did you get into product management?
Through Careem, actually. My Bachelor’s degree is in computer science so I was always close to tech. I studied at the University of Pennsylvania in the United States, then did an MBA at Stanford. While I was there, I got exposed a lot more to entrepreneurship and the tech ecosystem, and I did my summer internship at Zendesk. My role was product analytics working directly for the product manager leading their data products — that was how I first got exposed to product management.

When I graduated I decided to come back to the region because I felt like I could have more impact here. I was introduced to the Careem co-founders and immediately decided to join because Careem’s mission really resonated with what I wanted to do.

What does your typical day involve?
A core part of my role involves dealing with people, whether it’s relationship building, stakeholder management, collaborating with engineers or having one-to-ones with my team members.

To build the best experiences for our customers, we need to understand what are their pain points and needs. And given the diversity of our customer segments in our different markets, which vary substantially, it’s important to spend talking to colleagues in the city teams who are on the ground and closest to our customers to get an understanding of what’s happening as well as talking to customers.

Similarly we need to align with stakeholders in different functions to make sure we are working together on the same strategy and mitigate any risks and dependencies on each other. We need to be aligned on the vision of where we’re going, what needs to be done and then building the right products with the optimal priorities.

I would say 50–60 percent of my time goes there. Another 10–20 percent translating this into our product strategy and vision. The remaining time is spent looking at our organizational structure and how to enable the right environment to provide teams with enough autonomy to deliver. This also means constantly iterating on our product processes, what we’re building, and working with the team to conceptualise new ideas and share feedback on some of the experiences that we’re working towards.

We do design critiques on a weekly basis as well as product feedback sessions where a product manager would come and present something they’re working on to the rest of the product team to get insights, which helps them improve it. Generally, we have a very collaborative way of working and a lot of my time is spent working with people.

What’s been the most successful feature you’ve introduced?
Packages has been an incredibly successful product for us, and this is a product which Careem innovated in so it’s not something that’s been done before. We started by targeting the commute segment in Saudi, and have since evolved into various use cases. We offer two types of packages that customers can buy: ride packages and KM packages. For ride packages, the value proposition is that a customer pays upfront to save on their rides — by getting a discount on each ride — with access to priority customer support. With KM packages, customers buy a bundle of kilometres which enables them to pay one upfront payment for their upcoming rides at a discounted rate. They also benefit from no peak and other surcharges.

Our customers love using packages and it’s helped tremendously with the stickiness of our platform. We are focusing on further improving the package experience and addressing some of the current frictions, like being able to purchase a package with cash.

We also launched Careem Rewards which has been a great success for us, and our customers love it! Similar to packages, we are determined to make this even better for customers.

We just keep iterating and evolving.

Careem Tech profile: Creating the customer experience was originally published in Tech @ Careem on Medium, where people are continuing the conversation by highlighting and responding to this story.

Monolithic applications are increasingly being replaced with smaller, more easily maintainable, components or services that communicate with each other to accomplish the same task in a more efficient manner.

As services are separated from each other, it becomes easier to identify distinct services that are highly utilized and scale those services independently of the whole application.

Storing data from different domains in a single database has an additional side effect that should not be underestimated. It creates dependencies between different services/domains on a DB schema level. A breaking change on the schema level - e.g. changing the data format of a column — will have a ripple effect through the whole system, ultimately requiring a synchronous change and production deployment of all dependent subsystems. Finding these subsystems, however, is almost impossible as the database often doesn’t provide a reliable way to track those dependencies. Again, this significantly impacts the new feature development as it makes broader and high quality refactoring impossible.

Data isolation within the existing database

There is actually a way to approach the split safely using only master-slave replication. The idea is to split/isolate the data within the originating database itself (Main DB). This can be achieved by restricting the MySQL database users.

For example, the Services “A” data had to be split out of Main DB. So a Services “A” DB user was created and this DB user was restricted to read/write only to the Services “A” data tables. Similarly, all other DB users (one by one) had their access to the Services “A” data tables denied.

This is a long process that includes extensive monitoring for system abnormalities, because when a DB user (e.g. Services “B” user or Services “C” user) has the access cut, then the service that is using this user might suffer from lack of privileges in case it directly or indirectly (though joins) uses Services “A” tables.

If abnormalities are detected the user should have their access immediately restored. The process, however, helps with identifying and fixing of rogue services which directly use the Services “A” data tables. Needless to say, the service that is going to have its user access cut from Services “A” tables should have it’s code base thoroughly examined prior to the exercise. In summary, the process will be:

1. List all database users.
2. Identify the user that with high confidence doesn’t use directly the Services “A” tables
3. Cut the access to Services “A” tables to that user.
4. Carefully monitor the system for abnormalities.
5. If an abnormality is found immediately restore the DB user access, investigate and fix the reason and go to step 3.
6. Successfully eliminated the access of the selected user to the Services “A” tables. Go to step 2. for the next user until all users except the
7. Services “A” user has their access denied.

In the end, other users have their access to Services “A” tables cut, we can safely say that the Services “A” data is now private to the Services “A”. In other words, even though the data is still residing in the Main DB we have the process of extracting a microservice completed. The next logical step, of course, is to migrate this data to a new database i.e. reap the benefits of having a microservice by changing its data storage. This is now easy, the process is the following:

1, Create a Main DB read replica (this will be the new master database for microservices).
2. Wait for zero replica lag i.e. all data has been replicated.
3. Create read/write user in the replica exclusive for the new service
4. Reconfigure all DB reads to go to the new read replica.
5. Stop the new Services “A” i.e. all writes to the old DB.
6. Reconfigure all writes to go to the new read replica.
7. Wait for zero DB lag. Promote the read replica as a new master node.
8. Start the new service.

At a later stage, we can remove the irrelevant tables from the newly promoted database and same we can do to the old master database remove tables which we have migrated to the newly promoted database.

This is a simple two-step process, where first we restrict services on a set of tables after defining table ownership by using database privileges model and at a later stage, we just create a slave database for data migration.

How to Break the Monolithic Database was originally published in Tech @ Careem on Medium, where people are continuing the conversation by highlighting and responding to this story.