How Moka Infrastructure Team Handle One Million User Requests
It all started back in 2017 when Moka doesn’t have Infrastructure Team, yet.
There’s only Site Reliability Engineer (SRE) team and well, it’s not really a team, since there’s only one lone ranger inside the team. In August 2017, where Moka starts to grow, the outlets we have been growing and growing.
Until we hit 20,000+ active users per October 2019. Can you imagine how this leads to the incoming requests we got from users? We almost have 1,000,000 requests every 5 minutes!
This means we need to make our servers to work extra!
Other than that, we have another factor like unoptimized code and unoptimized logic from the application that makes a lot of burden to the server. Hence, we think, a good and solid Infrastructure team could help us to handle all those stuff.
From that point, we start to build the team since this team will be the one who will provide the short term and long term solutions to make the infrastructure (servers, networks, services, and systems) to be fault-tolerant and highly available.
As per right now, Moka’s Infrastructure team has 15 people in total. Under it, we have three sub-teams, which are Site Reliability Engineer (SRE) team, Database Administrator (DBA) team, and Performance Engineer (PE) team.
Each team is, of course, have an important role in managing the whole Moka infrastructure. What do they do? What are their focus areas?
As a Site Reliability Engineer at Moka,
Our focus area is infrastructure in general areas such as provisioning, CI/CD, patching, incident, monitoring, and automation. Their job is also include in ensuring Moka’s services, systems, and infrastructures to be fault-tolerant and highly reliable.
As a Database Administrator at Moka,
Our focus area is database structure, backup, archiving, partitioning, monitoring and ensures that the database is up and running all the time.
As a Performance Engineer at Moka,
Our focus area is to create a performance test to make the metrics better and to make sure our infrastructure was high performed and stable.
What is our Activity?
- Serving Pods (Operational work based on a request from Pods — a cross-functional team who can work autonomously with little dependency on other Pods, you can see the detail on this link)
- Legacy On Call (Operational work working being a person in charge to make sure the Legacy Deployment is stable. We have release trend schedule for legacy)
- Research and development (Doing some research and also implementation to make our infrastructure better, efficient, and always following the best practice)
- Infra Talk (A sharing session held every 2 weeks, inclusive for infrastructure team, where one of the team members will present their project documentation details to the rest of the team)
- Team Bonding (Could be lunch or dinner together, maybe sometime we will watch a movie together and do some outdoor activity)
How we monitor it?
- Standup Meeting (some quick meeting for plan, update, and review any activity inside the team to make sure the timeline and communication work better).
- Project Dashboard (we’re using Jira to be our project management software to make sure our activity can be organized well).
- Documentation (we using Jira Confluence to be our centralized documentation to make sure anyone in the team has the same direction about standardization).
We have many stories about how we fix some production issues here at Moka. Just wait for the next article because we will write about the activity details and more!
