e-mas Architecture Evolution
Hi all, this would be my first post for e-mas technology and product, currently i work as IT Head for e-mas apps, we’re an app for saving gold, buy and sell in realtime.
In this post i would like to talk about our architecture evolution. Previously we use straightforward monolith model, because we want to push on as many features as possible. Diagram below is shown for our early system architecture. (We’re sorry if we don’t have a picture of CPU usages, spike, concurrent users, as this happens from 2017 and 2018)
This system on the right has been modified a little bit, by adding Redis as second layer cache. We use AWS as our cloud architecture provider. We choose Singapore as our base because it is closer to Indonesia.
Our early system on 2017 from our API server is hitting directly to MySql DB, no Redis yet. Using Golang for our API Server has paid off for concurrent connection and there aren’t a lot of computation so CPU usage is still relatively low on our API server.
But we saw a bottleneck in our database server, with a lot of concurrent connection to database. When we reach 300 concurrent users, we saw a 100% CPU spike, more than 1300 concurrent connection to database.
For quick solution, we upgrade our DB Server to 16 core and 32GB RAM (i forgot instance type, and i know it’s too much). After monitoring our API for most used endpoint and fine tuning SQL, we decide to add Redis as our second layer cache. And after a week of experimenting and more fine tuning on expire time for Redis cache we got better results and can handle 300 concurrent users without CPU spike on our database server.
Moving on to next phase, our user base grow and our concurrent users grow too, but we didn’t saw a lot of spike on our API server. And we can lower our database server spec because there aren’t many hits to our database because some is handled by Redis. So it is time to move on to put a little microservices to our architecture system.
So based on diagram above we still use Redis as our cache, also we move all of our servers to Alibaba cloud. Main reason why we move it because we want to comply to Indonesian regulatory that state our physical server must be located in Indonesia (some says it’s not a must, but we don’t want to take a risk).
When we move our server from AWS to Alibaba cloud we find that our monthly cost is lower at Alibaba cloud than before at AWS. With same budget for MySql RDS at AWS we only get single AZ, but on Alibaba RDS we can get multi AZ. Overall for RDs, redis, ec2 / ecs instance also cheaper at Alibaba cloud than AWS (not promoting Alibaba cloud here, just based on our experience).
Based on diagram below we use each server for:
- API Server — this is our main server, we handle all hits from apps on this server, all registration, payment, product, transaction, shipping is handled on this server
- RabbitMQ Server — this is our queue management for logging to Elastic Search and sending email
- Callback and Scheduler Server — this is our server for handling callback from payment gateway and partners, we also put our scheduler here because this server doesn’t need to be “fast”, as long as it gets the jobs done
- ElasticSearch — we use for logging requests, responses to partner and payment gateways
- Redis — for second layer cache, has helped us a lot to lower database lod
- MySql RDS — our main database
- Cloudflare — we’ve a happy experience using cloudflare, easy to use, and automatic handling of DDOS attack
- Cloudinary — for image upload, automatic CDN and resize on the fly
- Sendgrid — for sending email, we saw higher click than using SES before
Using above architecture we saw 1000 concurrent users can be handled. We’ve stress tested it to handle 2000 concurrent users and still tweaking and improving to handle more users.
Last but not least, we’re looking for backend programmer, if you’re interested please send email to hr@orori.com with subject “Backend Programmer — Medium”.
