How did Helpshift solve slow deployment issues in AWS Beijing(China) region
Let me start with why is it challenging to have a setup in AWS China regions, it involves consideration of a lot of factors, why?
AWS China regions (Beijing and Ningxia) are not like other AWS regions because of the followings factors,
- AWS Beijing and Ningxia regions are completely separate from other AWS regions.
- Beijing Region is operated by Sinnet and Ningxia Region is operated by NWCD.
- Anyone who wishes to use AWS resources in either Beijing Region or Ningxia Region is required to create an AWS China Account which is separate from other AWS global Accounts.
- In order to open an AWS account in China, you need to have customers in China.
- Also, hosting internet content is not a straightforward procedure in China, it requires government approval in accordance with Chinese law and regulations in the form of Internet Content Provider (ICP) license.
- Users will experience performance issues like slowness, timeouts, etc. after accessing internet content from outside of China, the reason is that all traffic is monitored by the government in China.
- These are few examples of challenges we experienced, you will explore them more if you are thinking to open an AWS account in China.
What’s the actual problem?
As I explained above, we also noticed performance issues in our software deployment workflow in the Beijing region because the deployment content is hosted in the S3 bucket which is located in the non-China AWS region.
The deployment speed was terribly affected due to the slow download speed for the packages hosted outside (another region/location) of China.
How’s the existing Release Workflow?
- Right now, we are building the artifacts in the non-China AWS region and uploading them in the same region s3 bucket.
- During the Beijing region deployment, the release workflow downloads the build artifacts from the non-China region s3 bucket to the instances hosted in the Beijing region, extract the artifacts, and then proceeds with the deployment.
What were the problems faced?
- The download of build artifacts hosted in non-China AWS s3 bucket takes time/sometimes fails after retries (we have 3 retries set) in the Beijing region.
- We had also noticed problems with external repositories from the Beijing region, problems like No package matching ‘xxxx’ is available, this is due to timeout during external repo sync, slow download, etc.
Solutions Tried & Their Results
Solution 1: The first solution that came to our mind was, let’s use the existing process, i.e. build the artifacts in the non-China AWS region, upload the artifacts to the same region s3 bucket and then replicate to Beijing region s3 bucket, so that during deployment, the instances hosted in Beijing region will download the artifacts from there.
Result: CRR(Cross-region replication) is not supported in the Beijing region.
Solution 2: In order to solve the above problem, we thought, let’s manually replicate the packages i.e. build the artifacts in the non-China AWS region and then upload to Beijing region s3 bucket using awscli, from there Beijing region clients can download the artifacts and proceed with the release.
Result: We had gotten very slow upload speed when uploading the artifacts from non-AWS region to Beijing region s3 bucket, we were trying to upload 300MB of artifact file, it took 20mins to just upload 35MB of data ( FYI — Upload was happening in parts of 15MB )
Solution 3: We also thought to completely set up a separate build pipeline only for the Beijing region, i.e. build the artifacts in the Beijing region itself and then upload them in the same region s3 bucket.
Result: We noticed that the build process takes time in the Beijing region & also error-prone because of dependencies on external resources hosted outside of China.
Solution 4: Finally, we decided to give a try to s3 transfer acceleration(S3TA), it basically makes use of Amazon CloudFront’s globally distributed edge locations to logically shortens the distance to s3.
Result: As soon as we enabled S3TA, the results were quite impressive, please find below the actual before & after the time difference.
$ time aws s3 cp s3://test-bucket/build_artifacts/services/service-1.1.0.tar.gz . download: s3://test-bucket/build_artifacts/services/service-1.1.0.tar.gz to ./service-1.1.0.tar.gzreal 3m40.311s
$ time aws s3 cp s3://test-bucket/build_artifacts/services/service-1.1.0.tar.gz . --endpoint-url http://s3-accelerate.amazonaws.com download: s3://test-bucket/build_artifacts/services/service-1.1.0.tar.gz to ./service-1.1.0.tar.gzreal 0m8.035s
Before enabling S3TA, in normal conditions i.e. without any timeout issues, we were getting around 4 mins to download a file of size 230M & after enabling S3TA, the same file got downloaded within 9 secs.
Finally, we decided to stick with the S3TA solution which significantly improved our software deployment speed across all the AWS regions. But as we know everything comes with a price, S3TA added a little bit cost but that is worth investing in.