Helly Hansen Out of the Adobe Commerce Cloud Migration Performance Use Case.
Before I wrote about bad Magento Cloud performance, basically, Magento Commerce Cloud is broken and can’t be fixed. Merchants are really frustrated but still using it. Considering how much money they wasted on the wrong Magento 2 implementation, it is a difficult decision to trash it out and build modern eCommerce. However, Helly Hansen (HH) decided to move the website out of the Adobe Cloudy Magento solution and win. We have previous performance test results, and today we will compare metrics of Adobe Commerce Cloud with the new HH self-hosted environment after the exodus.
Previous Pivot Table Helly Hansen Magento 2 Performance on Magento Commerce Hosting
+-----------------------------+--------------+
| | Magento 2 HH |
+-----------------------------+--------------+
| 1.Home Page | 2367 |
+-----------------------------+--------------+
| 2.Product Page | 3580 |
+-----------------------------+--------------+
| 3.Category page | 6046 |
+-----------------------------+--------------+
| 4.Filter/Layered Navigation | 6009 |
+-----------------------------+--------------+
| 5.Search Result | 7320 |
+-----------------------------+--------------+
| 6.Search Result 50 users | 20201 |
+-----------------------------+--------------+Magento 2 Cloud HH Performance Chart:
Source:
We can see the result was just awful. I really never saw a good Magento 2 performance on the Magento(Adobe) Commerce Cloud. Basically, it is not a cloud. It is a marketing double, not a hosting solution. Instead of using the best cloud practices, Magento has usually reinvented the wheel. But the wheel has a square shape instead of a classical round one. Cloud was added to the name to mislead about the platform.
New Helly Hansen Platform performance:
Home Page Performance:
Tested URL: https://www.hellyhansen.com/ source=consolerunning (00m32.7s), 0/1 VUs, 20 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs 00m32.7s/10m0s 20/20 shared iters█ setup█ teardowndata_received..............: 4.7 MB 142 kB/s
data_sent..................: 24 kB 739 B/s
http_req_blocked...........: min=282ns med=302ns avg=2.16ms max=43.21ms p(95)=2.16ms
http_req_connecting........: min=0s med=0s avg=317.59µs max=6.35ms p(95)=317.59µs
http_req_duration..........: min=954.61ms med=1.46s avg=1.43s max=1.94s p(95)=1.89s
http_req_receiving.........: min=1.34ms med=1.62ms avg=2.72ms max=18.41ms p(95)=6.79ms
http_req_sending...........: min=29.69µs med=36.3µs avg=40.47µs max=105.69µs p(95)=52.6µs
http_req_tls_handshaking...: min=0s med=0s avg=741.04µs max=14.82ms p(95)=741.04µs
http_req_waiting...........: min=953.23ms med=1.46s avg=1.43s max=1.93s p(95)=1.87s
http_reqs..................: 20 0.610854/s
iteration_duration.........: min=97.04µs med=1.65s avg=1.48s max=2.14s p(95)=2.12s
iterations.................: 20 0.610854/s
Result: 1.46s vs 2.36s on Magento Cloud
We used the Median as the main metric however the best case is Percentile 95. It shows a real user experience without approximation. Percentile 95 for this page is 1.87s.
Helly Hansen Product Page Performance
Tested URL: https://www.hellyhansen.com/en_us/crew-midlayer-jacket-30253 source=consolerunning (00m43.2s), 0/1 VUs, 20 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs 00m43.1s/10m0s 20/20 shared iters█ setup█ teardowndata_received..............: 5.5 MB 128 kB/s
data_sent..................: 30 kB 686 B/s
http_req_blocked...........: min=269ns med=310ns avg=3.04ms max=60.91ms p(95)=3.04ms
http_req_connecting........: min=0s med=0s avg=385.12µs max=7.7ms p(95)=385.12µs
http_req_duration..........: min=1.27s med=1.89s avg=1.95s max=3.2s p(95)=2.52s
http_req_receiving.........: min=1.51ms med=1.81ms avg=2.88ms max=22.74ms p(95)=3.74ms
http_req_sending...........: min=39.9µs med=54.15µs avg=57.04µs max=129.48µs p(95)=67.77µs
http_req_tls_handshaking...: min=0s med=0s avg=880.96µs max=17.61ms p(95)=880.96µs
http_req_waiting...........: min=1.27s med=1.89s avg=1.94s max=3.19s p(95)=2.52s
Result: 1.89s vs 3.5s, almost 2 times improvement!
Percentile 95: for the Product page is 2.52 s
Helly Hanse Category Page performance
Tested URL: https://www.hellyhansen.com/en_us/mens/jackets/sailing-jackets source=consolerunning (00m42.1s), 0/1 VUs, 20 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs 00m42.1s/10m0s 20/20 shared iters█ setup█ teardowndata_received..............: 6.1 MB 146 kB/s
data_sent..................: 32 kB 769 B/s
http_req_blocked...........: min=271ns med=292ns avg=1.72ms max=34.56ms p(95)=1.72ms
http_req_connecting........: min=0s med=0s avg=310.99µs max=6.21ms p(95)=310.99µs
http_req_duration..........: min=1.27s med=1.98s avg=1.9s max=2.42s p(95)=2.23s
http_req_receiving.........: min=1.14ms med=1.63ms avg=6.11ms max=87.37ms p(95)=10.21ms
http_req_sending...........: min=41.81µs med=49.91µs avg=53.97µs max=136.4µs p(95)=72.78µs
http_req_tls_handshaking...: min=0s med=0s avg=722.55µs max=14.45ms p(95)=722.55µs
http_req_waiting...........: min=1.27s med=1.98s avg=1.89s max=2.42s p(95)=2.23s
http_reqs..................: 20 0.475089/s
Result: 1.98s vs 6s on Magento Cloud!
Percentile 95: 2.23s
Magento Cloud has 3 times worse performance than AWS Cloud's self-hosted and designed solution.
Helly Hansen Search Performance
Tested URL: https://www.hellyhansen.com/en_us/catalogsearch/result/?q=sailing&ct=regular_search source=consolerunning (00m45.7s), 0/1 VUs, 20 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs 00m45.7s/10m0s 20/20 shared iters█ setup█ teardowndata_received..............: 6.6 MB 144 kB/s
data_sent..................: 34 kB 748 B/s
http_req_blocked...........: min=262ns med=307ns avg=1.41ms max=28.26ms p(95)=1.41ms
http_req_connecting........: min=0s med=0s avg=309.22µs max=6.18ms p(95)=309.22µs
http_req_duration..........: min=1.72s med=2.08s avg=2.08s max=2.7s p(95)=2.65s
http_req_receiving.........: min=5.96ms med=6.02ms avg=7.25ms max=24.58ms p(95)=12.87ms
http_req_sending...........: min=44.25µs med=56.4µs avg=59.14µs max=131.58µs p(95)=71.24µs
http_req_tls_handshaking...: min=0s med=0s avg=717.2µs max=14.34ms p(95)=717.2µs
http_req_waiting...........: min=1.71s med=2.08s avg=2.07s max=2.69s p(95)=2.63s
http_reqs..................: 20 0.437227/s
Result: 2.08s vs 7.32s. Almost 4 times better performance!
The awesome result of bad Magento Commerce Cloud performance!!!
Helly Hansen 50 simultaneous users performance:
Tested URL: https://www.hellyhansen.com/en_us/catalogsearch/result/?q=sailing&ct=regular_search source=consolerunning (00m19.0s), 00/50 VUs, 250 complete and 0 interrupted iterations
default ✓ [======================================] 50 VUs 00m18.9s/10m0s 250/250 shared iters█ setup█ teardowndata_received..............: 82 MB 4.3 MB/s
data_sent..................: 454 kB 24 kB/s
http_req_blocked...........: min=260ns med=321ns avg=15.06ms max=85.05ms p(95)=78.21ms
http_req_connecting........: min=0s med=0s avg=1.59ms max=10.31ms p(95)=8.93ms
http_req_duration..........: min=1.73s med=3.35s avg=3.48s max=5.96s p(95)=5.24s
http_req_receiving.........: min=1.3ms med=7.17ms avg=9.76ms max=30.82ms p(95)=25.09ms
http_req_sending...........: min=17.08µs med=31.86µs avg=40.41µs max=175.25µs p(95)=79.03µs
http_req_tls_handshaking...: min=0s med=0s avg=5.46ms max=36.34ms p(95)=31.18ms
http_req_waiting...........: min=1.71s med=3.34s avg=3.47s max=5.94s p(95)=5.23s
Result: 3.34s vs 20.2s. Awesome, 6 times better performance, at least!
As we can see, Magento is one of the American companies about Elon Mask said:
Are top American executives focusing enough on product improvement? I think the answer is no. Spend less time on finances, less time in meeting rooms, spend less time on PowerPoint, and more on trying to make your product the best it can be.”
We can see that moving from Magento Cloud has great improvement hoverer Site performance is still awful. 2–5s TTFB Server response time and Google Page Speed insight:
Server Response Time report of the Helly Hansen
We can see that more than 60% of HH website visitors have a bad server response time. And after the migration out of the Magento slow cloud, the average performance is slightly better.
Thanks to the eCommerce performance guru Erwin Hofman (https://www.linkedin.com/in/erwinhofman/) for sharing how to use this report.
One of the many Adobe Cloud Issues that guys use X1 instance [2.3 GHz Intel Xeon E7–8880 v3 (Haswell)128 vCPU 1952 GiB memory] to scale cloud for the most expansile enterprise’s customer plan (120 CPUs). However, this instance type has a much worse performance (60% slower) than R5 [3.1 GHz Intel Xeon Platinum 8175 (Skylake)] and Graviton 2 (10% faster).
You don’t even need to run sophisticated performance tests to understand 2.3 GHz CPU clock speed is lower than 3.1 GHz.
X1 performance is:
The code took 0.16240096092224 seconds to complete.
Redis performance :
====== GET ======
100000 requests completed in 1.80 seconds
100 parallel clients0.01% <= 1 milliseconds
99.12% <= 2 milliseconds
99.79% <= 3 milliseconds
99.92% <= 4 milliseconds
100.00% <= 4 milliseconds
55 555.56 requests per second
R5 performance:
The code took 0.10573697090149 seconds to complete.
Redis performance 143 061.52 requests per second
C6g Graviton 2 CPU performance:
The code took 0.15448880195618 seconds to complete
Redis Performance 168 067.22 requests per second
Magento uses just the instance with the biggest number of vCPU (128) available. But, more cores(virtual) are not necessarily faster! Especially if it is a virtual CPU, not physical cores.
Basically, by moving out from Magento Cloud and fully controlling and fine-tuning your cloud environment, you can achieve 2–20 times better performance. If you need support, you can hire several 1–3 Cloud Architects full-time for a fraction of the Magento Cloud fee price.
Helly Hansen, guys, I’m, as a sailing racer, really frustrated about such a bad customers experience.
Magento Agencies which selling shitty Magento solution like in case of Helly Hanse Amazon wrote:
In the crowded world of eCommerce, it often feels like there are no new tricks to try out. Lots of agencies claim to be committed to shaking up stale industry protocol, but “talking big” only goes so far without authentically innovative approaches to back it up and just sales broken Magento 2 solutions hosted on slow Magento’s Cloud.
