PHP 7 deployment at Dailymotion

In march 2015, we started to think that code refactoring and architecture improvements, will not be the only way to optimize the response time on dailymotion.com. This is the core problem of websites with high load : “how to scale without investing too much in people/servers”.

HHVM experimentation

We’ve heard about HHVM, an open source interpreter developed by Facebook that can replace PHP interpreter, which was old and stalling for a long time (PHP6 fail). We started by installing HHVM on a development environment. At the beginning, a simple configuration of HHVM and 50 lines of code commented are enough to make our website work. We made a quick benchmark with ab and results were really surprising. All pages had a response time divided by two and same for memory / cpu.

We had a look on why we have errors on code we had to disable. Two reasons :

1. There is a lot of code incompatibilities between PHP and HHVM ( some functions doesn’t return the same results…, send extra warnings…) (list here)
2. We use custom extensions which are not compatible with HHVM.

So we developed our custom extensions in HHVM style following advices of this guide : http://blog.golemon.com/2015/01/hhvm-extension-writing-part-i.html. And we made some pull requests to correct some HHVM incompatibilities (list here).

We decided that before going further, we had to test HHVM on one production server. After the deployment, we discovered more and more incompatibilities that we fixed with pull requests on HHVM project until everything seemed good. HHVM has a short release cycle, which is great to quickly correct problems. They accept easily pull requests if you correctly explain what you’ve done and write a unit tests for what you changed. During this short test, we had several times HHVM stuck with all cpu core at 100% during this test (link issue). We didn’t used the Repo authoritative mode with better performance because it implied that we had to change our release process to build the bytecode file and we had some dynamic configurations exported by zkfarmer in PHP.

Firsts results in production with HHVM :

We were excited about good results, but some questions were yet unanswered :

• did we have hidden incompatibilities ?
• how to go back to PHP interpreter if we are bored by HHVM
• how to explain to developers incompatibilities with PHP (a lot of time wasted to explain / understand why this is broken)
• how much we will spend time to maintain this solution in production
• how to modify our release process to warm JIT cache.

During few months, this project wasn’t the priority, so we decided to wait the release of PHP 7 to compare performances .

PHP7 : easiest migration, same performance and more stable

In december 2015 PHP7 was released, and some big platforms like Badoo migrated to PHP7 in production with a good results (https://techblog.badoo.com/blog/2016/03/14/how-badoo-saved-one-million-dollars-switching-to-php7/). So we tried to deploy 2 servers, results were almost equal to HHVM performances in term of response time, cpu and memory usage. Moreover it appears to be more stable on servers and extensions are already developed (i.e.: extension like pinba, php-redis, custom Dailymotion extensions).

Graph comparison between PHP 5.5 / PHP 7 / HHVM on two servers to compare performances :

Phan developed by Etsy is a good tool to help you in your first effort, it’s a static analyser an it can find incompatibilities between PHP 5/7 in your codebase and more… It took less than a week to migrate our codebase (old 10 years old PHP monolith) and most of errors were warnings about E_STRICT that haven’t been fixed since a long time (reclassified in notice / warning ). And it took 4 hours to migrate our custom extensions.

So we decided to choose PHP7 because its easier to deploy, there is no regressions except backward incompatibilities listed on php manual, it will have (maybe) a better long time support, and same gain in term of response time / cpu / memory.

tl;dr: we can handle twice more traffic with same infrastructure. Most of big web companies that use PHP have to do this migration to improve their platform costs. And I’m happy to had the time to test different solutions before making the final jump .

Graph of response time / cpu / after PHP7 migration (datadog graph):

Pinba : memory footprint comparison.

PHP 7 :

PHP 5.5 :