Input Lag PS4 Results
If you have not seen my previous study on input lag, checkout the mega thread I did a few weeks ago as well as all the previous PC results assembled in this twitter moment. This post is about the study I did on PS4 games.
If you just want the results: go to the end!
[update 2018–07–15 16:00] Added the details on SF30AC stability and the corrected measures without frameskips.
[update 2018–07–15 20:40] Removed assumption that reversal cant be performed due to frameskip. It’s not the case on real hardware.
[update 2018–07–24 01:45] Added Koihime and KOFXIV in the final result graph
Foreword: PC issues and statistical stability
As you may know, I had trouble stabilizing the setup in order to get consistent and reproducible results. This is mainly due to the amount of knobs that you can turn to make the result different.
This time I wanted to be prepared especially considering the amount of instability I had to measure. That’s why I’m aiming to use a PS4 (standardized environment!) and collect more data points.
My PC analysis is made using 2min clips of me pressing buttons and counting by hand in the video by using a frame-by-frame viewing. Using a sync move I have around 60–80 data points for 17 classes. In this batch of tests I did 3min clips without sync move which gives me 180–200 data points. To speed up the analysis process, I created a python script that extracts exactly what I need:
- The green line position when the input happens to get the inner-frame timing
- The amount of frames before the character starts its animation
That way, even thought I had to fiddle with the exact conditions to have 100% precision extractions, it was worth it because in one day I was able to test 14 games.
The PS4 is a PS4 Pro graciously given to me by a staff member of the french team Bas Gros Poing. I tested everything that was on the console (I even added one more game).
The new stability metric
The previous stability metric was not well designed because it was dependent on the amount of samples. So I had to define properly this:
The stability is the probability of having the same input lag with the same timing
A stability of 100% means that when you press a button, the input lag is entirely predicable. A stability of 50% means that when you press a button, the input lag can vary by 1 frame even knowing the right timing.
The results in one picture
[UPDATE] Check the explanation and corrected results for SF30AC below
The letters represent the tier and each are 1 frame appart starting from “below 3 frames” for rank S and so on. All games with a stability above 90% are considered very robust.
Validation with noodalls results
The main test of noodalls is Blazblue. For him, it is the very reference of all PS4 tests.
And here is my version of this test using all the metrics
Now remember: we dont use exactly the same format for the report cards but the final lag calculation formula is the same (trust me we checked). The final result is clear: we get the same lag with ±2ms and a near 100% stability.
Conclusion: our methods can be considered equivalent!
Finally! We have a definitive answer on this question which was not so clear using PC. Yet another argument against PC configuration stability.
The shape of this report card is the reference: very stable results until the 14–15ms mark. After that everything is shifted little by little (because the input doesnt have the time to be considered for the current frame). All stable games have this report card shape.
Now, we can confirm noodalls statement: all blazblue games are the fastest games and can be considered as a reference points for PS4 games. UNIEL seems to be at the same level.
Street Fighter 30th Anniversary
This one is a bit tricky. First and foremost: let’s congratulate 3rd Strike which achieved a A-rank with stability. I don’t know the rank of the CPS3 version so we can assume this is decent enough and we can focus on the more troubling games. Here’s a report card for SSFII Turbo.
As you can clearly see, we have a ~50% stability across the whole possible values of timings. This is exactly why I needed a lot of data points: to confirm that for one input timing you can get two different values of input lag. What’s funny is that the shape is normal, just thicker. Really weird.
The same phenomenon appears on Alpha 3 and Hyper Fighting (didnt test the other games due to the lack of training modes).
[2018–07–15 UPDATE]
Few people on twitter mentionned that the stability issue may be due to the natural frameskips of the game. If you don’t know this, that is a design issues of early Capcom games where to speed the game up, some frames were just skipped.
Also, another person mentioned that normal moves in the SF2 series have one frame of additional visual lag. Which makes my testing flawed by 1f technically. I can confirm that throws for instance are 1f faster but it’s inconvienient to make 180+ data points without training mode.
Considering those two issues I used SF2 Championship Edition (CPS1 system like HF but with no skips), SSF2 (CPS2 system like Turbo with no skip) and Alpha3 with Speed=1 in the option panel (no skip). And here are the corrected results (SF2 games results are adjusted with -1f):
I get stable results so the emulation IS NOT an issue at least for stability. Although the input lag is still B-tier.
As a side note: there are no additional lag if you use filters in SF30AC or different screen size contrary to what was tweeted by damdai. I did the same test with different settings and the shape is the same. Judging by what I discovered, unless you do 100+ measures I doubt that any result can be conclusive.
Finally: the main engine is fast enough, the input display appears at a S-tier speed. But the actual animation starts after.
PS2 Games
I did two quick tests with Samurai Shodown and King of Fighters 98 UM. I reported the test of Samurai Showdown here.
The interesting part is that: firstly the lag is big for a game of that time (surely due to emulation) and secondly the bend at the bottom is a bit wider.
Results for KOF98UM are not reported because it is tricky to measure the start of the animation which can be confused with idle stance transition by an automated process. From what I saw the results are similar to Samurai Shodown.
Street Fighter V
Alright, let’s talk about the elephant in the room now. What the f*ck is this 40% stability with the highest lag of the 14 games I’ve tested… Let’s look at the report card:
Alright, you remember SF30AC results? It’s the same here, but worst. Trust me I’ve tested this 3 or 4 times and everytime I get the columns 5, 6 and 7 filled with results. Note: I get exactly the same results with the PS360+ in legacy mode.
Where does it comes from then? If you plot the frame number over test time, you get the following chart
We can now see some kind of periodicity of the results with the sequence of three steps. The period is about 1 minute wide (the x-axis is in frames). I have reproduced a similar shape on all my tests. This phenomenon is not observed for SF30AC so the issue is not the same.
That explains the weird results that noodalls obtained. In his protocol, he tests all the timings sequentially. Whereas I do random presses. That’s why I can get variations for a specific timing while he is potentially stuck in a specific part of the period. I’ll eventually ask him to do random tests to confirm this.
This instability also explains why people have found completely different results (I’m thinking about iBananaKiller on reddit and obviously DisplayLag).
I’d like to remind the reader that I’m running a PS4 Pro here and I’m on the training stage, so there is nothing laggy (also I can clearly see that no frame is dropped using Cammy’s hair or other little animations).
Finally: I have no idea on why this happens. I know that on PC an additional frame of lag between the input system and the game engine (to have the same experience offline and online I’m guessing), but this is another level of absurdity. It’s a shame that such a popular game has such a flaw and it’s been there since the launch apparently.
Conclusion
I encourage people to reproduce my results. You’ve got all the details of the methodology in the original thread. At first, I wanted to understand why SFV was such an issue in noodalls results. And now, I think that we have a new point of view on the matter now.
Aside SFV, we’ve got a good set of games that are played in very large tournaments: BBCTB is S-tier and GGXrd is A-tier with his UE3. After that, DBFZ and Tekken 7 are B-tier with UE4. Now B-tier doesnt mean it’s impossible to play, it’s basically similar to USF4 in PS3.
You can find all the data in this google spreadsheet. All the videos links can be found in the spreadsheet as well.
I would like to thank again Alba who lend me his PS4 with this large amount of games and Noodalls who helped me a lot on this setup.
