Controller Input Lag — How to measure it?
IMPORTANT NOTE: I have updated this article and explained most of my knowledge about input lag on my website inputlag.science. Go there to an updated view and more measures.
As you may know, I’m a huge input lag nerd. I’ve spent a lot of time trying to figure out what can introduce latency in a video game setup. But lately as thought I would revisit how controller are tested.
Current methods
As far as I know there are two main ways to measure controller lag.
- Controller Versus: plug two controllers in a system with a fighting game; bind the same button to both PCB; press it and count how many times the players have traded or not. The most notorious analysis is Teyah’s study (a lot of details there!).
- Video interruption: use the video interruption input lag method on a very stable game (say BlazBlue CPE). By analysing the report card, you are able to estimate the lag distribution. Obviously, the one behind this idea is Noodalls who did a full report specifically on this topic. Having tested this method myself, it is pretty effective but requires some hardware.
Obviously, there is the LED / Camera standard input lag method which is not specific to controllers but you could still use this to get an approximation. As a matter of fact some people have done extensive studies about this. I’m also aware of this post where Jim Hejl shows their input lag system which does man-in-the-middle on usb on a wired PS4 but afaik there are no details.
The issue that I saw in those methods was twofold. Firstly, they are not really convenient and need a lot of data point to get the full details of the performance. Secondly, you always need a complete system (console or pc/game) which introduces a lot of unknown variables. Also, I just wanted to push the idea further.
That’s my mindset going in: I want direct test of the device, no extra system, and if possible easily reproducible.
Understanding USB for controllers
First, we need to understand how a system gets the state of a controller. I won’t go into great details here because USB can be hella complex and it’s not the purpose here to go too deep.
The general idea is the following:
- When a device is plugged, the host (meaning the system which receives the controller) queries the device for it’s descriptor. This will describe what the device does and how to talk to the device.
For instance, it can say that it’s a controller and it will give the bus address to talk to and a polling rate. - Afterwards, if the host desires to get the status of the device, it can send a request on the desired address every few milliseconds according to the polling rate specifications.
This is a generic protocol, however it works only if the host knows what the device is talking about. Fortunately, there are two known protocols to get the state of controllers: HID (standard) and XInput (microsoft).
The main takeaway behind all this is: the device never sends its state by itself. The host asks the device, at his pace. And we will see that this has obviously significant impacts.
Hardware setup
Back to input lag. Knowing what we know, how could we measure input lag? The core idea is very raw: create a very small USB host, send a signal to press a button, measure the time when the device status will change. The remaining question is how to create a small USB host, and the device that changes everything is this thing: the USB Host Shield by Oleg Mazurov.
I’m actually not using this exact configuration (I have a Arduino Mega and a clone of the USB Host Shield rev1) but it’s the same, just more ghetto. Retail price of this combo is around $50 but you easily find clones of those boards everywhere for way cheaper (even <10$ if you’re way cheap) and it will work the same.
This is able to talk to USB devices using a programmable device. The only thing we need now is to activate a button on the controller using a wire bound to one of the free pin. Assuming you have a common ground PCB, you can wire it like the following schematics. Note that you don’t need soldering just hold the wire it takes just a minute to run a benchmark.
If you don’t have a common ground controller (xbox 360 for instance) it will need a bit more wiring that I won’t detail here because it’s too specific and better electronic enthusiasts than me will be able to help. Check on the SRK forums if someone made a dualmod setup with your PCB and check what parts they used.
Now we just need to install the software I prepared (follow the install instructions there) and you’re good to go.
The experimentation protocol
Once everything is plugged correctly plugged, we can start our device and this is what it says in the Serial Monitor of the Arduino IDE:
- The device is a Hori FC
- It supports the HID protocol
- Its polling rate is 5ms
Now if we send the command to launch the test it will change the state of the button 1000 times with random timings (to simulate random presses). The arduino program will time the duration between the change and when the state of the device changes.
Obviously, the polling rate will change a lot of things in the results. If your device is polled every 10ms, you won’t be able to have a 1ms latency all the time. You can tune that to three main modes.
- Descriptor: use the value that was given by the device.
- Usual: use the next power of 2 (1, 2, 4, 8, 16, 32ms). This is how usual systems like the PS4 or a PC will poll a device.
- Overclock: just use 1ms. Some systems uses this overclocking like the undamned USB decoder (confirmed by undamned himself) to read the device as fast as possible.
Afterwards, we will have 1000 measures for each polling rate, for each device. That gives us plenty of data points. Now, we need to measure a bunch of devices and see what happens.
Results
For my first measure I’m going to focus on the Hori Fighting Commander 4, 1st Gen. This has a native 5ms polling interval.
Let’s break down what is displayed here. The graph on the left shows the distributions of lag in milliseconds. The most important area is the blue one which shows how the device behaves in standard usage. Closer to 0 is better, thiner is better.
On the right side, you have a bunch of statistical metrics: minimum, maximum, average and standard deviation [note for the non-statistical folks: you can see the standard deviation as how much you are likely to see a value deviate from the average]. For instance, we can see that the standard deviation goes down as we poll quicker and the average is obviously faster as well.
In a 60fps gaming setup, assuming that the controller is checked once per frame by the game, the device status will be checked in the internal driver every 16.6ms (independently from the polling rate). I calculated a probability of having your input too late for the current frame in this context. For instance, here we can see that the FC4 will not be fast enough 47% of the time to be treated in the same frame as the button press.
Let’s look at another device. We know that the PS360+ in PS3/4 mode is one the fastest PCB available (it was Teyah’s reference). That’s what I used for all my lag tests [note: I know UFB is ever so slightly faster but I don’t have one at my disposal as I’m writing this].
This is what we want to see in a device. First observation: the polling interval is natively 1ms. This will instruct the host system to check the state of the device often, reducing variability. Then, we can now confirm that the PS360+ is very fast because its statistical results are astounding: 1.7ms ± 0.2ms. This device will hardly be the reason your input have skiped a frame. Perfect. However, let’s look at the XBox 360 Mode that was noted laggy by Teyah and others.
Here we can see that this device doesn’t behave properly. Even on an overclocked polling rate we get 5ms±1.4ms which is still pretty high. But on the usual settings, the variability and average are so high that the chances of the device to be on time are around 11%, and we have a high percentage of chance to have a 2-frames skip.
Let’s see the new revision of the FC4. They definitely improved the PCB on the input lag side. It is still not perfect but it is playable without back thinking. Few variability even with a 4ms polling interval and a decent average. But what is most surprising would be the PC settings of this controller. This setting changes the protocol to XInput (not XBox compatible thought).
This mode is even better. If it was described with a 1ms polling rate, it could be faster than the PS360+ in that mode. That is why I’m displaying the overclock scenario: to show you that lag is sometimes just a configuration value. Other than that, the results are quite impressive with a no skip rate higher than 80%.
I know this device is pretty laughable to see in real life. But there are two aspects that controller modders like about this one: 1) it is cheap 2) it has the easiest PCB to mount. That’s why my arcade stick has a mini 4 in it for now and I’m pretty sure other people have this as well. Unfortunately, this device is not great at all. On average only one third of your inputs will be on time.
You may have been wondering why my X-axis went all the way to 40ms. That’s why. This is my first arcade stick and I know a bunch of people who had it mainly because it was full-sanwa parts that regularly went on sales. Unfortunately this stick has the worst PCB, by far. In the usual settings, there are no chance that your input will be on time. None. You even have a 44% chance to see your input two frames later.
[update 2019–01–30] Then let’s turn to the most common controller that you will find on PS4: the dualshock 4 obviously. This device operates in two ways: over bluetooth, or over USB. Let’s look at the USB first:
This is half a surprise. The DS4 is notorious to medium lag over the USB. What pains the most is that: if Sony decided to put “1ms” in the USB description, the device would be blazing fast (very near PS360+). Now, let’s measure the bluetooth performance.
Look closely to the “usual” results, this performance is impressive and is very similar to what has been measured by noodalls. (nb: I know that there is another similar measure done by a japanese twitter account but I can’t find it again). The performance is better than the wired USB only because of a protocol configuration issue. Note that I did another measure with an old dongle in green and here the bluetooth protocol introduced latency. From what I’ve seen, if you don’t poll the BT dongle enough or if your dongle is too slow, then you will have latency issues. Finally, I would like to point out that due to the wireless nature, sometimes packets get lost and you have few data points that have +10ms latency.
You can find all the detailed results and raw data in the following spreadsheet.
Validation
How do I know that this method works and how well is it accurate? The first issue that could be a concern would be the speed at which the arduino works. I made a command to check the speed of the controller. If the device is not contacted, the main loop of the program takes 64µs. If the target device is polled during the loop then it obviously depends on the device but the loop takes around 340µs. Thus, the millisecond precision is definitely good.
To be sure that is accurate to the target device, I tested the FC4 first gen on PS4 using the most stable game I know: Blazblue CPE. Using my standard protocol to measure game input lag, I’m able to detect the percentage of frame skip on each frame timing. This way, I can confirm if my cumulative distribution matches or not. And here are the results
We can see that the yellow curve (4ms) follows strictly the PS4 measures, validating the method. You can check the raw data here.
Conclusion
That was a long one. If you read until this point: thank you so much. If you just skim it: no worries I do this all the time. As a gift, here’s the general graph of the devices I tested using the frame skip probability metric.
This project actually worked better that I expected. Now here’s what I would like people to do: please test your devices. You don’t need a lot of hardware and you definitely don’t need a lot of knowledge to do those. I published everything I have, even a template to analyze your results so it’s up to you now.
I tried to use the knowledge built here to reverse-engineer the behavior of each devices. Unfortunately, I was not able to do a simplified model of each devices, I have some leads but nothing definitive. Maybe later.
That’s it for me. Don’t hesitate to ask questions on my twitter WydD.
As always, enjoy!
Anticipated Questions & Answers
- Can you test {device-name}?
Unless you give me the device, I don’t have it and I don’t intend to spend money on buying controllers. That said I didnt test XBox 360 wired or DS4 due to the complexity to access the pads or to make it work with my setup yet. UFB will come soon as well (noodalls has graciously gave me one). - I don’t understand the lag distribution graphs
I tried to make an illustration on how to interpret this chart. I hope it’s decent enough.
- What about converters?
I don’t have any to test. But the idea is pretty simple, you just wrap a very fast encoder (and we know that Brook knows how) on top of a decoder. I don’t know if Brook overclocks its polling interval like the undamned USB decoder. If you know, please tell me.
