Stylus Low Latency
Creating organic Stylus experiences with Low Latency Graphics and Motion Prediction APIs
Introduction
With the increasing popularity of tablets and foldables, users are adopting new behaviors with their devices. The larger screen area enables new experiences and empowers productivity tasks that were previously reserved to laptops and desktops.
Handwriting, drawing, and sketching with a stylus unlock precise and organic workflows, like we could perform in the analog world. Yet the delay between gestures and on-screen rendering can prevent the natural feel like writing on a piece of paper, this is called latency.
I’m glad to share that the Android team has released two new libraries: low latency graphics and motion prediction, currently alpha02. The libraries address stylus input latency on Android and ChromeOS.
The goal of the low latency graphics library is to reduce the processing time between stylus input and screen rendering. The motion prediction library leverages a powerful algorithm to predict where the next stylus move will be. Implement both libraries together to provide the lowest perceived latency possible.
In this article, we are going to explore how these libraries work and how you can implement a best-in-class stylus experience in your app. While we focus on the stylus experience, these two libraries work with any pointer, finger, stylus, capacitive stylus, and mouse.
But, before entering into the details, let’s take a look at how good this experience can be:
Low latency graphics: under the screen
Without any latency optimizations, the system has to go through many steps to render pixels on the screen. The low latency graphics library minimizes latency by leveraging front-buffered rendering. The result is faster rendering to the screen.
Understanding traditional rendering
Traditional rendering uses multiple buffers. This ensures smoothness in user experience with no visual tearing, but the trade-off is additional latency between the user interaction with the screen and when the content is visible on the display. To learn more about rendering and buffers, watch the Project Butter presentation.
With a multi-buffered rendering approach, the display-buffer contains data the user sees currently on the screen, and the rendering buffers are used by the graphics engine to renders the data (into memory).
Once a render is finished, the display-buffer is swapped with the rendering-buffer so the user can see the result on the screen. The rendering-buffer becomes the display-buffer and the display-buffer is freed for future use, where the rendering now happens.
This is possible thanks to the SurfaceFlinger and WindowManager. The animation below showcases the process.
Front buffer rendering: directly to the screen
To achieve the front buffer technique, front and double buffered layers are introduced. The front layer is designed for quick rendering of small portions of the screen. The layer is transparent and ephemeral, as we will see in a moment.
The front layer inputs are rendered to the double buffered layer to persist the work. And then the front buffer is hidden.
But you may wonder how the front buffer manages to render faster than the regular doubled buffered approach and what are the caveats.
The low latency graphics library writes directly to the front buffer — in other words, directly to the screen. This works because only a small area of the screen is changed (only a small portion of the stylus stroke is modified each frame).
Once the stylus is lifted, the regular rendering with the double buffered technique is resumed, and the stroke is persisted.
When to use the low latency graphics library
The front buffer technique works best for specific use cases — in particular, handwriting, drawing, and sketching. These actions modify only a small portion of the screen, which is where the library shines. If we were to modify bigger areas, this could create tearing and artifacts.
The low latency graphics library is not intended for rendering an entire screen, for example a game, or when large areas are modified like panning or zooming. Android has other tools, such as Android Game Development, and the regular OpenGL rendering, that help developers in those circumstances.
The low latency graphics library is available for Android 10 (API level 29) and higher and ChromeOS devices running Android 11 and higher.
Before we see how the code looks, you may want to think about your application and evaluate what work should be rendered to the front buffer (short work that affects a small part of the screen, like a stroke) and what should be processed in the double buffered layer (larger areas, like panning and zooming).
Implementation
Let’s start coding! The GLFrontBufferedRenderer works with your own data type, which would typically contain x and y coordinates but also color, brush type, pressure, and so forth.
The method requires two callbacks where OpenGL code will be executed, one for rendering on the front buffer (delta of the stroke) with onDrawFrontBufferedLayer, and the other for double buffer (persistence) with onDrawDoubleBufferedLayer.
I recommend the Stylus for your App presentation, from my colleague Nader Jawad, as well as the Android Graphics presentation released at the 2022 Android Developer Summit.
You can also consult the low latency graphics library documentation.
Now that you know how to reduce latency on the graphics layer, let’s see how we can further improve responsiveness by reducing the perceived latency with the motion prediction library.
Prediction: staying a step ahead
The goal of the motion prediction library is to stay a step ahead of rendering and reduce the perceived latency even further by creating artificial points based on the mathematical-prediction-powered Kalman Filter algorithm. Motion prediction improves the user experience by saving additional milliseconds and contributing to the organic feeling that the ink flows from the stylus.
Prediction techniques: the Kalman Filter algorithm
Without diving too deep into how the Kalman Filter algorithm works, it’s important to know that the algorithm is based on direction, velocity, pressure and time to predict where the next point will be.
The algorithm predicts very quickly by leveraging a set of equations with little data and by understanding the variation and uncertainty of the input.
This method of prediction is fast and memory efficient. The algorithm is used to track planes, satellites, and now stylus motion on Android. Other prediction techniques may be used on different devices, depending on OEMs.
Predicting motion on Android
The motion prediction library gets real user inputs as MotionEvent objects. MotionEvent contains information about x and y coordinates, pressure, time… all leveraged to feed the motion predictor and predict the next MotionEvent.
As we have seen earlier, the touch sample rate can vary from one device to another. The higher the touch sample rate, the faster data points can be provided to the motion predictor, the faster the algorithm can predict accurately. The motion prediction library is able to detect the sampling rate and use it to gather information faster and improve the prediction accuracy.
When to use motion prediction
The motion prediction library is based on MotionEvent, so as long as you’re working with MotionEvent objects, you can leverage the library.
Predicted motion events are artificial and inaccurate by definition. Use the predicted event to reduce the perceived latency, but ensure the predicted data is replaced with the real data once it is received.
The motion prediction library is available from Android 4.4 (API level 19) and higher and ChromeOS devices running Android 9 and higher.
Implementation
The motion event predictor offers two methods, record that takes MotionEvent objects and predict that returns a MotionEvent .
See the motion prediction library documentation for more details.
Performance directly related to screen features
For touch screens to register pointer inputs with accuracy, the system must frequently check the screen for touch. The touch or stylus sampling rate is how often the system checks the screen. Most screens have a touch sampling rate that’s twice the screen refresh rate. For example, on a standard 60Hz display, the sampling rate would be 120Hz. In other words, the screen will record touch input every 8ms.
The stylus sampling rate can be even higher, on ChromeOS some devices have a stylus sampling rate up to 400Hz. On those screens, the user experience is even better as the screen is more responsive.
Conclusion
In this article, we explored two ways to improve the stylus experience and reduce latency when it comes to drawing, sketching, or handwriting. The low latency graphics Jetpack library enables applications to render faster. In addition, the motion prediction Jetpack library predicts the next MotionEvent to always stay one step ahead of the rendering engine.
Combined, these innovative libraries help you achieve best-in-class stylus experiences.
We want to hear from you
The low latency graphics library and motion prediction library are available as an alpha release today, so you can start integrating and testing them in your apps. We look forward to incorporating your valuable early feedback as we continue to improve the libraries in future releases.
Please share your feedback for the low latency graphics library at androidx.graphics feedback, and for the motion prediction library at androidx.input feedback. Thanks so much for reading. Happy coding!
