Rich text within Twitch for iOS
Rich text is pervasive across the Twitch iOS experience, from channel chat to our newer social features such as Channel Feed and Whispers.
The expectation to have a responsive chat during video playback requires us to have a rich text implementation that minimizes the burden on the main thread and does not leave a big memory footprint.
Out of all the features in Twitch with rich text requirements, channel chat was the feature that required the most care to keep up performance. In this article we will walk through the challenges we faced to make the chat experience great on iOS, and the different solutions we tried.
A brief history of rich text on iOS
Prior to iOS 6, UIWebView was the simplest way to render rich text. CoreText was introduced in iOS 3.2 as an alternative but, being a low-level Framework, integration was difficult.
With iOS 6, Apple added NSAttributedString support to UITextView and other UIKit controls to render rich text. However this offered no advantages in terms of performance or memory since UITextView rendered using a web view under the hood.
The situation appeared to be improving with iOS 7, when Apple added the TextKit framework and switched UITextView to be backed by TextKit instead of WebKit.
Note: UIWebView is used to embed web content. NSAttributedString manages character strings and associated sets of attributes such as font, color etc. UITextView supports the display of text and supports text editing.
Evolution of Twitch Chat
The implementation of Twitch chat evolved alongside Apple’s APIs. For the sake of expediency and acceptable performance, our first implementation was based on UIWebView, but its high in-process memory usage would result in the app being jettisoned while the user was engaged in a broadcast. Once we moved our minimum requirement to iOS7, we wanted to switch chat to TextKit and bid adieu to UIWebView.
First attempt with UITextView
Our first attempt at replacing our UIWebView backed chat with UITextView fell short of our expectations.
The three main issues were :
- Images were presented using Core Graphics’ software rendering path, saturating the main thread.
- The main thread saturation issue was further aggravated by expensive calculations related to updates to UITextView’s underlying NSTextStorage.
- There were a lot of unwanted relayouts and notifications triggered by UITextView and its backing NSTextStorage.
Note: NSTextStorage is the fundamental storage mechanism of TextKit.
The main thread was spending a lot of time in the render phase of each message, which included all layout related calculations. This caused the main events queue to get saturated and UI related events such as a scroll event to be processed with significant delay, resulting in sub-60FPS frame rates.
Switch to hardware rendering of images
With our naive UITextView based approach, the main thread was bogged down with the task of software rendering images.
Our solution was to create bitmaps for the individual images and assigning the contents of those bitmaps to CALayers and adding them to the UITextView’s backing layer.
Although this improved performance, the main thread was still saturated during fast chats due to the other expensive calculations mentioned earlier.
Note: A bitmap is simply an array of pixels. A CALayer can be thought of as the backing drawing unit of a view.
Processing layout calculations on background threads
With no way of decoupling UITextView and its TextKit stack from the main thread, we decided to investigate a UITextView free system.
Our solution to free the main thread from CPU intensive layout related operations was to switch to a multi-threaded rendering pipeline.
The process of going from a raw chat message to pixels rendered on the screen involve 4 primary steps.
- The main thread dispatches layout calculation events to a background queue. (Let’s call this the layout queue.)
The main thread dispatches expensive operations related to layout and building of the backing text storage to the layout queue.
The main thread is no longer stuck processing the CPU intensive render events and is now free to process any UI events, allowing scrolling in our chat to be as smooth as silk.
2. The layout queue processes the render events.
The layout queue is backed by a pool of background threads. Available threads pull from this queue and perform layout calculations using a TextKit stack. Once the layout calculations are done, we create a bitmap and draw the text onto this bitmap (just like we do with images) in the background thread itself.
3. The background threads dispatch back to the main thread.
After a bitmap for a particular message is generated in a background thread, an event is dispatched back to the main queue to process this bitmap.
4. The main thread processes bitmaps.
The main thread processes bitmaps in the order received and updates the UI. Each message has a corresponding CALayer associated with it. The only operation done on the main thread is to swap this backing CALayer’s contents with the corresponding bitmap. Meanwhile, the background threads can process render events for other chat messages in parallel.
Additional Support
We added support for link detection, copy/paste, maximum number of lines to be displayed, and truncation to bring our custom text view to have feature and behavior parity with Apple’s UITextView.
Results
To compare the main thread utilization in the UITextView backed implementation and our custom text view implementation, we can look at the CPU usage on the main thread over a small time sample while in a high velocity chat.
As can be seen the average main thread (Thread 1) utilization for the UITextView implementation is quite high.
Now, let’s take a look at our custom native implementation’s performance.
The average main thread usage in our custom background layout enabled implementation is significantly lower than the corresponding custom UITextView implementation. We reduced the average main thread usage during high velocity chats by almost 50%!
Note: These cpu reports were generated using XCode.
Let’s take a look at a more compelling visual to truly appreciate the difference in responsiveness in chat. We alluded to the issue of frame rate dropping below 60FPS in the UITextView based implementation earlier.
As can be visibly seen, lots of image attachments causes responsiveness during scrolling to be greatly compromised.
Let’s compare that with scrolling in the background layout enabled chat implementation.
The difference is obvious. Scrolling in a chat, dense in images no longer stutters.
When compared to the UIWebView backed implementation, we reduced our memory usage on initialization of channel chat by 10MB, without compromising our chat’s responsiveness.
In conclusion, to quantify the powers of our native implementation….
Here at Twitch’s iOS team, we are constantly working to figure out ways to improve our users’ experience. We want to ensure that as our presence on the platform grows, our app continues to perform well. If these sort of challenges interest you, we are hiring!
