Automatic, GPU-based object highlighting in deck.gl Layers
deck.gl 5.0 introduced automatic, GPU-based highlighting. Enabling this feature (by adding the new autoHighlight property to your layers) means that objects in deck.gl layers are automatically highlighted when the mouse “hovers” over them. Using the autoHighlight feature can significantly simplify the creation of interactive visualizations. It eliminates the need for the application to do things like adding event callbacks to keep track of hover state, or rendering additional highlight layers, just for the purpose of highlighting the hovered object. In addition, since all highlight-related operations (object comparison and rendering) are performed on the GPU, the highlighting is extremely fast and can be used with confidence even in layers with millions of elements.
Auto Highlighting
All of the deck.gl core layers support “Auto Highlighting”, all you need to do is set the props to enable picking and auto highlighting.
Example code to enable automatic highlighting:
new ScatterplotLayer({…pickable: true,autoHighlight: true})
Example code for controlling highlight color:
new ScatterplotLayer({...pickable: true,autoHighlight: true,highlightColor: ...})
With this feature a single object currently being hovered by the mouse is highlighted. When enabled, an additional render pass is performed to obtain details of the current object under the mouse pointer and passed to the next drawing cycle. This data is consumed by the GPU to highlight the object. To enable this, set layers ‘pickable’ and ‘autoHighlight’ props. This feature is implemented using deck.gl’s “Shader Module System”. The following sections give more details about shader modules.
Shader Module System
To perform tasks on the GPU one has to write shaders. deck.gl comes with a set of shader modules (like, “picking”, “project”, “lighting” etc), each focused on performing a task. A shader module provides a set of functions and takes a set of parameters that are passed to the shaders as uniforms. For example, the “lighting” shader module provides the “getLightWeight” method that will calculate ambient, diffuse and specular light factors and returns the amount of light falling on a given vertex. To enable lighting in your layer, enable the “lighting” module when creating a “Model” object and use the “getLightWeight” method directly in the shader code, no need to write all instructions needed for lighting. Internally the deck.gl Shader Module System will assemble all functions from the enabled shader modules and make them available to use in layer’s shaders. Also, during runtime, light settings (like light position, light strength, etc) can be sent to the GPU by setting lighting shader module parameters, which internally the Shader Module System converts to valid uniforms and sets during rendering.
Picking shader module
The “Picking” shader module provides functions to perform picking operations on layers. Every time a pick operation is requested on a layer (like a mouse hover), deck.gl renders all geometries using the picking color on a separate framebuffer texture, that is not shown in the screen. Each geometric instance gets a unique picking color, generated by an instance index and layer index, for example in a scatterplot layer, every circle gets a unique picking color. Once picking colors are rendered, based where layers are picked, a single picking color is read from GPU textures to uniquely identify the layer and geometric instance within that layer that is picked. This information is passed to the application, which can then collect information of the selected instance and present it to the user.
Implementation
Object highlighting is implemented as an extension to the “picking” shader module. New methods and parameters are added to the “picking” shader module, that will detect a picked geometric instance and render it using a highlight color preset during draw time. All the layer has to do is use these methods in the shaders and set the parameters during draw time. All core layers shaders are updated with these methods and base Layer class sets all the parameters, enabling this feature in all our core layers.
Example shader code for enabling auto highlighting:
In the below code samples, “picking_setColor” and “picking_filterHighlightColor” are methods provided by the “picking” shader module.
Vertex shader:
attribute vec3 aPickingColor;main() {...// Compares selected instance picking color with current instance picking color// passes results to Fragment shader.picking_setColor(aPickingColor);...}
Fragment shader:
main() {...// Select actual color or highlight color based on this fragment is picked or notgl_FragColor = picking_filterHighlightColor(gl_FragColor);... apply any filters on gl_FragColor ...}
Once the layer props are set actual operations are performed on the GPU using the “picking” shader module. Following steps summarize how this is achieved in the GPU:
- CPU provides input to GPU : Layer sets couple of “picking” module parameters, one to uniquely identify the object to be highlighted and the other to set highlighting color. On every mouse move event, an additional render pass is performed to render all layer geometry to a offline Framebuffer object, where each object is rendered usings its unique picking color. Then based on current mouse position, a color value is read from the Framebuffer’s texture and set as module parameter. Also highlight color is set as module parameter. These are converted into uniforms and set for the next render cycle.
- Highlighting happens on GPU : Once the uniforms are set, all required operations for highlighting are done in the Vertex and Fragment shaders. Vertex shader compares the current vertex picking color (provided as an attribute) to selected object picking color (provided as uniform in first step) and passes the result (true/false) as a varying to the Fragment shader. In Fragment shader, if this result is true, it outputs highlighted object color (set as a uniform), if false, it uses actual fragment color. Doing this in Fragment shader also enables applications to apply any additional filtering on the highlighting color for advanced graphics effects (like per-pixel-lighting, shadows etc.), more details below.
Integration with advanced graphics effects
Auto highlighting is designed in way that allows to integrate highlight color with advanced graphics effects. As shown in the following luma.gl demo the top cube projects its shadow on the bottom cube and both cube faces get different lighting as they rotate. These graphics effects are achieved by using custom shaders. When using “picking” shader module, users can integrate the highlight color with any custom graphics effects. As shown below, the highlight color is correctly blended so that lighting and shadow effects are still visible. This kind of tight integration is not possible if you are rendering additional layers or objects to achieve object highlighting.
Custom Highlighting
In this mode a single object of choice is highlighted by setting its instance index to ‘highlightedObjectIndex’ layer prop. When set, layer calculates and sets the uniforms needed to highlight this object. When set irrespective where the mouse position is, object corresponding to the ‘highlightedObjectIndex’ is highlighted, this index can be programmatically changed, for example this value can be changed when user hits arrow keys. This setting takes precedence over Auto Highlighting.
Highlighting feature is specific to a Layer, when rendering multiple layers, each layer can have their own settings for Highlighting. I.e. an application can render 3 layers, one without highlighting, one with Custom highlighting and another one with Auto highlighting or any other combination.
Limitations
The GPU highlighting can only highlight one object (or picking color) per Layer. If you need to highlight multiple objects, you will need to resort to traditional techniques. We are working on adding this feature in future revisions.
Conclusion
I hope you enjoyed the exploration of behind-the-scenes for object highlighting; looking forward what you’ll be building with this new feature in your applications!
