Best multi-object selection methods for virtual reality
Object selection is an important task in virtual environments and one of the most frequent activities for most common user interactions. Tasks like moving or manipulating objects are often preceded by a selection task.
The amount of information available around selection methods within virtual environments is rather extensive and describes in-depth the use of different interaction metaphors such as voice, gaze, laser, and virtual hands. Surprisingly not much is available when looking at MULTI-object selection methods.
Especially when looking at productivity apps there is a very tight relationship between the user’s performance and his ability to accurately select objects in space.
Creative tools are a great example as the action of selecting, moving and manipulating multiple objects simultaneously are core activities the user performs to give shape to his ideas and creative vision. Over the past years this category of tools has found various solutions to tackle some of the challenges that make multi-object selection in VR so unique.
The 2D world went through its share of experimentations and adopted some conventions: “click and drag” or “alt+click” both allow to select multiple objects, while “cltr+click” can be used to add or remove from the selection. 3D adds many more dimensions to the problem (pun intended):
- The selection of small distant object is more difficult than selecting what lies within the motor space of the user
- The addition of depth makes occlusion an ubiquitous challenge…
- …That just get worse in dense environments
- The subjective impression of the users about an interaction technique can play a bigger role than merely “performance”
We also have to consider the “embodied” aspect of immersive interactions and the ways in which we could potentially translate the physicality of grabbing objects with our real hands into a satisfying virtual interaction.
The aim of this article is consequently to compare four multi-object selection methods currently implemented in five creative tools: Google Blocks, Tilt Brush, Tvori, Maquette and Gravity Sketch.
The ultimate goal is to identify and share commonalities and differences that could be leveraged in order to innovate and design an even better user experience for future creative tools.
I will only focus on the “virtual hand” interaction metaphors as these types of creative tools already use a “clutch to move” system to allow the user to get closer to the target objects and “laser pointers” metaphors are usually adopted only when an interaction occurs with the UI.
Throughout the comparison I will look at four main aspects:
- Entry point of the selection tool: creative applications often require the user to switch between different “tools” to perform a wide variety of tasks (e.g. measure a distance, erase an object, modify a shape etc). The need to access a menu in order to pick the “selection tool” adds to the cognitive load of the user making the task of selecting objects more cumbersome and interrupting the state of creative flow.
- Discrete or Volumetric interaction method: the discrete and volumetric methods are analogous to the “ctrl + click” and “click and drag” respectively for flat surfaces
- How to add or remove from the selection: quite self explanatory😉
- Experience summary: represents my general impression of the interaction taking into account the overall experience and the level of embodiment (where embodiment refers to the level of physicality and immersion that a certain interaction conveys to the user).
Google blocks
One of Google Blocks’ unique selling points is simplicity. Using just six simple tools you can create low-poly scenes without prior modeling experience. This level of simplicity is also something transferred in its multi selection system.
- Entry point: there aren’t a lot of choices to be made but switching from creation to selection still requires swapping tools by looking at the main menu on the secondary hand. The choice is rather quick because instead of icons or text the developers decided to go for a very distinguishable and clear big 3D hand model to indicate the object-selection and grabbing tool.
- Discrete or volumetric: Google blocks selection method is discrete (with style). The style is in the way the system communicates back to the user its function and activation. By holding the grip button the hand clutches and a trail is left in space following the movement of the selection tool. This invites the user to use the tool as a sort of brush to select all target objects.
- Add or remove from the selection: this system completely lacks the ability to add or subtract from the initial selection and is probably its biggest weakness. In case of mistakes due to inaccuracy or occlusion the only option available is to deselect everything and start over. This makes the selection within a dense occluded scene a real nightmare.
- Experience summary: I personally like the simple approach and how affordances are communicated: a hand → is used to grab, A trail in space → makes me want to explore what would happen if I get it through objects. The fact that the size of the selection zone cannot be adjusted in conjunction with the inability to add or remove from the selection made the experience quite frustrating at times.
Tilt Brush
Tiltbrush lets you paint in 3D and was one of the first creative apps coming to VR. It also became completely open-source recently allowing developers to customize and expand its features. Out of all the ones in this list, it is the most free form and the one that in my opinion is in less need of a selection tool. On the other hand, it has a very different approach that I thought would be worth exploring.
- Entry point: The first step in order to access the selection tool is to enable advanced mode. After that’s done, Tiltbrush requires the user to swap between creation and selection like Google Blocks. Probably due to the wide variety of features and much broader possibilities when compared to its google brother, there are several ways of doing that.
- On the menu on the secondary hand it is possible to pick the selection tool indicated by the icon of a mouse cursor. The fact there is already a lot going on in the same control area (picking brushes, other tools, selecting color etc) made this my least favorable option.
- Luckily the “tools” panel can be detached from the secondary hand and be positioned in space. This saves the user a lot of actions and lowers significantly the cognitive load of navigating the UI.
- Last (and most “hidden” option) is to hold the A button on the drawing hand to open a quick-selection menu that allows to pick between different tools.
2. Discrete or Volumetric: Unlike Google Blocks, Tiltbrush uses a volumetric system. Holding the trigger changes the color of the sphere and makes it spin. This continuous movement gave me the feeling that the action of selecting was also continuous and made it clear that I could wave my controller to keep selecting multiple objects (yet using a radically different method: spinning sphere Vs trail). It is also possible to change the volume of the sphere (something I missed initially).
3. Add or remove form the selection: This was probably the most painful part of the experience with Tiltbrush. Subtracting from the selection is possible after enabling “deselect mode”. This would not be a problem if it wasn’t that the non selected objects would disappear in front of my eyes. Reenabling “add to selection” would make the rest of the scene reappear. It is not clear to me why the highlight around the selected objects wouldn’t be enough as a cue and the whole scene has to continuously “flash around” the user. On the other hand Tiltbrush offers the option to invert the selection, something that none of the other tools have.
4. Experience summary: Tilbrush developers were absolute pioneers in this field and laid the foundations for a lot of great interactions and subsequent experimentations. I like the freedom of setting up your own workspace and accessing the selection tools in various ways. Despite having a spheric selection area I still had the feeling I was holding a brush and the action of selecting multiple objects didn’t feel radically different from Google Blocks Subtracting to the selection is certainly an addition when compared to Google Blocks but I didn’t like the way the system exclude the unselected objects from the scene leaving me disoriented and a little confused.
Tvori
Tvori is a VR creative used by designers to create animated prototypes of spatial experiences, products, and services. It is currently available for PC VR and is scheduled for a release on Oculus Quest in 2021 🚀. An update in late 2020 implemented what is probably the most extensive multi-selection toolset but, as we all know, “from great powers derive great responsibilities” 🦸.
- Entry point: Tvori allows the user to select multiple objects without interrupting the creative flow. The user can start the selection in three ways (all of which do not require picking a specific “selection tool”):
- Pushing the A or B button on the highlighted objects
- Holding the trigger button to start a one-handed volume selection
- Join the 2 controllers and expand a two-handed volumetric selection
2. Discrete Vs Volumetric: Tvori offers both options to the user. Very much like Google Blocks or TiltBrush holding down the B button allows the user to wave through objects and select all the targets. As briefly mentioned in the previous section, Tvori allows the user to start a volumetric selection with one or two hands. The uniqueness comes from the fact that there are three different “shapes” from which the user can select: a sphere, a cube, and a straight line. Switching between these three “modes” is possible by flicking the thumbstick. To add to the depth of the system enabling the grid makes the selection volumes snap to the orthogonal axis, taking away some freedom but providing extra precision.
3. Add or remove from the selection: Adding to the selection is possible by clicking with the A button on the extra targets or holding the same button and using the brush-like selection mode. Subtracting instead is only possible using the A button but waving through objects that should be deselected is not supported at the moment.
4. Experience summary: Tvori team went ahead and implemented what probably is the most complex multi-object selection method currently available in the market (to my knowledge). Besides the number of choices available the experience is very polished and has a very high level of embodiment. To start off there is no functional distinction between primary and secondary hand allowing to simultaneously select objects with both hands. Moreover, the ability to expand the selection volume (or line) with 2 hands resembles a real-world movement (a bit like grabbing a large number of objects from a shelf with both arms). On the other hand, there is really a lot going on… so much actually that could be summarized in a flow chart. For a small group of objects, I ended up picking the path with the least steps because not making any choice felt just quicker. In more dense environments this “time Vs. cognitive load” economy would instead lean toward the volumetric approach.
Maquette
Maquette is another prototyping tool equipped with many functionalities from other 2D design tools like procedural shapes and text formatting. Maquette's multi-object selection method is very similar to Tiltbrush in many ways.
- Entry point: Like Tiltbrush you need to switch between creation and selection and that can be done by picking the selection tool on the secondary hand or enabling the quick menu.
- Discrete Vs Volumetric: Once the selection tool is selected holding the trigger makes the color of the pointer change and you can pick the various objects singularly or waving the pointer through them. By moving the thumbstick left or right the user can also turn the selection pointer into a sphere. Unlike Google Blocks or Tiltbrush the hint that the selection could be a continuous action (the trail or the spinning sphere) was not very evident and I almost missed this feature.
- Add or remove from the selection: This works very well and the sphere can be used to both add or remove from the initial selection.
- Experience summary: I found the overall approach taken by Maquette’s team very solid and functional. It felt the least brave of all the one discussed in this article, and that is certainly not a bad thing. When people approach these new tools having a familiar experience could be a tremendous added value.
Gravity Sketch
Gravity Sketch is probably the only real outlier in this list for a few reasons. It is a 3D sketching tool and has a rather different use case from the others. Consequently, the “items” that are often part of the creative process (and consequently multi-selection) are not full objects but parts of them like vertices, faces, or edges. Even though this might feel a little confusing I deliberately decided to include Gravity Sketch because of its uniqueness. It is also the only one that introduced a weighted approach to his selection (something I am going to explain shortly). Given the many differences, I decided to focus on the selection of various vertices rather than complete shapes.
- Entry point: In order to select vertices it is necessary to enable SubD mode (only available for SubD geometries). After a recent update, it is possible to select and edit the vertices (not faces) of all the selected objects by holding the SubD button. At this point, the wireframes of the selected objects appear and by flicking the thumbstick on the drawing hand right or left you can enable different selection modes (vertices, edges, or faces).
- Discrete Vs Volumetric: This is the only method in the list that is fully volumetric. A sphere attached to the drawing hand can be expanded or reduced in size. Overlapping the sphere with the vertices makes them black indicating that they are selected. Holding down the grip button while pulling the thumbstick down creates a concentric sphere. The inner and outer spheres assign different weights to the selected vertices. The vertex becomes bold the moment it enters inside the outer sphere and it gets gradually darker until it is completely inside the inner sphere.
- Add or remove from the selection: this is a feature missing from Gravity Sketch and a major challenge to tackle with this completely volumetric selection method.
- Experience summary: Grabbing vertices and faces in Gravity Sketch feels very “stretchy”. The selection sphere does a great job when dealing with surfaces but less so in other circumstances where I would have preferred to select separate and distant faces or vertices. The way the weighted selection is introduced is very intuitive (despite being a little hidden) and a great implementation to allow a different degree of manipulation and new creative opportunities.
This was a rather descriptive approach to the “state of the industry” when it comes down to multi-selection methods in virtual reality. Comparing the effectiveness would be unfair and very much useless since task performance is very dependent on the subsequent step of the creative process: manipulation. The different tools are also geared towards users with different goals and levels of expertise. Google blocks simplicity makes it an ideal entry point for a casual user, Tvori focuses on spatial designers, while Gravity sketch’s main audience is made of product and industrial designers. Nonetheless, some of the approaches developed by the different software could be adapted to other types of interactions and I hope this structured overview will inspire designers and developers to push the limit forward and lead to the new generation of spatial interactions.