Negative Latency: Understanding Human Intention meets Anticipatory Networking
Co-authors : Patrick Seeling
This post summarizes a demo paper where we presented a testbed to evaluate how far in advance one can infer grasping intentions based on glove data. The demo has been accepted for presentation at the UbiComp / ISWC 2023 conference.
The Challenge of Network Latency
The metaverse promises a digital universe where we can interact, explore, and create. However, to fully realize the potential of this virtual realm, developers must overcome a crucial hurdle: latency. In this blog post, we delve into the challenges posed by existing communication networks and how the concept of negative latency can help to achieve true metaverse immersion.
Latency in the metaverse refers to the delay between an action and its corresponding response. In the context of communication networks, it encompasses the time taken for data to travel between network nodes, be processed on servers/cloud and be sent back to users. The hallmark of a successful metaverse is its ability to feel immersive to users. For this, it must cater to our senses — sight, sound, and touch. While audio and visual stimuli can tolerate a degree of latency, the tactile sense requires minimal delay. Human brains are wired to perceive touch with astonishing sensitivity, demanding less than 1 millisecond of latency to perceive it as instantaneous. Including other factors like computation on network nodes limits the distance from metaverse server to client or client to client to be no larger than 25km. Moving beyond a 25km radius, thus, requires new approaches to decrease the delays imposed by hardware, network, computing, and physics as limitations.
We use the term negative latency to generally describe predicting the future demand of network services or communicating before an event happens. It means that network providers can proactively orchestrate network components and allocate resources to fulfil latency, reliability, and security demands of metaverse applications. In practice, this could mean that an ultra-low-latency connection for tactile rendering between a client and server is established prior to the launch of the application requiring such a connection. If such connection already exists, data could be send prior to an event, resulting again in negative latency .
Demo Setup and Experimentation
As providing network functionalities takes time, we were interested in the time budget a.k.a. negative latency that can be achieved by predicting human intentions from their grasping behaviour. This is a widely investigated topic in the field of neuroscience, robotics, human-machine interaction and social studies. Our hand gesture capture system features the Manus Meta Prime 2 glove to sense angular information of finger shape, a force-resistive sensor to timestamp the moment of interaction, and a laptop. We trained an off-the-shelf neural network to anticipate the grasped object based on the gradual finger-shaping of the grasping motion. We incorporated eight objects commonly found in daily living and collected a small-scale dataset of subjects grasping multiple objects and trained a neural network to anticipate the object to be grasped. To handle the trade-off between predicting future events and corresponding uncertainty, we calibrated the model. As the model gradually becomes more confident as the grasping motion progresses, we timestamp the moment the model’s confidence exceeds a set threshold. Our demo setup allows us to gather timestamped data to estimate the time budget to attune the network to services.
While we empirically found that negative latencies of up to several hundred milliseconds are possible, forecasting grasping intention comes with inherent challenges. Similar objects might not be easily distinguishable and trying to anticipate the future further ahead comes with higher uncertainty. Moreover, different people excite different grasping patterns, requiring subject-specific models. In addition, as the anticipated intentions would be used for decision-making for network resource allocation, having trustworthy estimates of model uncertainty will be key to deploying these methods in practice.
Conclusion & Future Research
The concept of negative latency, predicting future demand for network services, offers potential solutions to mitigate network latency. We explored this idea by capturing human grasping behaviour data to anticipate intentions, which led to encouraging results, though challenges remain in dealing with uncertainty and the similarity of objects. The next step is to create a large dataset of timestamped grasping motions to benchmark different anticipation techniques and make interpretable inferences on the cues that give away certain intentions.
