How well does my robot work?

RoQME: A project aimed at measuring quality of service in robotics

Photo by Alex Knight on Unsplash

In the coming years, service robotics is called to play an important role in our lives. Robots will not be limited to simple domestic tasks, but they will end up being our companions, co-workers, assistants in shopping centers, or caregivers for our elders, to name just a few examples. Much of the research in service robotics focuses on functional aspects, that is, on the development of the robot tasks. Note that domestic, outdoor or crowded public spaces, like any other real-world environment, are inherently open-ended and, therefore, show a huge number of potential situations and contingencies. This poses a great challenge to the functionality of a robot and, as a result, the non-functional aspects are usually forgotten.

But, what do we mean with non-functional aspects? Broadly, non-functional aspects (a.k.a., non-functional or extra-functional properties) define “how” a system performs rather than “what” it does. Examples of non-functional properties include safety, reliability, efficiency or usability, among others. These properties can be observed during the robot operation to provide information about the quality of its service. Although engineers can use this feedback to evaluate and improve their designs, ideally, a robot should be able to adapt its own behavior automatically in order to optimize certain non-functional properties, such as user satisfaction or energy consumption.

RoQME (Robot Quality-of-Service MEtrics) is a new project coordinated by the University of Extremadura and participated by Biometric Vox and the University of Málaga. It is a 1-year project, as of March 2018, funded by the EU Horizon 2020 as part of the RoMoSys project. RoQME aims at helping roboticists to deal with non-functional properties by providing them with a toolchain to create global Quality of Service (QoS) metrics. The automatic estimation of these metrics at runtime, based on the contextual information observed, can then be used for different purposes, such as the adaptation of the robot behavior, among others.

A robotic salesman as an illustrative example

1. The robot body

Figure 1. A possible design of the salesman robot. Image obtained from CLARC: a Robotic Architecture for Comprehensive Geriatric Assessment

2. Programming the functionality

Going back to our example, we can use the SmartMDSD Toolchain to develop the software of our salesman robot. This is an Integrated Development Environment (IDE) for robotics that is compliant to the RobMoSys approach. In short, it offers a graphical modeling editor that allows us to design our robotic system by selecting and connecting existing components or new ones, specifically created by us, for our robot. Once we complete our graphical model, we can automatically generate the software to be executed in the robot by simply pressing a button. Although this process involves many details that we have not explained, in essence, this kind of practice seems much better (more maintainable, cost-effective, extensible, less prone to errors, etc.) than programming from scratch or carrying out a “copy-and-paste” software reuse (or rather, misuse).

Figure 2. Screenshot of the SmartMDSD Toochain

3. Dealing with non-functional properties

All these features refer to the functional part of Robbie, but what about the non- functional aspects? How well does Robbie do as a salesman? How can we measure its performance?

To answer these questions, first, we have to establish the non-functional properties under which Robbie is going to be evaluated. A service can be good or bad depending on the criteria considered. For that reason, it is not the same to look at the service paying attention to energy consumption or under the prism of customer satisfaction. In our example, we are going to consider “effectiveness”, which, like most non-functional properties, is an abstract concept with a vague interpretation. However, in this case, we will define effectiveness as the degree to which Robbie is successful in producing the following desired results:

  • Acceptance among customers.
  • Robbie manages to attract the interest of the customers.
  • Robbie engages customers in the service, achieving a high level of interaction.
  • High ratio of customers served to customers detected.
  • Robbie covers most of the shopping center.
  • Robbie does not bump into anything or anybody.

Now, we need to quantify how “effective” Robbie is. For that, we use a QoS metric, which expresses, as a real number in the range [0,1], the degree of fulfilment of the non- functional property. In other words, our QoS metric will be a score of how well the system performs according to the previous definition of “effectiveness”. If the metric is 1, the system is optimal in terms of this property, while 0 indicates the opposite. Besides, this value can change as the behavior of the robot evolves during its operation. Thus, the system will show an improvement if, in a period of time, this metric varies from 0.43 to 0.52, or a deterioration if the variation trend is the opposite. Lastly, as the challenge is to evaluate subjective properties, QoS metrics should not be taken as accurate and rigorous measures, but as rough indicators or estimates. With all that, RoQME ambitions to provide a comprehensive set of tools enabling (1) the specification of QoS metrics, defined in terms of the contextual information available; and (2) the continuous evaluation of these metrics at runtime.

4. Modeling QoS metrics at design time and using them at runtime

Following our example, Table1 shows the context variables that could be specified with RoQME. In addition, Table 2 includes some context patterns defined in terms of these variables. The detection of a pattern will be a positive (or negative) observation that reinforces (or reduces) the belief that Robbie is optimal in terms of effectiveness. Basically, the corresponding QoS metric is the quantification of this belief after applying probabilistic reasoning. That is, a value of 0.67 can be understood as the Robbie’s probability of showing maximum effectiveness.

Once the QoS metrics have been specified, we will have to deploy them on the RoQME runtime platform, installed in the robot. This platform will consist of (1) a number of monitors, in charge of updating the context variables from raw contextual data; (2) an event processor, to search for the event patterns described in the RoQME models and, when found, to produce observations; and, finally (3) a probabilistic reasoner that will compute a numeric estimation for each metric. At runtime, this information will be accessible to other components so that it can be used for different purposes, such as adapting the robot behaviour or benchmarking. Finally, Figure 3 shows a simple example of how effectiveness would vary during the operation of Robbie.

Table 1. Context variables for Robbie
Table 2. Context patterns for Robbie according to the definition of effectiveness previously introduced
Figure 3. Example of the effectiveness evaluation at runtime

Why to use RoQME?

  • RoQME promotes a systematic, model-driven, and tool-supported method to develop QoS metrics. This approach contrasts with the ad-hoc implementation of the QoS concerns, e.g., using C++ (or any other programming language), which usually leads to increased complexity and poor reuse.
  • Unlike general-purpose programming languages, the RoQME DSL addresses a very particular problem in a more compact and efficient way thanks to its specialized features. In this vein, among other elements, we plan to include advanced operators to express trends and correlations in context data. Moreover, the RoQME framework will allow designers to validate the syntactic and semantic correctness of their QoS metric specifications in order to significantly reduce runtime errors.
  • In RoQME, a QoS metric specification can be transformed into a probabilistic network, which is a well-known mathematical abstraction that has been successfully applied to many domains, such as medical diagnostics. As a result, we can benefit from existing tools and techniques that are used for the analysis and simulation of probabilistic networks.
  • The RoQME framework has been designed to be loosely coupled with the robotic architecture, so that, although a continuous monitoring of the context may have an impact on the robot performance, the runtime platform of RoQME could easily be activated or deactivated without affecting the basic functionality of the robot.
  • RoQME promotes the application of QoS metrics to develop new robotic features. For instance, robot self-adaptation at runtime based on QoS metrics.
  • RoQME is an Integrated Technical Project (ITP) of the EU H2020 RobMoSys project, which is a major effort to realize a step-change towards a European ecosystem for software development in robotics.
  • All the RoQME software artefacts and documentation will be distributed using open-source and open-access policies.

RoQME has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 732410, in the form of financial support to third parties of the RobMoSys project

Biometric Vox

Biometric Vox is a spanish startup working in new…