Workshop on Quality and Reliability Assessment of Robotic Software Architectures and Components

June, 4th, 2020 @ICRA 2020, Paris, France

The development of intelligent robotic systems, both at present and in the future, will require greatly strengthened capabilities across sensing, reasoning, information management, and acting. The innovations required in these fields will primarily rely on the development of enhanced software components, software connectivity, and software architectures. This need is further emphasized by the increasing adoption of modular, open-source, and open-data contributions both within the research community and across the industry, for example, the widespread use of open-source modular middleware such as ROS. Although many significant research contributions deal with the analysis of correctness, robustness, or reliability of algorithms and theoretical formulations of robotic capabilities, relatively few deals with design and analysis concerning the quality and reliability of the software that supports the execution of these capabilities. This workshop aims to bridge the gap between practical software engineering, program verification, and applicable robotics by bringing the topics of quality and reliability assessment of software to the fore. The workshop will achieve this through a combination of talks from invited speakers that have relevant contributions and projects, together with contributions from the research community to welcome the latest ideas and contributions relevant to this topic.

Call For Papers

We solicit contributions from scientists/practitioners working on topics related to the workshop, in the form of short papers (max 3 pages). We also strongly encourage authors to submit a video clip to complement the submission.

Short papers must follow the IEEE RAS format (

Video length is limited to 3min. The content/format of the video is free. You can either send the video file with your paper or provide a Youtube or similar link.

All selected papers will be presented during the poster session.

Videos of selected papers will be screened in specific sessions during the workshop. An award distinguishing the best contribution will also be decided during the workshop, to be selected only among those that also submitted a video.

Short papers and videos should be submitted to:

Important Dates

  • Submission Deadline: 23 March 2020
  • Acceptance Notifications: 2 April 2020
  • Workshop Date: 4 June 2020

Invited Speakers

Cristina Vicente-Chicote

Associate Professor at Univ. Extremadura, Spain

The Key Role of System-Level Non-Functional Properties in Robotics Software

Non-Functional properties (NFPs) play a key role in most software-intensive systems. There is a lot of literature on what NFPs are but, sadly, there is no consensus on how to deal with them. In the context of robotics, NFPs such as safety, dependability or resource consumption must not only to be taken into account, but must be guaranteed to the greatest possible extent for robots to be viable, in particular, when they are intended to interact with humans or to perform critical missions. In the context of the RobMoSys H2020 Project, NFPs are treated as first-class citizens and their modeling, estimation and management is considered a key facilitator to achieve better robotics software. RobMoSys has funded two Integrated Technical Projects (ITPs) focused on NFPs: RoQME and MIRoN. The former has contributed a model-based framework enabling the modeling and estimation of Quality-of-Service (QoS) metrics defined on system-level NFPs, while the latter (still ongoing) aims at enabling the modeling and implementation of adaptive robot behaviors, based on how the previous QoS metrics evolve over time and trying to balance the (usually contradicting) requirements imposed on the different NFPs. The latest achievements both in RoQME and in MIRoN will be presented in the talk.

Cristina Vicente-Chicote is an Associate Professor in the Department of Computer and Telematics Systems Engineering at UEX (Spain), where she belongs to the Quercus Software Engineering Group (QSEG). She received a MSc. in Computer Science at the University of Murcia (Spain) in 1998, and a PhD in Computer Science at the Technical University of Cartagena (Spain) in 2005. Her research interests include model-driven engineering, component-based software development and context-aware and self-adaptive systems. In the last decade, she has focused on the application of these and other software engineering approaches to the development of robotics software. She has published more than 80 academic papers in peer-reviewed journal and conferences and, and has served as a reviewer for several journals, conferences and workshops. She has coordinated the RoQME Integrated Technical Project (2018-2019), and currently coordinates the MIRoN Integrated Technical Project (2019-2020), both funded by the RobMoSys H2020 Project.

Matt Webster

Postdoctoral Researcher, Univ. Liverpool, UK

Verifiable Software Architectures for Autonomous Robotic Systems

Autonomous robotic systems are increasingly deployed in a wide range of safety- and mission-critical applications and environments, including aviation, manufacturing, healthcare and the automotive industry. As a result, it is essential to be able to verify these systems to ensure that they are safe and reliable for deployment. Verification is often applied to robotic systems after they have been developed. However, it is often preferable to design the robotic system with verification in mind to allow us to maximise the utility of the various verification techniques available. In this talk we examine some of these verifiable autonomous robotic system architectures, show how they are being used at the University of Liverpool for space and offshore applications on the FAIR-SPACE and ORCA hubs, and present recommendations for ensuring that these architectures can be verified to be safe and reliable.

Matt Webster is a senior postdoctoral researcher at the University of Liverpool. With over 15 years of academic and industrial research experience, his research aims to make computer systems safer and more reliable through the development and application of techniques from formal methods. His research interests include verification of AI in space robotics, certification of autonomous unmanned aircraft, the Internet of Things, human-robot interaction, model-checking agent programming languages, computer security and artificial life. He is currently working on the EPSRC-funded ORCA and FAIR-SPACE Robotics and AI hubs.

John-Paul Ore

Associate Professor, NCSU, USA

Incorporating Real-World Semantics into Program Analysis of Robot Systems

Robotic software is plagued both by faults that menace all software (null-pointers, index-out-of-bounds) and also faults specific to its physical interaction with the real world, such as dimensional inconsistencies. These software hazards occur when developers incorrectly manipulate real-world quantities with physical units, such as confusing force with torque or measuring an angle in degrees instead of radians---something we have shown frequently happens in practice. We also found that existing solutions to these problems are time-consuming and error-prone. To address the state of the art, we designed a program analysis technique and its corresponding tool 'Phys' to automatically detect dimensional inconsistencies in robotic software with minimal developer burden. Phys uses probabilistic reasoning and dataflow analysis to infer what variables mean in the real world. Phys works on systems that use the popular `Robot Operating System' (ROS). I will present an evaluation showing that Phys has an 85% True Positive rate. I will present evidence that dimensional inconsistencies lurk in at least 6% ( 211 / 3,484 ) of open-source robotic software repositories. I will further show the results of an empirical study showing that developers correctly identify the physical units of variables only 51% of the time, motivating our future work on automatically suggesting physical unit types. Finally, I will present a vision of future robotic software research enabled by our techniques that aims to help developers build robots with more reliable robotic software.

John-Paul Ore is an assistant professor at the North Carolina State University. His research is in software engineering and field robotics. His Ph.D. work focused on how to automatically detect dimensional inconsistencies in robotic software without time-consuming developer annotations. Specifically, he builds techniques and tools that infer physical unit types (like ‘meters-per-second’) using probabilistic reasoning to combine facts from dataflow with evidence from uncertain sources like variable names. He also performs empirical studies of developers to assess their ability to make decisions about robotic software. Overall, his goal is to help robotic system developers create better and safer systems. John-Paul received an Othmer fellowship, a US Patent for Aerial Water Sampling (#US9606028B2), ‘Best Masters Thesis’ Award (2014), ‘Best Tool Demonstration’ (ISSTA’17), and is on the program committee for Robotic Software Engineering Workshop (RoSE, part of ICSE’19). He has a B.A. in Philosophy from the University of Chicago.

Ana Cavalcanti

Professor at Univ. of York, UK

Diagrammatic physical robot models in RoboSim

Simulation is a favoured technique for analysis of robotic systems. Lack of standardisation and portability between simulators, however, has impact on usability and cost of simulations. We present RoboSim, a diagrammatic tool-independent domain-specific language to model robotic platforms and their controllers. It can be regarded as a profile of UML/SysML enriched with time primitives, differential equations, and a formal process algebraic semantics. In RoboSim, a robotic platform is specified by a block diagram, which can be linked to a data model to characterise how events, variables, and operations of the software controller map to inputs and outputs of sensors and actuators. The behaviours of inputs, outputs, and joints are specified by systems of differential algebraic equations. Simulations and mathematical models for proof can be generated automatically from RoboSim models.

Andrzej Wąsowski

Professor at IT University, Copenhagen, Denmark

ROS Bugs: what do we learn about robotic systems engineering from studying and finding issues in rosdistro repositories?

I will analyze the data set of historical issues collected from ROS repositories and ask what kind of bugs are faced by ROS developers. We will classify the bugs and ask ourselves why do they appear, which can be found with automatic analysis and testing tools, and which are inherently difficult due to interactions with hardware and the environment. Finally, we will attempt to derive lessons and best practices for robot developers helping to avoid the identified issues.

Andrzej Wasowski works with design and use of technologies that improve quality of software, including issues such as correctness and maintainability. He has worked extensively with software product line methods---ways to develop software for similar products at lower cost but with higher quality. He has collaborated with open source projects (Linux kernel and ROS among others) and with industry (for example with Danfoss). Currently, he is investigating quality assurance methods for robotics platforms, in the H2020 project ROSIN.Andrzej Wasowski is a professor of Software Engineering at IT University in Copenhagen (ITU). He holds an MSc degree from Warsaw University of Technology and a PhD degree from ITU. He has previously held visiting positions at Aalborg University (Denmark), INRIA Rennes (France) and University of Waterloo (Canada).

Bernhard Dieber

Head of Institute for Robotics and MechatronicsJoanneum Research, Austria

The third S for good robots

Safety and Security in robots have long been known to go together hand in hand in order to make robots safe around humans. Safety ensures that a robot is physically no danger to a human. Security is required to safeguard the safety functions that are realized in software. In modern, intelligent robots however, where software a dominating part, the quality and reliability of software is a key issue. If a robot does not behave as expected or is not as reliable in its task performance as required, safety and security only provide a “last-line-of-defence” but do not enhance its functional adequacy.

We view Software (quality) as a third dimension to build good robots. It is what goes beyond protecting humans and the robot and gives the modern robot its usefulness. However, developing business software for robots is very complex and error-prone with the methods we use today. Improved methods, processes and tools are required to enable developers to effectively and efficiently program robots. In this talk, I go into detail how software quality affects security of robots using examples from the ROS environment. I show resulting vulnerabilities and hint towards approaches to improve software quality in robotics.


Alcino CUNHA

INESC TEC, University of Minho, Portugal

Charles LESIRE

ONERA/DTIS, University of Toulouse, France

Michael FISHER

CAST, University of Liverpool, UK

Program Committee

  • Davide BRUGALI, University of Bergamo, Italy
  • David DOOSE, ONERA/DTIS, Toulouse, France
  • Christophe GRAND, ONERA/DTIS, Toulouse, France
  • Jérémie GUIOCHET, LAAS-CNRS, Toulouse, France
  • Nico HOCHGESCHWENDER, Bonn-Rhein-Sieg University, Germany
  • Nuno MACEDO, INESC TEC & University of Minho, Portugal


This workshop is supported by: