Quality and Reliability Assessment of Robotic Software Architectures and Components @IROS2021

Workshop on Quality and Reliability Assessment of Robotic Software Architectures and Components

IROS 2021 Workshop

October 1st, at 14:00 CEST (see current CEST time)

The development of intelligent robotic systems, both at present and in the future, will require greatly strengthened capabilities across sensing, reasoning, information management, and acting. The innovations required in these fields will primarily rely on the development of enhanced software components, software connectivity, and software architectures. This need is further emphasized by the increasing adoption of modular, open-source, and open-data contributions both within the research community and across the industry, for example, the widespread use of open-source modular middleware such as ROS. Although many significant research contributions deal with the analysis of correctness, robustness, or reliability of algorithms and theoretical formulations of robotic capabilities, relatively few deals with design and analysis concerning the quality and reliability of the software that supports the execution of these capabilities. This workshop aims to bridge the gap between practical software engineering, program verification, and applicable robotics by bringing the topics of quality and reliability assessment of software to the fore. The workshop will achieve this through a combination of talks from invited speakers that have relevant contributions and projects, together with contributions from the research community to welcome the latest ideas and contributions relevant to this topic.

Invited Speakers

Cristina Vicente-Chicote

Associate Professor, Univ. Extremadura, Spain
https://sites.google.com/view/cristina-vicente-chicote

The Key Role of System-Level Non-Functional Properties in Robotics Software

Non-Functional properties (NFPs) play a key role in most software-intensive systems. There is a lot of literature on what NFPs are but, sadly, there is no consensus on how to deal with them. In the context of robotics, NFPs such as safety, dependability or resource consumption must not only to be taken into account, but must be guaranteed to the greatest possible extent for robots to be viable, in particular, when they are intended to interact with humans or to perform critical missions. In the context of the RobMoSys H2020 Project, NFPs are treated as first-class citizens and their modeling, estimation and management is considered a key facilitator to achieve better robotics software. RobMoSys has funded two Integrated Technical Projects (ITPs) focused on NFPs: RoQME and MIRoN. The former has contributed a model-based framework enabling the modeling and estimation of Quality-of-Service (QoS) metrics defined on system-level NFPs, while the latter aimed at enabling the modeling and implementation of adaptive robot behaviors, based on how the previous QoS metrics evolve over time and trying to balance the (usually contradicting) requirements imposed on the different NFPs. The latest results obtained in both RoQME and MIRoN will be presented in the talk.

Cristina Vicente-Chicote is an Associate Professor in the Department of Computer and Telematics Systems Engineering at UEX (Spain), where she belongs to the Quercus Software Engineering Group (QSEG). She received a MSc. in Computer Science at the University of Murcia (Spain) in 1998, and a PhD in Computer Science at the Technical University of Cartagena (Spain) in 2005. Her research interests include model-driven engineering, component-based software development and context-aware and self-adaptive systems. In the last decade, she has focused on the application of these and other software engineering approaches to the development of robotics software. She has published more than 80 academic papers in peer-reviewed journal and conferences and, and has served as a reviewer for several journals, conferences and workshops. She has coordinated the RoQME Integrated Technical Project (2018-2019), and currently coordinates the MIRoN Integrated Technical Project (2019-2020), both funded by the RobMoSys H2020 Project.

Matt Webster

Senior Lecturer, Liverpool John Moores Univ., UK
https://www.ljmu.ac.uk/about-us/staff-profiles/faculty-of-engineering-and-technology/school-of-computer-science-and-mathematics/matt-webster

Verifiable Software Architectures for Autonomous Robotic Systems

Autonomous robotic systems are increasingly deployed in a wide range of safety- and mission-critical applications and environments, including aviation, manufacturing, healthcare and the automotive industry. As a result, it is essential to be able to verify these systems to ensure that they are safe and reliable for deployment. Verification is often applied to robotic systems after they have been developed. However, it is often preferable to design the robotic system with verification in mind to allow us to maximize the utility of the various verification techniques available. In this talk we examine some of these verifiable autonomous robotic system architectures, show how they are being used at the University of Liverpool for space and offshore applications on the FAIR-SPACE and ORCA hubs, and present recommendations for ensuring that these architectures can be verified to be safe and reliable.

Matt Webster is a Senior Lecturer in the School of Computer Science and Mathematics at Liverpool John Moores University, UK. His research aims to make computer systems safer and more reliable through the development and application verification and validation techniques such as formal methods and simulation. His research interests include verification of AI in space robotics, certification of autonomous unmanned aircraft, the Internet of Things, human-robot interaction, model-checking agent programming languages, computer security, and artificial life. He is currently working with the Future AI and Robotics for Space (FAIR-SPACE) and Offshore Robotics for Certification of Assets (ORCA) RAI hubs, both of which are funded by EPSRC.

John-Paul Ore

Associate Professor, NCSU, USA
https://jpwco.com/

Incorporating Real-World Semantics into Program Analysis of Robot Systems

Robotic software is plagued both by faults that menace all software (null-pointers, index-out-of-bounds) and also faults specific to its physical interaction with the real world, such as dimensional inconsistencies. These software hazards occur when developers incorrectly manipulate real-world quantities with physical units, such as confusing force with torque or measuring an angle in degrees instead of radians---something we have shown frequently happens in practice. We also found that existing solutions to these problems are time-consuming and error-prone. To address the state of the art, we designed a program analysis technique and its corresponding tool 'Phys' to automatically detect dimensional inconsistencies in robotic software with minimal developer burden. Phys uses probabilistic reasoning and dataflow analysis to infer what variables mean in the real world. Phys works on systems that use the popular `Robot Operating System' (ROS). I will present an evaluation showing that Phys has an 85% True Positive rate. I will present evidence that dimensional inconsistencies lurk in at least 6% ( 211 / 3,484 ) of open-source robotic software repositories. I will further show the results of an empirical study showing that developers correctly identify the physical units of variables only 51% of the time, motivating our future work on automatically suggesting physical unit types. Finally, I will present a vision of future robotic software research enabled by our techniques that aims to help developers build robots with more reliable robotic software.

John-Paul Ore is an assistant professor at the North Carolina State University. His research is in software engineering and field robotics. His Ph.D. work focused on how to automatically detect dimensional inconsistencies in robotic software without time-consuming developer annotations. Specifically, he builds techniques and tools that infer physical unit types (like ‘meters-per-second’) using probabilistic reasoning to combine facts from dataflow with evidence from uncertain sources like variable names. He also performs empirical studies of developers to assess their ability to make decisions about robotic software. Overall, his goal is to help robotic system developers create better and safer systems. John-Paul received an Othmer fellowship, a US Patent for Aerial Water Sampling (#US9606028B2), ‘Best Masters Thesis’ Award (2014), ‘Best Tool Demonstration’ (ISSTA’17), and is on the program committee for Robotic Software Engineering Workshop (RoSE, part of ICSE’19). He has a B.A. in Philosophy from the University of Chicago.

Ana Cavalcanti

Professor, Univ. of York, UK
https://www-users.cs.york.ac.uk/~alcc/

Diagrammatic physical robot models in RoboSim

Simulation is a favoured technique for analysis of robotic systems. Lack of standardisation and portability between simulators, however, has impact on usability and cost of simulations. We present RoboSim, a diagrammatic tool-independent domain-specific language to model robotic platforms and their controllers. It can be regarded as a profile of UML/SysML enriched with time primitives, differential equations, and a formal process algebraic semantics. In RoboSim, a robotic platform is specified by a block diagram, which can be linked to a data model to characterise how events, variables, and operations of the software controller map to inputs and outputs of sensors and actuators. The behaviours of inputs, outputs, and joints are specified by systems of differential algebraic equations. Simulations and mathematical models for proof can be generated automatically from RoboSim models.

Ana Cavalcanti is a Professor at the University of York, UK, and a Royal Academy of Engineering Chair in Emerging Technologies. Her current work is on Software Engineering for Robotics, with a leading role in the RoboStar group (https://robostar.cs.york.ac.uk/). From 2012 to 2017, she held a Royal Society - Wolfson Research Merit Award. In 2003, she was awarded a Royal Society Industry Fellowship to work with QinetiQ on formal methods. She has published almost 200 papers, and chaired the Programme Committee of various well-established international conferences. She has a long-term interest in refinement, safety-critical systems, object-orientation , concurrency, and real-time applications. She has played a major role in the design and formalisation of a state-rich process algebra, namely, Circus, and its development techniques using the Hoare and He's Unifying Theories of Programming. She is currently Chair of the Formal Methods Europe association.

Andrzej Wąsowski

Professor , IT University, Copenhagen, Denmark
http://www.itu.dk/people/wasowski/

ROS Bugs: what do we learn about robotic systems engineering from studying and finding issues in rosdistro repositories?

What kinds of bugs do programmers encounter when developing ROS applications? Are those bugs unique to ROS? What can we do to avoid them? To answer these questions, we have been conducting a joint industry-academia effort over a two-year period to systematically identify and forensically analyse 221 bugs across 9 ROS packages from diverse projects. We created ROBUST, a publicly available set of ROS bugs developed in collaboration of several companies and universities engaged with robotics engineering. The dataset consists of carefully described and classified historical issues, along with a scripting machinery that allows to reconstruct a historical setup in which each bug was reported, in order facilitate reproduction and analysis research.

I will analyze the data set of historical issues collected from ROS repositories and ask what kind of bugs are faced by ROS developers. We will classify the bugs and ask ourselves why do they appear, which can be found with automatic analysis and testing tools, and which are inherently difficult due to interactions with hardware and the environment. Finally, we will attempt to derive lessons and best practices for robot developers helping to avoid the identified issues.

Andrzej Wasowski works with design and use of technologies that improve quality of software, including issues such as correctness and maintainability. He has worked extensively with software product line methods---ways to develop software for similar products at lower cost but with higher quality. He has collaborated with open source projects (Linux kernel and ROS among others) and with industry (for example with Danfoss). Currently, he is investigating quality assurance methods for robotics platforms, in the H2020 project ROSIN.Andrzej Wasowski is a professor of Software Engineering at IT University in Copenhagen (ITU). He holds an MSc degree from Warsaw University of Technology and a PhD degree from ITU. He has previously held visiting positions at Aalborg University (Denmark), INRIA Rennes (France) and University of Waterloo (Canada).

Michael Rathmair

Head of ROBOTICS evaluation Lab,Joanneum Research Robotics, Austria
https://www.joanneum.at/en/robotics/the-institute/team/detail/rathmair

Benjamin Breiling

Robotic Engineer, Institute of Robotics and Mechatronics, Joanneum Research Robotics, Austria
https://www.joanneum.at/en/robotics/the-institute/team/detail/breiling

Software Quality as a Key Requirement for Safe and Secure Robot Applications

Safety-the protection of humans from physical harm- has always been a significant key element in robotics. Security has become an urgent requirement for robots in recent years. In particular, the protection of subsystems implementing software functions for safe operation is crucial. As robotic technology progresses and is more and more pervaded by general-purpose computing technologies, a third "S" besides safety and security arises: Software(quality). More and more functions of robots are implemented using modern software techniques bringing along also the drawbacks of complex software systems. On the other hand, this also allows for the usage of software verification techniques to strengthen safety and security functions along with the robots business functionality.

In this talk, we survey the landscape in robot safety and security along with the most important standards and guidelines. Approaches that have the goal to bridge the gap between practical software engineering, program verification, and safety evaluation applicable in the robotics will be brought to the fore. We then highlight where quality of software is crucial to ensure safe and secure robots and discuss research activities that point out where modern software engineering techniques could be used in future to improve robot safety and security.

Benjamin Breiling is a Robotic Engineer at the Institute of Robotics and Mechatronics at JOANNEUM RESEARCH. He received his Masters degree in Applied Informatics from Alpen-Adria Universität Klagenfurt. His research interests include robotic security and robotic software engineering.Michael Rathmair is Head of the competence center ROBOTICS evaluation Lab at JOANNEUM RESEARCH ROBOTICS. He graduated with a PhD degree from the Technical University in Vienna and focusses his research activities on verification, validation and test of safety -critical systems.

Workshop Program

The workshop invited speakers have pre-recorded a long version of their talk, that you can find either directly in the IROS2021 gCon system, or in the QRARSAC2021 Youtube Playlist or directly following the links in the detailed program below. We kindly suggest you to have a look at the videos before the workshop live session, planned on Oct. 1st. During this live session, each speaker will have give a short talk to sumup the main ideas of his/her talk in 5 minutes, followed by questions and answers. If you attend the workshop, you will be able to ask the question live to the speakers! We also gather questions before hand: whether you can attend the workshop or not, you can send your questions before Oct. 1st to charles.lesire@onera.fr; for those who cannot attend, we will post on this website the minutes of the workshop with the answers to your question!

Finally, in addition to the short talks and Q&A, the workshop will have some panel discussion on the hot topics of quality and reliability assessment of robotic software architectures and components.