This website provides the supplementary materials of the paper "Towards Building AI-CPS with NVIDIA Isaac Sim: An Industrial Benchmark and Case Study for Robotics Manipulation", which presents detailed research workflow and experiment results not shown in the paper due to the page limit. 

The website is organized as follows:

• Home page: The motivation why an industrial-level benchmark is urgently needed, which is followed by an introduction of our research workflow.

• Benchmark: This section introduces our benchmark of robotics manipulation that is constructed based on NVIDIA Omniverse Isaac Sim. 

• Falsification Framework: This section introduces our falsification framework that is compatible with physical simulators and OpenAI Gym environments. 

• RQ1: Advantages and Limitations of Isaac Sim: We conducted a survey with industrial and academic practitioners around the world to gather their comments on the advantages and limitations of Isaac Sim as compared to other physical simulators. The purpose of the survey is to gain insights into the unique features and benefits of Isaac Sim that may make it stand out from other simulators, as well as to identify any potential drawbacks or areas where improvements could be made.

• RQ2: Performance of AI controllers:  We propose various metrics to evaluate and examine the performance of different AI controllers on the robotics manipulation tasks presented in our benchmark. The result reveals the strengths and limitations of different AI controllers in robotics manipulation and allows researchers to identify the most effective methods for achieving specific control objectives. 

• RQ3: Performance of Falsification: We analyze the performance of different optimization methods, i.e., random, Nelder-Mead, and dual annealing, in falsifying robotics tasks with AI software controllers. By using our benchmark, we conduct a falsification test that reveals the robustness and reliability of AI controllers and helps identify potential failures and vulnerabilities in the system. Additionally, this test also demonstrates the extensibility and applicability of our benchmark.

• Summary: We make a summarization of the discussions, challenges, and opportunities of this work.

Abstract

As a representative cyber-physical system (CPS), robotic manipulator has been widely adopted in various academic research and industrial processes, indicating its potential to act as a universal interface between the cyber and the physical worlds. Recent studies in robotics manipulation have started employing artificial intelligence (AI) approaches as controllers to achieve better adaptability and performance. However, the inherent challenge of explaining AI components introduces uncertainty and unreliability to these AI-enabled robotics systems, necessitating a reliable development platform for system design and performance assessment. As a foundational step towards building reliable AI-enabled robotics systems, we propose a public industrial benchmark for robotics manipulation in this paper. It leverages NVIDIA Omniverse Isaac Sim as the simulation platform, encompassing eight representative manipulation tasks and multiple AI software controllers. An extensive evaluation is conducted to analyze the performance of AI controllers in solving robotics manipulation tasks, enabling a thorough understanding of their effectiveness. To further demonstrate the applicability of our benchmark, we develop a falsification framework that is compatible with physical simulators and OpenAI Gym environments. This framework bridges the gap between traditional testing methods and modern physics engine-based simulations. The effectiveness of different optimization methods in falsifying AI-enabled robotics manipulation with physical simulators is examined via a falsification test. Our work not only establishes a foundation for the design and development of AI-enabled robotics systems but also provides practical experience and guidance to practitioners in this field, promoting further research in this critical academic and industrial domain.

Research Workflow

In this paper, we take the first step in investigating the perspectives of practitioners and establishing the foundations to support research and development in AI-enabled robotics manipulation with Isaac Sim. We arrange our investigation and study design with the following steps.

To identify the most critical industrial demands, we first conducted an survey involving academic and industrial practitioners from the global robotics and AI communities. Through a series of questions, we gathered valuable insights into the advantages and drawbacks of Isaac Sim compared to other physical simulators. In addition, we sought participants' input on their requirements for designing and developing AI-enabled robotics applications. This survey not only reveals under which application scenarios Isaac Sim may outperform other simulators but also uncovers the community demands and future directions for developing a high-performing physical simulator. Based on the survey results, we recognize the urgent need for a benchmark to unveil the characteristics of Isaac Sim in the context of developing AI-enabled robotics manipulation.

To bridge this gap, we initiate an early exploratory study that establishes a public industrial benchmark comprising eight representative robotics manipulation tasks, along with multiple software controllers trained with DRL algorithms. This benchmark behaves as a cornerstone for supporting the design and development of AI-enabled robotics manipulation, promoting further research in this critical academic and industrial domain. Furthermore, together with our industrial partners, we design the benchmark with great emphasis on extensibility and applicability, allowing for an end-to-end software development lifecycle towards trustworthy AI-enabled systems. To further evaluate the performance of AI controllers in manipulation tasks simulated with Isaac Sim, we conduct an extensive and in-depth evaluation to assess their capabilities across different tasks. Our experimental results demonstrate that these controllers deliver satisfactory performance in a wide range of tasks and exhibit a commendable level of robustness against action noise.

In addition, we notice a lack of dedicated testing tools specifically designed for physical simulators like Isaac Sim. Testing plays a crucial role in the software development lifecycle as it helps identify scenarios or conditions in which the system fails or exhibits unsafe behaviors. Nevertheless, it remains uncertain whether traditional testing methods are still effective when applied to the scope of physical simulators. To address this gap and demonstrate the practical applicability of our benchmark, we develop the first Python-based falsification framework that is compatible with physical simulators and OpenAI Gym environments. Our findings indicate that the effectiveness of traditional falsification methods varies across different tasks. Therefore, new testing methods that can combine the characteristics of the task, as well as the information from AI components, are anticipated.