The survey consists of three question types: short-answer, multiple-choice, and Likert scale questions. The Likert scale questions ask respondents to indicate their level of agreement with statements such as Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree, and I don't know.

The survey consists of five parts:

• Demographics: We aim to gather information about the practitioners' demographics and experience in robotics development.

• General robotics simulation using Isaac Sim: We target to investigate the pros and cons of Isaac Sim in terms of general robotics simulation and DRL-related modules.

• Development challenge of DRL robotics tasks in Isaac Sim: In this part, we ask about the practitioners' experience in developing DRL robotics tasks in Isaac Sim.

• Comparison with other robotics simulators: This part asks the practitioners to compare Isaac Sim with other popular robotics simulators, such as Gazebo and PyBullet.

• Testing of robotics tasks in Isaac Sim: This part studies the practitioners' habits in performing testing in robotics tasks and the status of testing support in Isaac Sim.

We also provide an open-ended question at the end of the survey, inviting practitioners to share their opinions and comments about Isaac Sim in the context of robotics and DRL development.

We recruited practitioners through two methods: email contact and forum recruitment.

• To obtain email contacts, we searched for research papers published in top conferences such as ICSE, ESEC/FSE, ASE, ICRA, IROS, AAAI, IJCAI, and NeurIPS from 2019 to 2023 that involve developing robotics tasks in Isaac Sim. We collected the email addresses of the authors of these papers, resulting in a list of 75 researchers whom we contacted individually via email.

• For forum recruitment, we posted topics about the survey in various online communities, including the Isaac Sim forum, the NVIDIA official Discord community, and the GitHub discussion page of relevant libraries, e.g., SKRL and Isaac Gym.

In total, we received 17 responses from nine different countries. Details about the information of the respondents are shown in the following figure.

• As depicted in Fig. (a), the majority of practitioners agree that Isaac Sim supports diverse types of sensors (74\%), and the USD (Universal Scene Design) file format makes it easier to develop custom scenes and robots (80\%). However, some respondents point out that certain simulation features, such as simulating soft or deformable objects, are not well developed.

• Regarding challenges in developing DRL tasks in Isaac Sim, Fig. (b) shows that most practitioners think that the API documentation lacks proper explanation (67\%) and the community support is not active enough (66\%), which highlights the need for improved documentation and official support for users of Isaac Sim. Moreover, more than half (53\%) of the practitioners consider that it is not easy to develop and test new DRL algorithms. 

• In terms of comparison with other platforms (Fig. (c)), all practitioners believe that the training speed in Isaac Sim is faster than other simulation platforms due to its good compatibility with NVIDIA GPUs. Additionally, most of them (63\%) find it easier to train multiple DRL agents in Isaac Sim. However, more than half of the practitioners (57\%) think that the community and documentation support is not as good as in other platforms. Moreover, as the simulation prototype is ultimately expected to be deployed in the real world, we are also interested in the accuracy of Isaac Sim's physical simulation. While half of the practitioners think that the simulation is more accurate, the other half are neutral or disagree with this point. 

• In addition, we ask practitioners about their opinions on testing robotics tasks in Isaac Sim (Fig. (d)). Most of them (93\%) believe that it is a good idea to provide additional testing support, with one practitioner stating, "An advanced plug-and-play testing module will be an asset". This encourages our development of a falsification framework for Isaac Sim and OpenAI Gym environments.

Conclusively, numerous participants have pointed out the lack of a clear pipeline for DRL and robotics development in the current version of Isaac Sim. Notably, many responses indicate that users face challenges in initiating their development process due to insufficient instructions and documentation. Moreover, the absence of a baseline for comparison and calibration poses difficulties for developers who lack guidance and standards to assess their progress accurately. Therefore, we consider an AI-enabled robotics manipulation benchmark can greatly benefit practitioners in the following aspects: 1) an easy-to-use development playground and pipeline; 2) a standard baseline for observation and comparison;  and 3) an extensible and applicable environment for quick demonstration and prototyping.