Empirical Study

In this section, we introduce how we collect subject CPS from diverse industrial domains, and explain the selection criteria, and sources.

Selection Criteria

In order to collect the most representative CPS into our benchmark, we list following requirements to select the valid candidates out from the vast models.

  1. Industry oriented

As we stated in the abstract, the purpose of our paper is to establish an industry-level benchmark to improve the safety and reliability of AI-enabled CPS; so we only take into consideration the systems that reflect industrial complexity, have actual applications, and are representative for specific domains. Browsing through a lot of papers in AI control areas, we notice that many AI controllers are implemented on game scenarios such as Cart-Pole, Inverted Pendulum, and Mountain Car.

Indeed, these environments are classic and effective to be platforms for designing, testing, and validating the algorithms or effectiveness of an AI control system. However, these examples are too simple to reflect the complexity, dynamics, and demands for industry-level applications. That is, systems in industries have much more complex system dynamics, safety and functional requirements, and disturbance from environments. We aim to select the systems which can conclude all the characteristics above, thus our benchmark is capable to meet the industrial demands in reality.


  1. Open-sourced

There are plenty of exciting CPS projects designed and established by companies and researchers, but not all of them are open-sourced. Many industrial enterprises develop their core products based on CPS-related technologies, so they keep their systems as confidential secrets which makes it challenging to find a publicly available CPS collection. Moreover, in order to analyze the system dynamics and controller functionalities, then replace the traditional controllers with AI controllers, we need to find CPS projects in which the complete systems are available.


  1. Documentation available

While browsing the available candidate CPS, we notice that many works are missing comprehensive documentation or description files, in which we are hard to determine the tasks of the systems and the functional requirements. Tracing back the design purposes and performance requirements from the bare system models without explanations and guidance from a handbook may cause misunderstandings and deficiencies. Furthermore, the design of AI controllers and falsifications require detailed system requirements. Without their official documentations, our benchmark is not convective to be used as a designing and testing platform. Thus, while we select the subject CPS, we carefully check if the documentation is available with detailed environment descriptions, safety requirements, and system configurations.


  1. Simulink based

CPS projects can be implemented in various programming languages and software applications, and the functionalities of these tools can be much more diverse. If we use multiple tools to build our benchmarks on various platforms, it will take incredibly large time and labour efforts to unify the construction process and evaluation metrics. In addition, different modelling tools may have different procedures to design DRL controllers and some applications do not support DRL at all.

In order to make the systems in our benchmark under a single standard and easy to be used by followers, we decide to collect the CPS which are constructed with Simulink on MATLAB. Simulink is a MATLAB-based modelling platform developed by MathWorks and is wildly adopted in industries. Many practical applications are initially modelled and tested on MATLAB to verify the functionalities and performance. Also, DRL is supported by MATLAB in Reinforcement Learning Toolbox, which makes us easier to set up the AI controllers and run the benchmarks.

Source

We mainly focus on two sources that potentially release valid subject CPS: MATLAB control-related toolboxes and CPS related literature.

MATLAB Toolbox

Many toolboxes released by MathWorks contain valid CPS examples, and we find some specific ones that are well fit to our demands, such as Model Predictive Control Toolbox, Automated Driving Toolbox, Aerospace Toolbox, and Control System Toolbox. We read through the documentation of these examples and run the demonstrations to obtain an overview of the system behaviours. We notice that some of the systems are developed by scripts only without a Simulink to hold the system structures and some are too complicated with redundant components which deviate from the purpose of our paper. We collaborate the requirements and considerations from diverse aspects to select the systems that are the best fit for our work.

CPS Related literature

The literature we focused on for collecting CPS is an annual workshop, namely, ARCH. ARCH aims to mitigate this problem by bringing together CPS benchmarks and holding competitions for different research topics. The most relevant competitions to this paper are Artificial Intelligence and Neural Network Control Systems and Falsification. However, the benchmark in the second competition only includes traditional CPS rather than AI-enabled CPS. While the benchmark in the first competition includes AI-enabled CPS, their benchmark includes less and simpler CPS such as Cart-Pole, which are not from industrial application domains. Furthermore, their AI controllers are simple feed-forward neural networks (FNN), rather than DRL models.

We list some of the subject systems below to illustrate how we pick them out from the vast examples.

  1. Lane Change Assist Using Nonlinear Model Predictive Control [1]

  2. Highway Lane Change [2]

  3. Lane Keeping Assist System Using Model Predictive Control [3]

  4. Lane Following Using Nonlinear Model Predictive Control [4]

  5. Highway Lane Following with Intelligent Vehicles [5]

  6. Obstacle Avoidance Using Adaptive Model Predictive Control [6]

  7. Traffic Light Negotiation [7]

  8. Automate Testing for Highway Lane Following [8]

  9. Highway Lane Following with RoadRunner Scene [9]

  10. Adaptive Cruise Control with Sensor Fusion [10]

  11. Highway Trajectory Planning Using Frenet Reference Path [11]

  12. Truck and Trailer Automatic Parking Using Multistage Nonlinear MPC [12]

  13. Generate Code for Lane Marker Detector [13]

  14. Generate Code for Vision Vehicle Detector [14]

  15. Extended Object Tracking of Highway Vehicles with Radar and Camera [15]

  16. Grid-based Tracking in Urban Environments Using Multiple Lidars [16]

  17. Track Vehicles Using Lidar: From Point Cloud to Track List [17]

  18. Track-to-Track Fusion for Automotive Safety Applications in Simulink [18]

  19. Simulate Radar Ghosts due to Multipath Return [19]

  20. Design Lane Marker Detector Using Unreal Engine Simulation Environment [20]

  21. Parking Valet Using Nonlinear Model Predictive Control [21]

  22. Parallel Parking Using Nonlinear Model Predictive Control [22]

  23. Automated Parking Valet [23]

  24. MIMO Control of Diesel Engine [24]

  25. Land a Rocket Using Multistage Nonlinear MPC [25]

  26. Approximate High-Fidelity UAV model with UAV Guidance Model block [26]

  27. UAV Package Delivery [27]

  28. Control of Quadrotor Using Nonlinear Model Predictive Control [28]

  29. Fixed-Structure Autopilot for a Passenger Jet [29]

  30. Trajectory Optimization and Control of Flying Robot Using Nonlinear MPC [30]

  31. Multiloop Control of a Helicopter [31]

  32. Swing-up Control of a Pendulum Using Nonlinear Model Predictive Control [32]

  33. Multi-Loop PI Control of a Robotic Arm [33]

  34. Active Vibration Control in Three-Story Building [34]

  35. Vibration Control in Flexible Beam [35]

  36. Digital Control of Power Stage Voltage [36]

  37. Nonlinear and Gain-Scheduled MPC Control of an Ethylene Oxidation Plant [37]

  38. Optimization and Control of a Fed-Batch Reactor Using Nonlinear MPC [38]

  39. Economic MPC Control of Ethylene Oxide Production [39]

  40. Water Tank [40]

  41. Nonlinear Model Predictive Control of an Exothermic Chemical Reactor [41]

  42. PMSG based Wind Power Generation System [42]

  43. Automatic Transmission [43]

  44. Steam condenser [44]

  45. Abstract Fuel Control [45]

  46. Neural-network Controller [46]

  47. Chasing cars [47]

  48. Aircraft Ground Collision Avoidance System [48]

  49. Steam condenser [49]

  50. DC-to-DC Power Converter [50]

  51. Magnet Control NN [51]

...

From the large number of systems we collected above, we find many examples are not perfectly fit our goals. Since we focus on the control systems in this paper for evaluating the performance of AI-enabled CPS compared with the traditional CPS. So, we take more consideration on how the controllers have been designed and operated in these systems, whereas in some systems, controllers are not the core components. For example, in some automated driving examples and UAV systems [10, 13, 14, 27], the main purposes are to demonstrate the path generator or sensor fusion design. Applying AI controllers to these systems, we have to face serious compatibility problems which take much time to fix or redesign the entire system.

We also remove the systems which contain complicated and redundant components (e.g., some systems have UnrealEngine embedded in, which are hard to be distributed on cloud computing services) to better keep the way straight to our major goal.

Finally, the 9 subject CPS are:

  1. Adaptive Cruise Control (ACC)

  2. Lane Keeping Assistant (LKA)

  3. Automatic Parking Valet (APV)

  4. Exothermic Chemical Reactor (CSTR)

  5. Land a Rocket (LR)

  6. Abstract Fuel Control (AFC)

  7. Wind Turbine (WT)

  8. Steam Condenser (SC)

  9. Water Tank (WTK)

All these subject CPS are selected as representative and control system-oriented from industrial domains, and they can better highlight the capability and performance of controllers. Each system ships with a built-in traditional controller. We experiment with different types of learning algorithms, various agent configurations, and diverse reward functions to explore the capability of AI controllers. AI-CPS construction We believe our benchmark with these 9 systems can cover most of CPS applications in industrials and bring in-depth understandings and insights about the functionality and reliability of AI-enabled CPS.

Reference

  1. https://www.mathworks.com/help/mpc/ug/lane-change-assist-using-nonlinear-model-predictive-control.html

  2. https://www.mathworks.com/help/mpc/ug/highway-lane-change.html

  3. https://www.mathworks.com/help/mpc/ug/lane-keeping-assist-system-using-model-predictive-control.html

  4. https://www.mathworks.com/help/mpc/ug/lane-following-using-nonlinear-model-predictive-control.html

  5. https://www.mathworks.com/help/mpc/ug/highway-lane-following-with-intelligent-vehicles.html

  6. https://www.mathworks.com/help/mpc/ug/obstacle-avoidance-using-adaptive-model-predictive-control.html

  7. https://www.mathworks.com/help/mpc/ug/traffic-light-negotiation.html

  8. https://www.mathworks.com/help/driving/ug/highway-lane-following-with-roadrunner-scene.html

  9. https://www.mathworks.com/help/releases/R2021a/driving/ug/highway-trajectory-planning-using-frenet-reference-path.html

  10. https://www.mathworks.com/help/driving/ug/highway-lane-following-with-roadrunner-scene.html

  11. https://www.mathworks.com/help/releases/R2021a/driving/ug/highway-trajectory-planning-using-frenet-reference-path.html

  12. https://www.mathworks.com/help/mpc/ug/truck-and-trailer-automatic-parking-using-multistage-mpc.html

  13. https://www.mathworks.com/help/driving/ug/generate-code-for-lane-marker-detector.html

  14. https://www.mathworks.com/help/driving/ug/generate-code-for-vision-vehicle-detector.html

  15. https://www.mathworks.com/help/driving/ug/extended-object-tracking.html

  16. https://www.mathworks.com/help/driving/ug/grid-based-tracking-in-urban-environments-using-multiple-lidars.html

  17. https://www.mathworks.com/help/driving/ug/track-vehicles-using-lidar.html

  18. https://www.mathworks.com/help/driving/ug/track-to-track-fusion-for-automotive-safety-applications-in-simulink.html

  19. https://www.mathworks.com/help/driving/ug/radar-ghost-multipath.html

  20. https://www.mathworks.com/help/driving/ug/radar-ghost-multipath.html

  21. https://www.mathworks.com/help/mpc/ug/parking-valet-using-nonlinear-model-predictive-control.html

  22. https://www.mathworks.com/help/mpc/ug/parallel-parking-using-nonlinear-model-predictive-control.html

  23. https://www.mathworks.com/help/mpc/ug/automate-testing-for-highway-lane-following.html

  24. https://www.mathworks.com/help/control/ug/mimo-control-of-diesel-engine.html#:~:text=Modern%20Diesel%20engines%20use%20a,(EGR)%20to%20reduce%20emissions.&text=This%20example%20shows%20how%20to,12%20mg%20per%20injection%2Dcylinder.

  25. https://www.mathworks.com/help/mpc/ug/landing-rocket-with-mpc-example.html

  26. https://www.mathworks.com/help/uav/ug/approximate-high-fidelity-uav-model-with-guidance-model.html

  27. https://www.mathworks.com/help/releases/R2021a/uav/ug/uav-package-delivery.html

  28. https://www.mathworks.com/help/mpc/ug/control-of-quadrotor-using-nonlinear-model-predictive-control.html

  29. https://www.mathworks.com/help/control/ug/fixed-structure-autopilot-for-a-passenger-jet.html

  30. https://www.mathworks.com/help/mpc/ug/trajectory-optimization-and-control-of-flying-robot-using-nonlinear-mpc.html

  31. https://www.mathworks.com/help/control/ug/multi-loop-control-of-a-helicopter.html

  32. https://www.mathworks.com/help/mpc/ug/swing-up-control-of-a-pendulum-using-nonlinear-model-predictive-control.html

  33. https://www.mathworks.com/help/control/ug/multi-loop-pid-control-of-a-robot-arm.html

  34. https://www.mathworks.com/help/control/ug/active-vibration-control-in-three-story-building.html

  35. https://www.mathworks.com/help/control/ug/vibration-control-in-flexible-beam.html

  36. https://www.mathworks.com/help/control/ug/digital-control-of-power-stage-voltage.html

  37. https://www.mathworks.com/help/mpc/ug/nonlinear-and-gain-scheduled-control-of-ethylene-oxidation.html

  38. https://ww2.mathworks.cn/help/mpc/ug/optimization-and-control-of-fed-batch-reactor-using-nonlinear-mpc.html

  39. https://www.mathworks.com/help/mpc/ug/economic-mpc-control-of-nonlinear-chemical-reactor.html

  40. https://www.mathworks.com/help/slcontrol/gs/watertank-simulink-model.html

  41. https://www.mathworks.com/help/mpc/ug/nonlinear-model-predictive-control-of-exothermic-chemical-reactor.html

  42. J. M. Jonkman, S. Buttereld, W. Musial, and G. Scott, \Denition of a 5-MW reference wind turbine for o shore system development," National Renewable Energy Laboratory, Tech. Rep. NREL/TP-500-38060, 2009.

  43. Peter Heidlauf, Alexander Collins, Michael Bolender, and Stanley Bak. Verification challenges in f-16 ground collision avoidance and other automated maneuvers. In Goran Frehse, editor, ARCH18. 5th International Workshop on Applied Verification of Continuous and Hybrid Systems, volume 54 of EPiC Series in Computing, pages 208–217. EasyChair, 2018.

  44. Shakiba Yaghoubi and Georgios Fainekos. Gray-box adversarial testing for control systems with machine learning components. In International Conference on Hybrid Systems: Computation and Control (HSCC), 2019.

  45. Xiaoqing Jin, Jyotirmoy V. Deshmukh, James Kapinski, Koichi Ueda, and Ken Butts. Powertrain control verification benchmark. In Proceedings of the 17th International Conference on Hybrid Systems: Computation and Control, HSCC ’14, pages 253–262, New York, NY, USA, 2014. ACM.

  46. Simone Schuler, Fabiano Daher Adegas, and Adolfo Anta. Hybrid modelling of a wind turbine. In Goran Frehse and Matthias Althoff, editors, ARCH16. 3rd International Workshop on Applied Verification for Continuous and Hybrid Systems, volume 43 of EPiC Series in Computing, pages 18–26. EasyChair, 2017.

  47. https://www.mathworks.com/help/deeplearning/ug/design-narma-l2-neural-controller-in-simulink.html

  48. Peter Heidlauf, Alexander Collins, Michael Bolender, and Stanley Bak. Verification challenges in f-16 ground collision avoidance and other automated maneuvers. In Goran Frehse, editor, ARCH18. 5th International Workshop on Applied Verification of Continuous and Hybrid Systems, volume 54 of EPiC Series in Computing, pages 208–217. EasyChair, 2018.

  49. Shakiba Yaghoubi and Georgios Fainekos. Gray-box adversarial testing for control systems with machine learning components. In International Conference on Hybrid Systems: Computation and Control (HSCC), 2019.

  50. https://www.mathworks.com/help/deeplearning/ug/design-narma-l2-neural-controller-in-simulink.html

  51. Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, pages 1928–1937, 2016.