5.2.1) OEO Console

Overview

The OEO system has been developed by the AERPAW team in order to provide an Operator a way of monitoring and controlling an experiment.  Although the OEO system relies on several processes, the main way of interaction between an Operator and the system is through the OEO Console. 

Each instance of an OEO Console has three windows:

• The events window (at the top), where the system reports significant events in the experiment; for example when a vehicle is armed, if a vehicle is trying to exit the geofence, an error will be displayed in this window. In the figure shown, the events window shows that vehicles on nodes 2 and 4 are disarmed. 

• The monitoring window (in the middle) allows the Operator to continuously monitor the status of several variables. In the figure, the node ID is shown, together with the armed state of the vehicle, their current mode, altitude, and heartbeat.

• The command window (at the bottom) allows the Operator to send commands to a particular node, to an OEO service, or to all nodes.

Starting the OEO Console

To start the OEO Console, first the Operator needs to gain access to the OEO Console VM. Then run these commands:

source /root/.ap-set-experiment-env.sh

cd; ./startOEOConsole.sh

This will start an OEO Console. The OEO system allows multiple simultaneous OEO Consoles to run in a single experiment: this can be done by the same Operator (by logging in repeatedly in the OEO Console container), or by different experimenters sharing an experiment. This allows one or multiple Operators to monitor different sets of variables, and to issue different commands.

To quit the OEO Console, type Ctrl+d.

In the digital twin, only one OEO Console works at one time. Close the other consoles (Ctrl+d) before you start a new one.

If the node table is empty, or if all start_experiment does nothing, you did not run source /root/.ap-set-experiment-env.sh in this shell. Run it, and then start the console again.

Events Window

The event window offers the Operator feedback from the system for displaying info/errors while the experiment is running. In general, the syntax for information displayed on the console will follow the following format: [<sender>]-[<severity_level>] <message>. 

<sender> is the system that sent the message. Most often, this will be the node_id of the node that sent the message. <severity_level> is an indication of the severity of the message. The system currently supports the following severity levels:

• INFO - Informational messages that indicate normal operation or significant events.

• WARNING - Messages that indicate potential issues or conditions that may require attention.

• ERROR - Messages that indicate failures or errors that need attention.

• CRITICAL - Messages that indicate critical failures or system-level issues.

A list of possible events that will be printed:

• Errors signaling an invalid commands/parameters sent by the operator

• MAVLink vehicle status messages

• MAVLink filter aborting the experiment due to invalid vehicle commands

• Collision Avoidance service predicting a collision and aborting experiment due to the forecast. More information at Collision Avoidance Service

• ADS-B flight notifications (only shown while in Testbed). For more information, see ADS-B Microservice.

For increased flexibility AERPAW offers a mechanism for the Experimenters to output messages on the OEO Console Event Window. The  mechanism is intended to allow for an Experimenter to notify the Operator of an important event in the experiment (e.g., that the drone is ready to be armed, or that the radio measurements are complete, or that the experiment failed and the safety pilots need to bring the drones back). Do not use this mechanism as a convenient means of debugging your Experimenter code (and clogging the OEO Event Window in the process). 

Monitoring Window

The monitoring window offers the Operator the ability to monitor a large number of parameters relevant to the current experiment. The relevant commands for adding, renaming, and deleting a monitoring field (issued in the command window) are:

• add <field> where <field> is the new field to be monitored. 

  • For example add vehicle adds a field showing all the relevant values for vehicles (only for nodes having a vehicle).  add computer does the same for the computer parameters.

  • More specific fields can be added by using / as a field separator. For example add vehicle/position/x or add /vehicle/battery/voltage.

  • The agent root data fields available are:

    • vehicle - for vehicle related data, and

    • computer - for node health information

• remap <field> <new_name> is a simply a way to rename the field names to fit the preferences of the operator. For example remap vehicle/battery/voltage volts.

• del <field>  removes the specified field from the monitoring window.

• save_config <config_name> saves the current configuration of the monitoring window (including the remapped fields) in a json file. If config_name is not provided, uses a default configuration loaded at startup.

• load_config <config_name> loads a previously saved configuration for the monitoring window. If config_name is not provided, uses a default configuration loaded at startup.

Vehicle Parameters

Following is a list of vehicle parameters that can be monitored. All parameters can be simultaneously seen if add vehicle is entered in the command window.

• status  -  0 if vehicle is disarmed, and 1 if the vehicle is armed. Once armed, if there is no follow-up instruction, the vehicle automatically disarms in 10 seconds.

• type - quadcopter for UAV, and rover for UGV

• mode - 

  • GUIDED while under the control of the vehicle software (unless in landing phase)

  • LAND when in landing phase

• battery -

  • voltage - main battery voltage (in volts). A 3S LiPo battery (11V - 12.5V) is used for development, a 6S is used for SAMs, and a 12S is used for LAMs.

  • current - current drawn by the vehicle (in amperes).

  • level - state of charge: 0% is empty- vehicle will fall out of the sky in testbed, but will continue to fly in development mode. 100% is brand new battery. The simple way to change batteries in development mode is to restart the cvm processes (restart_cvm in the command window).

• position -

  • latitude - absolute coordinates

  • longitude - absolute coordinates

  • altitude - in [m]

• velocity - the x, y, and z components of the ground velocity in m/s.

• speed - the ground speed in m/s

• gps

  • satellites - number of visible satellites

  • fix - the fix type:

    • 0 - no GPS

    • 1 - there is GPS but no fix is available

    • 2 -  2D position

    • 3 - 3D position

    • 4 - 3D position with DGPS/SBAS corrections

    • 5 - RTK float, 3D position

    • 6 - RTK fixed, 3D position - this is the default in development mode

• home_location - the starting position of the vehicle

  • latitude - absolute coordinates

  • longitude - absolute coordinates

• geofence - the geofence the vehicle is using: ("phase 1", "phase 2")

Computer Parameters

Computer parameters provide the operator with information about the computer running the corresponding node. The following parameters are currently available:

• heartbeat - a 10Hz counter indicating that the node is alive and well

• heartbeat_quality - a metric (0-100%) that measures the quality of the link between the computer and the vehicle. For more information, see Link Quality Microservice 

• cpu_utilization - overall CPU utilization in the computer running that node. In development mode it is likely that all nodes share the same server, which will then have the same (or similar) CPU utilization at all nodes.

• memory_utilization - shows the percentage of memory utilization in that node.

• screens_running - shows a list of the screen processes running in each E-VM

Command Window

The command window allows an Operator to issue commands to the nodes in the experiment. The format of the commands are:

• {node_id | all} agent_command [arguments]

  • Send agent_command to a particular node_id node or to all nodes (see supported agent commands below for details).

  • Example: all arm  - arms all nodes that support arming

  • Example: 1 takeoff 20 - sends takeoff command to node 1, with a target altitude of 20m

  • Example: 2 start_location 35.734942 -78.703000 45.0 - changes start location of node 2 to 35.734942, -78.703000, with a heading of 45.0°.

  • Example: 2 start_location 35.729828 -78.698366- changes start location of node 2 to 35.729828, -78.698366. The heading will remain what it was previously.

Supported Agent Commands

For Vehicles:

• arm - arms vehicle. If no further instructions are issued, the vehicle automatically disarms in 10s. In the digital twin, arm (for example, 2 arm) takes the place of the safety pilot. If the vehicle stays in RTL mode, send 2 set_mode GUIDED before you arm.

• disarm - disarms vehicle

• land - sets vehicle mode to LAND mode (descends until it lands, then disarms)

• rtl - sets vehicle mode to RTL (return to launch). The vehicle climbs to RTL altitude, travels to the home position (where it was armed), then lands, and disarms.

• takeoff <altitude> - makes vehicle take off to <altitude>. The vehicle needs to be armed and in GUIDED mode to takeoff

• set_mode <mode> - sets vehicle mode to <mode> where <mode> is one of (GUIDED, MANUAL, ALT_HOLD, RTL, STABILIZE)

• start_location { reset | <latitude> <longitude> [<azimuth>] } - sets vehicle home location to <latitude>, <longitude>, at heading <azimuth> (in degrees). <azimuth> is optional; by default, it will be set to 0.0.  To reset the home location to default, run the command: start_location reset. The restart_cvm command must be run afterward in order for these changes to take effect. The new home location will persist through multiple restarts of the C-VM.

• set_geofence <geofence_phase> - sets vehicle geofence to <geofence_phase> where <geofence_phase> is one of (1, 2). 1 corresponds to Phase 1. 2 corresponds to Phase 2. See Geofences for the different geofences for each phase. The restart_cvm command must be run afterward in order for these changes to take effect.

• vehicle_check - runs the parameter checker to report on common vehicle parameters and ensure they are within acceptable ranges. More details here.

• Vehicle Camera Tracking Commands (For more information, see Vehicle Camera Tracking)

  • track_camera { on | off | auto } - Tells the camera to track the vehicle node that was used in this command. The arguments tell the camera when to record: 

    •  on - The camera will immediately start recording (if it is not already)

    • off - The camera will stop recording

    • auto - The camera will record when the vehicle is armed

  • list_recordings - Lists all the recordings that have been recorded. Lists them with an index/recording_id attached to each (starting at 0).

  • download_recording <recording_id> - Downloads the recording corresponding to <recording_id> to the E-VM ~/Results folder. The <recording_id> is the index printed when using the list_recordings command

For Computer:

• start_experiment - runs /root/startexperiment.sh on node’s E-VM

• stop_experiment - kills all screen sessions on node’s E-VM Always use all stop_experiment after the vehicle lands. A stop in the air can leave the vehicle in RTL mode and can stall the C-VM.

• restart_cvm - restarts c-vm scripts in the corresponding node - this will reset the heartbeat counter, and for vehicles return them to their home positions (disarmed) Use it when the console shows "E-VM Vehicle MAVLink Connection BROKEN" or "Link timeout, no heartbeat". Wait about 60 s before you arm the vehicle again.

• download_logs - downloads last date's dataflash log files from the UAV or UGV's internal autopilot storage to the C-VM. Use transfer_logs to transfer this data in the E-VM.

• transfer_logs - transfers MAVLink logs, MAVLink Filter logs, OEO logs, dataflash binaries from the C-VM to the E-VM  /root/Results/c_vm_logs directory

• ping - makes all running computer helpers print a response to the console. Useful for verifying functionality of OEO infrastructure.

• Reset commands in the digital twin:

• restart_cvm - run it in the OEO Console, for example 2 restart_cvm. It resets the C-VM of one vehicle.

• reset_chem - run it in a shell on any E-VM, not in the OEO Console. It resets only the CHEM channel emulator. Add -v for more output.

• /root/reset.sh (or exp_reset) - run it in a shell on any E-VM. It resets all C-VMs and CHEM.

• The OEO Console has no reset_chem command.

Interacting with AERPAW Services

The OEO system comprises several microservices. Below is a short description of each of these microservices - for a full description see the corresponding manual page:

• The collision avoidance microservice is watching all vehicles in an experiment and stops an experiment if a collision is imminent.

• The link quality microservice is providing a link quality estimate for each node in the experiment. In the development environment the link quality should always be high.

• The ADS-B service provides warning to the operator in the field if a manned aircraft is expected to pass too close to our flying field. This service is only active (and relevant) when an experiment is in Testbed mode.

• The vehicle camera service allows for automated tracking of vehicles during an Experiment in Testbed mode (no camera service is available in development mode)

All commands interacting with a "microservice", such as the service management service or anti-collision service must be preceded by the service's identifier. See subpages for more information about specific services.

There is a service to perform high-level operations (starting/stopping/etc) other microservices that is accessed with the service name. Commands are listed below:

• Service management commands

  • service service_list - List all running services (out of all possible) on the OEO-Server

  • service service_restart <service_name> - Restart/start (if not already running) a service by name. Shorthand names (e.g. col) are displayed with service_list

  • service service_stop <service_name> - Stop a service by name. See above.

  • service service_start_needed - Launch all non-running services that are considered "essential" for an experiment

  • service service_shutdown - Stop all non-"essential" services, for the purpose of cleaning up lingering resources and killing computationally expensive services