1.8) Experiment Lifecycle Overview

Interaction Details in Canonical Experiment Lifecycle

The following diagram shows the main stages of an Experimenter's use of AERPAW.  This diagram should be considered in conjunction with the diagram on the "1.3) Overall Platform Architecture" page earlier in this section; please read that section in its entirety before reading this page, since we refer to concepts and terminology introduced there.

Below we describe briefly the various steps in which we have decomposed the interactions that occur in this lifecycle.  More detailed information and help is available for each of these interactions in subsequent sections of this User Manual.

Steps {3-5, 9-14, 16-19} are not detailed below, beyond their self-explanatory listing in the diagram.  These steps are provided here to help AERPAW Experimenters understand the bigger picture of the user-programmed but batch-executed testbed discipline. They are undertaken by AERPAW Ops and the automated AERPAW Platform Control system, and are completely transparent to the User.  The rest of the steps shown in the diagram are explained below.

Your AERPAW Work Computer

To use the AERPAW Web Portal, and later to access the AERPAW Virtual Environment to develop your Experiment, you need to use a computer of your own - typically your work desktop or laptop.  In what follows, we refer to this as your work computer.  (Of course, you may possibly use different work computers at different times.)

While most computers with basic Internet access will suffice for the web-based interactions with the AERPAW Portal, to continue successfully with Virtual Environment access as required for Canonical Experiments, you must have:

• Administrative authority over your work computer - the ability to install and configure software packages, the ability to install VPN profiles;

• Unrestricted Internet access - the ability to reach the VPN ports that are dynamically generated for your specific Experiment.

In our experience, the best match and most predictable experience in accessing AERPAW is provided by using a work computer running some variant of the Linux Operating System.  While we continue efforts to fully support access using Windows or MacOS computers, the lack of several key pieces of open-source software on those platforms, and the limitation of capabilities even for Administrative users, makes success unpredictable at best.  Further, the required reconfigurations of the OS (and networking stack) may not be something Users may be comfortable with in making on their own personal productivity laptop.

If at all possible, we strongly suggest using a Linux-based work computer, and one in which you do not keep other sensitive productivity-related information, for Canonical Experiment access.

Platform Access (Web Portal)

Step 1: Register and Supply Credentials

To access AERPAW using the Portal website, an Experimenter needs to initially use the Portal and create an AERPAW User ID; this is a one-time operation.  To be able to create Projects and Experiments, it is necessary to request corresponding specific roles.

• Section 2 provides the link to the Web Portal (also linked from aerpaw.org), but we strongly advise you to finish reading this section through once completely before attempting to access the Web Portal for the first time.

• When you first register as a User in the AERPAW Portal, you have no assigned Roles.  In order to use AERPAW for performing experiments, you must have at least the Experimenter role.  So, you should request the Experimenter role at this point.  This request will need to be reviewed by AERPAW authorities (typically the AERPAW PD, or possibly the Steering Committee), so it might be a day or two before you receive approval.

• After you receive approval to be an Experimenter, you can similarly request a role as a Principal Investigator (PI) if you expect to be in position to oversee the work of various Experimenters working with you in your organization, and in particular if you will be the person responsible for payment.

• More specific information on these actions are provided in Sections 2.1 and 2.2 of this User Manual.

In a later step, you will use the ssh utility to access the AVNs, with keyed access that ensures nobody else has access to these AVNs.  Using the Portal you have to supply the public part of the key to AERPAW Ops, so it can be installed into those AVNs.   The logical time to supply such credentials is now, right after registering yourself as an AERPAW User.

The ssh credentials take the form of a public-private key pair.  You should supply the public key to AERPAW via the Portal, and save the private key in your work computer.

• This is described in more detail in Sections 2.2 ;  use of these credentials is described in Step 7 below, and in Section 2.6.

Experiment Creation (Web Portal)

Step 2: Create Experiment, request development session

In AERPAW, Experiments are created under Projects - no Experiment can exist without a Project to house it.

• Experimenters with the additional PI role are able to create Projects, and add other Experimenters to their Project, and even delegate Project Owner authority to additional project members other than themselves.

• Projects are either public or private:

  • Any Experimenter can ask to join any public Project; Project Owners can approve or decline such requests.

  • An Experimenter can only become a member of a private Project by being added proactively to the Project by a Project Owner.

• Sections 2.3 provides detailed help on the Portal interactions to create and manage a Project.

• Once a member of a project, an Experimenter can create an Experiment under that Project.

The first step in creating an Experiment is to give it a name, a short description, and to enter more details .

• In the Portal, a short name and a short description of the Experiment should be entered.

• More detailed information must be entered using the  "AERPAW Experiment Information Request Form" linked from this page of the Portal.

• Section 2.4 provides help on the Portal interactions to create an Experiment within a project.

------------- Beyond this point, the rest of the description is only applicable to canonical Experiments (see flowchart above) -------------

The next step is to define the set of ARN resources that your Experiment will target in Testbed execution, and save this definition.

• You should select the set of AERPAW Fixed Radio Nodes on which you intend to run parts of your Experiment.  These are the actual AHNs on which your code will actually run during Testbed execution (although you will develop this code in a Virtual Environment).

• You should select a set of AERPAW Portable Nodes that are the same in number and functionality as the ones you intend parts of your Experiment to run on.   (In AERPAW, each Fixed Node is a unique resource, due to its location; but all Portable Nodes of a given class - e.g. Large Portable Nodes - are interchangeable; your Experiment will run on some specific set of nodes that will be assigned to your Experiment at the time of Testbed execution, and that are each of the same type as the corresponding node you specified.)

  • For each Portable Node, you also have the option of specifying a programmable ground vehicle or air vehicle.

• In selecting the set of Fixed Nodes, and specifying the set of Portable Nodes (and optionally vehicles), it is a good idea to refer to Sample Experiments (Section 4 of this User Manual), and model your Experiment initially after one of them.

• Section 2.4 provides help on the Portal interactions to create an Experiment within the project.

Once you have completed creating a new Experiment request and saved it, you can use the Portal to "Initiate Development Session" for this Experiment.

• You can also use the Portal to view all the Experiments you have either created, or are a member of, and view their states.  You can request a Development session for any Experiment that is currently in "Saved" state.

• Once you trigger your first Development session for a newly created Experiment, you can no longer modify the set of target resources.

• Section 2.4 provides help both on managing your Experiments, and requesting a Development session.

Usage of AERPAW, and in particular the traversal of this workflow, is conditional on your acceptance of the AERPAW Usage Policy.

• By traversing this workflow, and making any use of AERPAW equipment, software, or services, you are indicating that you have read and accepted the AERPAW Usage Policy.

• Section 2.5 of this User Manual specifies the AERPAW Usage Policy.

Receive Experiment Access (Web Portal)

Step 6: Receive Virtual Manifest Info

When creation of the Development Mode Experiment in the AERPAW Virtual Environment is complete, you should receive an email from AERPAW Ops (either <aerpaw-operations@ncsu.edu> or <aerpaw@gmail.com> , please make sure mail from these addresses are not going to your Spam or Junk).  However, you can also check the status of your Experiment on the Portal at any time you choose; simply re-load or refresh the page on your web browser.  (If you suspect you may be seeing out-of-date information, log out and log in again, or follow mechanisms specific to your browser to delete cache or other context.)

At this point, as you view this Experiment on the Portal, you will see that links to two files have been added.  One of them is an OpenVPN profile file that allows you to securely access the Virtual Environment that houses your Development Mode Experiment, in the next step.  This environment is unique to your experiment, and contains AERPAW Virtual Nodes (AVNs) - virtual versions of the exact set of nodes you identified in Step 2.  Your virtual environment is distinct from that of other Experiments belonging to other Experimenters, and should be accessed only by yourself.   You should not share these OpenVPN credentials with anybody else.

The second file is a "Manifest" - simply a list of the IP addresses you will use, in the next step, to access each of the AVNs.

• Section 2.4 provides help on accessing and using both the OpenVPN profile file, and the Manifest file, from the Portal.

You should download and save both these files in your work computer (the computer from which you intend to remotely access the AERPAW Virtual Environment - the same computer on which you have stored your SSH credentials in Step 1 above).

Experiment Development (OVPN, ssh)

Step 7: Login to AVNs, Code, Test

You can now login to each AVN of your experiment, examine and re-use provided sample code, write your own code, and test them in the virtual environment of the Development Session.

• Section 2.6 provides detailed help on the various steps listed below for accessing the Development Session.

First, you must access the custom Virtual Environment that has been created for your Experiment.  For this, you will utilize a OpenVPN or compatible client (depending on the OS on your work computer) to securely connect your work computer to this custom environment.

• This step will create a new virtual network interface on your work computer, which will be on the same virtual Layer-2 network as all the AVNs on your experiment.  This Layer-2 network is your Experiment Management (XM) network, and is provided exclusively for the purpose of accessing and controlling/coordinating your Experiment.

• This new network interface will receive an IP address like 192.168.X.Y , with a netmask of 255.255.255.192 , where X is a number between 100 and 200.  The subnet indicated by this address, 192.168.X.0/26, is expected to be dedicated to your Experiment Management (XM) network.  You must not have any conflicting routes to any part of this address space, or any conflicting interface address assignments, on your work computer.  Please ensure this is true before attempting to VPN in; otherwise the access will likely not work, and the results are unpredictable.

Now that you have securely connected to your XM network, you can access each of your AVNs.

• The Manifest file you retrieved in Step 6 lists the IP addresses of each AVN in your Virtual Environment Experiment.

• You can access each of them only using the ssh credentials you provided in Step 1; no ID-password based login is possible to these AVNs.

• The Manifest file provides the mapping between the target AFRNs/APRNs you selected in the Portal (in Step 2 above) and the IP addresses.  This allows you to configure and code each AVN according to the role you envision for them in the Experiment.

  • For example,  your Experiment may comprise two AERPAW Nodes - the Fixed Node LW1, and an APRN.

  • On the Portal when you view your Experiment, you will see that a numbering has been established for these nodes - say LW1 has been numbered Node 1, and the APRN as Node 2.  This numbering is specific to this Experiment, so we call these numbers the Experiment Node Numbers.

  • The Manifest file will  list one IP address for Node 1, and one for Node 2.

• In addition to the AVNs, the Manifest file also provides the IP address for one additional Virtual Node: a custom OEO Console for your Virtual Environment Experiment.  You will use this OEO Console to control and configure specific aspects of your Experiment.  More details are provided in Section 2.6, and specific Sample Experiments also provide specific instructions on how to use the OEO Console.

• Caution: The AVNs are equipped with whatever credentials were saved in the AERPAW system for you at the time you used the Portal to "Initiate Development Session".   If you change your credentials after making this request, the AVNs will not reflect that change.

 You can add other AERPAW Users to your Experiment, and plan to develop your Experiment on the AVNs jointly.  The Experiment itself can be only in one state, and only the User who created the Experiment (you, in this running example) can use the Portal to "Initiate Development Session".  However, once a Virtual Environment has been instantiated as a development session, all the Users who are Members of an Experiment can access the Virtual Experiment as above, in sequence, or even simultaneously.

• Provide the same OpenVPN Profile file to each such User (or ask them to retrieve it from the Portal, as above), and access the Virtual Experiment.

• Each User will be able to log in to AVNs using their corresponding ssh credentials.  Please do not share ssh credentials between Users.

• It is up to you to coordinate the work of the multiple Users accessing the Virtual Experiment; the AVNs provide equal privilege to all, and simultaneous Users must exercise care not to clobber each others' work.

• Caution: If you add other AERPAW Users to your Experiment after you used the Portal to "Initiate Development Session", the AVNs will not have their credentials, and they will not be able to login to the AVNs.

The Virtual Environment is, of course, just to enable you to develop the Experiment - the meaningful experimental data can only come from executing that Experiment in the physical testbed environment.  However, you will likely want to develop the Experiment in incremental stages, and test it for correct execution at each intermediate stage.  The Virtual Environment includes an Emulation system for both radio control software and vehicle control software, so that the Experiment software can have the same programming interface (function calls, returns) as when it executes in the Testbed.

• Please note, however, that the Emulation included in the Development Session is merely an enabler for the software development/test process.  The R/F and airflow/flight models included in this Emulation are simplistic, and no realistic data can be produced by virtual experimentation in the Development Virtual Environment.

• As above, Sample Experiments are a good way to get started with this process; each of them is a complete Experiment that can be run and tested as soon as you complete setting it up (following corresponding instructions in Section 4), before you start adding/modifying your own code to build on it.

At any time during development, you can use the "Save" feature for this Experiment from the Portal, to cause full images of each AVN to be saved at this time.  If your Virtual Experiment Environment should crash, it may be possible for AERPAW Operations to restore the last such saved checkpoint.

• Please note, however, that the "Save" operation is quite costly in time, as well as disk operation and computation.  You will lose access to your Experiment during the time it is being saved, which can easily take 20 - 30 minutes (more if you have a larger-than-normal Experiment).   Please use this feature only for very critical needs, and only sparingly.

• Section 2.4 provides help on the Portal interactions to "Save" an Experiment.

Submit to Testbed (Web Portal)

Step 8: Submit Experiment for Testbed Execution

During the development process, you can perform interactive operations - a typical example is starting the Experiment run (kicking off the various processes at each AVN in your Experiment), another is stopping the Experiment run when you are satisfied that the purpose of the run has been fulfilled.  More sophisticated examples include performing coordinated actions to facilitate study of some phenomenon ("... thirty seconds into the run, start a heavy traffic flow from Node 1 to Node 2...", or similar).  However, recall that AERPAW is a batch-mode facility, and therefore during testbed execution you will not have this backdoor access to the AERPAW Nodes.  Any such action, including starting and stopping the run, must be scripted for automatic execution, before submitting the Experiment for Testbed execution. 

• Again, the Sample Experiments provide examples of how to choreograph your Experiment execution completely from the provided bootstrap execution points at each AVN.

• To facilitate your building in such Experiment execution coordination in your code, the XM interface at each AERPAW Node maintains the same IP address in the Testbed Environment as they had in the Virtual Environment, so you can code Experiment management/control messages to go back and forth between AVNs depending on these addresses.

After you are satisfied that your Experiment code will do what it is intended to do when it is kicked off on the Testbed, log out of all AVNs, and use the "Save and Exit" feature from the Portal.

• Section 2.4 provides help on the Portal interactions to "Save and Exit" an Experiment.

• As with the "Save", this operation will likely take 20 - 30 minutes, or more.

• After the "Save and Exit" process completes in the back-end, the status of your Experiment will again show as "Saved" on the Portal.

At this time, you can submit it for Testbed Execution.

• Section 2.4 describes the Portal actions to submit an Experiment to the Testbed.

Request Re-Development Session (Web Portal)

Step 15: Request Re-development of Returned Experiment

When Testbed Execution is complete, you will receive an email notification; the status of the Experiment will also change on the Portal to show again as "Saved".

In order to view the results, you will again need a Development Session for your Experiment.  You will need to again "Initiate Development" from the Portal, exactly as in Step 2 above.

• Section 2.6 also describes this workflow; please also refer to the aforementioned help for Step 2.

Examine/Retrieve Results (OVPN, ssh, scp)

Step 20: Access Development Session, View Results

When AERPAW Ops has again set up the Development Environment Experiment Session for your Experiment, they will once more send you email; you can also check the status of the Experiment in the Portal.  At this time, you can again log in to the AVNs, as in Step 7 above.

• Section 2.6 also describes this workflow; please also refer to the aforementioned help for Step 7.

At this point, any results saved by the Experiment logic during Testbed Execution will be available to you in the Development Session AVNs.

• Section 2.6 also describes this workflow.

If you wish, you can continue modifying/extending your Experiment in this Development session.

Repeat as Necessary; Retire when Done

The above Develop-Submit-Review cycle can be traversed as many times as necessary.  When you are confident that you will never use the Experiment again in its current form, we ask that you "retire" the experiment, so that we can purge the Experiment from our system, and free up resources such as storage, CPU, IP/VLAN/VPN namespace, etc.  Prior to doing so, please remember to retrieve ALL of your data and code from your AVNs.

• This can be done from the Portal, please see Section 2.4; you can also email AERPAW Ops for help.

• If an Experiment is dormant for a significant period, we may conclude that it is in effect retired, and remove it from our system.  We will always attempt to get in touch with you before finally doing so.

• To "archive" the AVNs of your Experiment, you can use the "Save and Exit" feature (as in Step 8 above) to store AVN images from your Experiment on AERPAW itself.  Please contact AERPAW Operations in order to arrange procedures for you to retrieve these AVN images.

• In future, we hope to add the functionality of using such images to bootstrap subsequent experiments you create.  Unfortunately, this capability is not yet available.

See Complete Example

Section 3.3 of this User Manual provides a complete example of traversing this workflow, with a specific example that makes a specific choice of ARNs, decision to use specific sample software, etc.

Also this video on creating canonical experiments using the Program-it-Yourself approach (showing Portal interactions as of March 2022) may be of use.  (NOTE: as of January 2023, with the Portal migrating to a new version, this video is outdated, and of limited value; we hope to replace it with an updated video soon.)

Non-Canonical Workflows / Specialized Equipment

AERPAW includes some industry standard commercial equipment typically not available in academic laboratories; and the goal of the AERPAW project is to make them available for use by our experimenters.  However, many of these are equipments or ecosystems of components that were designed exclusively for live access and control by an operator (specially trained for that purpose), and do not support a programmed or scripted mode of operation, nor functionality that can be accessed through high-level, easily accessible APIs suitable for a generic researcher.  We use the broad term “Third-Party Black-Box Equipment” (3PBBE) to emphasize this essential nature common to them all.  Confirmed such equipment at the time of this writing includes Keysight RF Sensors, Ericsson 4G/5G base stations and RAN/core system, and Facebook Terragraph system.

The canonical mode of operation cannot automatically encompass such 3PBBE, because of the lack of (i) virtual computational models of such 3PBBE that can be incorporated into AERPAW’s Development or Emulation environments, (ii) programmable control/configuration capabilities, to enable batch operation of Experimenter code (without live console access), and (iii) onboard programmable supervisor to run AERPAW Node orchestration software.  Such capabilities also cannot be expected in the future from the respective vendors, since they pose no commercial advantage.

Accordingly, we do not expect to provide 3PBBE access through the canonical workflow.  The following non-canonical workflows can be used to provide interested and qualified experimenters with such access.

Experiment-as-a-Service (EaaS).  This workflow is essentially identical to the canonical workflow, and can be used even for canonical Experiments by Experimenters who are not comfortable with SDR or UAV programming, and uninterested in learning.  In this workflow, the Experimenter simply describes (in documents, and through interaction with the AERPAW team) their experimental intent, and the experiment design, implementation, and execution, are all carried out by AERPAW personnel.

Although it is a much more protracted process than Program-it-Yourself, it has the advantage that non-programmable 3PBBE resources can be accommodated in the experiment without additional effort.  

Experiment-Development-as-a-Service (EDaaS).  This is an amalgamation of the canonical workflow and EaaS.  As in EaaS, AERPAW personnel designs and develops the experiment based on experimenter intent, but once the base experiment is programmed, it is turned over to the experimenter who can further develop and customize it, to create future experiments.  Beyond the handover, the experimenter follows the canonical workflow.

However, if 3PBBE are included, then the experiment cannot return to the canonical workflow.  For such experiments, the EDaaS workflow degenerates to the EaaS workflow.

Live Limited Access.   By going through some technical material and cooperative agreements, after some initial level-setting, qualified experimenters can be provided with testbed sessions in which they will have live login access to a specific predetermined subset of 3PBBE.  Such use will not include the ability to integrate canonical experiment resources (i.e. SDR nodes and UAVs/UGVs) into such live access.

General Availability

In Phase-2, the following systems are included as part of General Availability:

• Ericsson system: Please see Sections 1.4.4, 4.1.6, and 5.1 of this User Manual on details of how to use this system in an experiment.

• Keysight RF Sensors: Please see Sections 1.4.5, 4.1.5, and 5.2 of this User Manual on details of how to use one or more of these sensors in an experiment.

Please make sure you have read these parts of the User Manual before reaching out to AERPAW Operations to propose a non-canonical Experiment.

Please also make sure to fill out the Experiment Information Request Form before reaching out (see the flowchart at the top of this page).

Custom / One-of-a-kind Experiment Proposals

Beyond all of the above, AERPAW is open to collaboratively defining custom uses of the testbed, one-of-a-kind or hero experiments.  These are likely to be extremely costly, since they presuppose a good proportion of dedicated effort by AERPAW DevOps personnel, and potentially preclude or curtail much of the normal use of the facility for an extended period of time.

Mixed and Custom Service Approaches. If none of the above approaches can accommodate some particular, complex requirements of an experimenter, the AERPAW team will be open to discussing and coming up with unique custom approaches to support such requirements.  For example, it may be possible to have a mix of various previous approaches where part of the experiment is executed in batch mode programmed by the experimenter, a part is manual by the operator, and part is live limited access. This might allow the experimenter to observe the effect of transmitting a specific waveform from a fixed node (programmed by the experimenter in canonical mode) on a Keysight RF sensor (limited live access), while a DJI drone is being flown by an AERPAW pilot (EaaS by AERPAW Ops).

Bring Your Own Device (BYOD):  In this scenario, an Experimenter may provide custom wireless equipment that they desire to be installed (temporarily) in the AERPAW testbed, or otherwise integrated into experiment operation.  AERPAW is receptive to the concept of BYOD, however our goal and responsibility remains to be able to manage the process so that the platform’s ability to verify and enforce safe operations, staying within appropriate regulations, is not compromised.  BYOD cases are natural fits for one or the other non-canonical workflows.  For example, some researchers interested in passive signature-based drone detection may want to specify that AERPAW personnel should fly a certain model of drone (or send us their own custom drones) in our wireless field, to test detection algorithms.  Such a drone would typically not be programmable, but we would still be able to manually execute the experiment, if desired by the experimenter.  We do not envisage being able to, or commit to, providing a canonical workflow for BYOD equipment.  Part of the case-by-case interaction we will undertake when considering any BYOD scenario is to assess physical, electrical, and software characteristics of such custom devices, and their manageability by AERPAW Ops during the experiment.

Please make sure to fill out the BYOD Experiment Information Request Form (see the flowchart at the top of this page) before reaching out to propose any custom Experiment.  (Fill out the same form for all custom Experiment proposals, whether you actually intend to provide additional wireless equipment for your Experiment or not.)