3.5) Data Post-Processing

Overview

The figure above summarizes the data post-processing in AERPAW. In short data from the processes running in the testbed or development mode is saved in timestamped text files, each file corresponding to the respective processes generating it. The text files (each with a format particular to the process that generated the file) are then converted to Comma-Separate Values (csv) files. The csv files are then merged into a single csv integrated file  (assuming an integrated file is desired). Finally, the integrated file is then processed into a form that is useful for data analysis (e.g., Matlabl figures, kml files, etc.). In what follows, we provide details on each of these steps.

Producing the Log Files

As described in the overview page, both in the testbed environment, as well as in development modes, all experimenter processes produce timestamped log files. Each different process will produce a differently formatted text file, which is characteristic to that process. The only common characteristic of all the log files is that each line of text is prepended (prefixed) with a timestamp, which is recorded at the time the line was printed by the corresponding process that produced the line. Also, importantly, all the timestamps use the computer's system time, and are, thus synchronized. Also importantly, logs produced on different computers (possibly in development, and definitely in the testbed), are, therefore, not necessarily synchronized.

Log Post-Processing

The first step in processing the log files generated on the testbed or in development mode is converting them to a coma separated values (csv) format. To this end, the AERPAW team is providing for each type of process we provide support for a way to convert from the text files to csv files. Power users can write their own for post-processing custom software.

Log to CSV File Conversion

The log files are generated in .txt format. Log files may include various information useful to the experimenters (including failures, warnings, errors, etc.). However, for processing the data we developed a script that converts each type of log file to csv (so it can be further processed). The script is located at:

/root/Profiles/PostProcessing/log2csv.py

Usage:

python3 log2csv.py file1.txt file2.txt -m <mode> -o <outputFileName>  

    --mode or -m   : <MODE>  modes are specified as below

        ue, enb, epc, ping, iperfClient, iperfServer, cellSearch, vehicleLog, vehicleOut, gnuradioOctave, gnuradioOfdm 

    --output or -o : <output.csv>  output file name. 

Example usage:

 python3 log2csv.py ueLog-12-8-21.txt -m ue -o ueLog-12-8-21.csv

The command will generate a csv file for the srsLte UE log file.  For other types of log files, you need to pass related mode parameter. 

 Merging CSV Files

The second (optional) step in processing the log files is merging of data from different csv files with time-stamps into a single integrated csv file.  To assist the Experimenters, the AERPAW team has provided a csvMerge script that can merge two csv files into an integrated one with a single timestamp. In short, the csvMerge script can be called to merge two files using the following command:

python3 csvMerge.py   radioOrTrafficLog.csv vehicleOut.csv  --format [c]|[i]  --output <output.csv>

More details are provided on the csvMerge page.

Please keep in mind that only csv files originating on the same computer (on the same E-VM as far as the Experimenter is concerned) are guaranteed to have their time-stamps synchronized. Merging files originating from different computers may produce meaningless files (possibly misleading files).

Processing the Integrated Data

The final step in processing the data is heavily dependent on the experiment being performed and on the needs of the Experimenter. The resulting csv file can be easily imported in Matlab, Excel, or a variety of processing applications capable of producing graphs, or other forms of visualization.  For ease of visualization the AERPAW team is providing a script that can be used to produce .kml files that can then be visualized in Google Earth. A sample figure produced with this script is included below, showing the ping duration from a uE on a UAV to the enB at LW1. Red is used for higher ping duration, while blue shows the lower ping duration.