CSVMerge Script

Overview

To facilitate the integration of CSV data from multiple csv file with time-stamps we provide the Experimenters with a merging script called csvMerge.py. The script takes two csv files as an input and creates a third one as the integrated output of the two files, using the time-stamps in both files.

Location

This script is found in:  /root/Profiles/PostProcessing/csvMerge/csvMerge.py

Execution

Minimal usage:

    python3 csvMerge.py radioOrTrafficLog.csv vehicleOut.csv

Optional parameters:

    --output: <output.csv>      the .csv file in which to store the merged data; by default the script outputs on the standard output (screen)

    --format: [c]|[i]          how to merge data from second input file; by default the script uses interpolation on all the fields of the second file

    --no-trim:                  using this option means the non-overlapping time-stamp intervals will not be trimmed

Details

The two input CSV files (radioOrTrafficLog.csv and vehicleOutLog.csv) are assumed to start with a timestamp. By default, in AERPAW the timestamp uses the format produced by the Linux utility ts with the following format:

ts [%Y-%m-%d %H:%M:%.S]

For example, this is a resulting time-stamp produced by ts using that format:

[2021-09-15 20:55:30.544299]

The first thing the csvMerge script does is trim (throw away) the data rows for non-overlapping time-stamp intervals. In other words, the resulting output.csv file only includes time-stamps strictly in the range of time-stamps in both input files. If the experimenter wishes to keep all timestamps, then they can use the --no-trim parameter.

The resulting output file will have a copy of all the remaining (after trimming) timestamps in the first input file as well as the associated data from the first input file. In addition to the columns from the first input file, each column in the second input file will be either copied or interpolated in the output file. By default, the data from the second input file will be interpolated in the output file. However, if the experimenter desires to copy the data instead of interpolation, then the experimenter will have to use the --format parameter.

• Interpolation: for each time-stamp in the output file, the data will be linearly interpolated using the two adjacent timestamps in the second input file.

• Copying: for each time-stamp in the output file, the data will be taken from the closest previous timestamp in the second input file

Example

Assume that an Experimenter desires to merge the following two csv files:

1. logUe-12-8-21.csv

2. vehicleOutLog.csv

To merge these two csv files, the following command should be run

   python3 csvMerge.py logUe-12-8-21.csv vehicleOutLog.csv --output ueMerged.csv

As a result, "ueMerged.csv" file will be generated.