Benchmark-online

The program "filcorr-online-perf-bench.jar" is used for benchmarking the online performance of the FilCorr algorithm. It is based on the JMH testing framework, but we use the maximum number of time series and pairs that the system can handle without causing any delay as the benchmark results instead of the results from JMH which is measuring execution time.

The program will simulate the time series streams by generating a certain number of observations at a specific rate specified by the user inputs. The basic idea is that the system cannot handle that much time series if the execution time exceeds the available time. Such execution time is measured as one round lagged correlation computation for all the pairs at a timestamp, and the available time for such computation is the period for the output rusults which is

(1/output Rate), the output rate is computable as (f/step). For example, if the sampling rate is 100Hz, step is 10 then the output rate is 10Hz, and the available time is 100ms.

Our approach is to check whether the final round of computation finishes within the available time. The final round exists since we only simulate a specific number of observations. The reason to not check each computation time is the error will accumulate if the system cannot handle that many time series, so all results of the lagged correlation computation will reflect at the final round. The error is the extra time needed for one round of computation to finish beyond the available time. The program will run eight times for a certain amount of time series. If there are more number of runs pass than the fail, we consider it passes. When tie, we compare the average time with the available time. If the average time is less than the available time, then pass.

Since FilCorr is data-independent so we use random synthetic data generated automatacially by the program based on the user inputs.