RQ3: Can our technique help in detecting discrepancies between repOK specifications and the API's object generation ability?

For RQ3, we analyzed the repOKs provided with the classes of the four benchmarks, in terms of the structures that these classified as valid, with respect to the structures can be constructed using correct API builder routines, for the same scopes.

We ran BEAPI to generate a bounded exhaustive set of structures from the API (SA), and Korat to generate a set of structures from repOK (SR). Thus, for a repOK to be an exact description of the structures generated by the API, SA should be equal to SR for same scopes.

As an example, in the table below, we show a run for the NCL case study, where we check if the structures produced by BEAPI are included in the set of structures produced by Korat and vice versa not only for the same scope but also comparing each scopes with equal or higher scopes.

• To replicate this experiment for this case study and for all cases study you can run the script run-all-begen-serialize-exp.sh located in folder scripts (see home page)

• We provide a script (process-results-inclusion.sh) to read and process the results obtained after running the script mentioned in the previous item and display them as shown in the table here.

• We also get log files containing the structures reported as not included. In folder scripts/results-begen-serialize, for example, for NCL, korat, scope 3 , you can find (after running) structures not included in BEAPI, scope 3 in structures-not-included-3.txt log file.

One canonized structure produced by korat that we can not get from NCL API routines invocation , reported after the described process is

As we can observe the value of dummy node of main list is 1 and should be forced to be null!