Chapter 10 Scoping Reconn-exion
A Ph.D. Thesis by Andrew Le Gear
[Back to Home Page] [Previous Chapter] [Next Chapter]
“What you perceive, your observations, feelings, interpretations, are all your truth. Your truth is important. Yet it is not The Truth.”
-Linda Ellinor.
In chapter 6, the concept of validity was explained. It is the purpose of this section to revisit the two evaluation chapters (chapters 8 and 9) and examine the evaluation in terms of validity. The evaluation in this thesis attempted to heighten validity by examining variations of the potential populations during the studies:
•Many different component types were recovered during the course of the studies. These component types included:
–A highly reusable utilities component.
–Two unidirectional user interface components:
*A character screen user interface component (RF_SCREENS).
*A web interface component.
–A data access layer component (helpers DTO).
•The participants had many degrees of experience ranging from 2 to 16 years experience and in between.
•Applications from a variety of domains were used, including:
– An Enterprise resource planning (ERP) domain.
– A Distributed learning and collaboration.
•All applications were of varying size ranging from 20,000 LOC to 2,000,000 lines of code.
•For the two larger case studies, two participants participated in each, increasing the population size and therefore the degree of control for each study. However, this population size falls far short of cancelling the invalidity effect and this is possibly the largest threat to validity of the studies. The low population size prevents
us from drawing significance from statistical analyses. However large numbers are difficult to obtain in in-vivo studies and further replication by interested researchers may serve to buttress these results.
In line with state of the art recommendations described in chapter 6, the studies undertaken were qualitative, and very high in ecological validity:
•All participants were maintainers of the systems examined.
•The component encapsulation tasks were in line with the maintenance agenda of the participants.
•The studies were undertaken in the natural work environment of the participants. One threat to validity in this evaluation is the inability to implement control and isolation of variables when using case studies. However, as highlighted in chapter 6 by (Segal et al., 2005) the inability to replicate an experiment, due the lack of controls
and known variables is not at issue. Rather the replication of interpretation by others is what should be achieved. This position is evidenced by the many peer reviewed publications arising from the work of this thesis (Le Gear, 2004; Le Gear and Buckley, 2005b,c,a; Le Gear, Buckley, Cleary and Collins, 2005; Le Gear, Buckley, Collins and
O’Dea, 2005; Le Gear et al., 2004; Le Gear, Cleary, Buckley and Exton, 2005; Le Gear et al., 2006). That is not to say that experimental control should be ignored. The studies undertaken in this thesis, do lack the laboratory control. This removes a valuable avenue of triangulation of evidence and should be addressed as future work. However, as a preliminary evaluation of this prototype process the findings here provide valuable
initial evidence.
Another threat to validity arose during the evaluation of the components recovered with the original architect of the AIM application. Some of the questions posed to the architect were leading and may have biased his subsequent answers.
Also, it is possible that the use of observation and think-aloud may have introduced an element of the Hawthorne effect, whereby the very knowledge of being observed can affect the performance of a participant (Wikipedia, 2006a). While an element of this may be unavoidable, the contextual evidence suggests that its influence on the study is small.
Familiarisation time with tools used in the studies is also a factor that should be taken into account and would have had an impact on the participants’ productivity at the outset of the study. However, it should be noted that any such difficulties were quickly overcome, to such an extent that the participants have even recommended the
tool to colleagues in the organisation. At present, 20 further installations of the jRMTool have been made, with reports of 6 software engineers using the tool to aid their regular working tasks.
Indeed, if anything, this unfamiliarity would have lowered productivity and raised resentment to the process. Yet our findings contradict these hypothesised trends.
It is also possible that there is a threat to the construct validity of the studies. By examining think-aloud data, quotations were gathered that appear to show evidence in support of the research questions under study. However, deciding what qualitative data supports the question being analysed is a subjective task, and introduces an element of ambiguity to the interpretation of results.
Finally, it should also be noted that due to spacial, copyright, anonymity and ethical reasons approximately only 25% of the relevant quotes gathered have been included in this thesis. In chapter 7 Reconn-exion was described. Its potential merits were discussed and justified in literature, and a demonstration provided to show the usefulness and benefits of the technique. This section aims to constructively criticise the the technique by examining its theoretical liitations.
A first, and pervasive limitation is the inability for static analysis techniques to identify all dependencies in source code. Dependency identification is necessary when identifying the source model used to implement the variation on Reflexion Modelling in Reconn-exion. An example of where dependencies would be missed is when analysing
C or C++. When analysing these languages is may not always be possible to know where pointer references point to at a given time, thus some call dependencies may be missed. In this thesis the languages Progress 4GL and Java were analysed. In Progress 4GL an equivalent to a function pointer can sometimes be used to store a reference to
a procedure. This resulted in some, but not a detrimental amount, of call dependencies being missed in the source model. In the Java source code analysed in this thesis, some of the calls in the source code were made using SOAP (a distributed messaging service) (W3C, 2006), which were not identified in the generated source model. Fortunately, as with Progress 4GL, the dependencies missed were not very prevalent.
Similarly dynamic analysis, as used in the creation of the reuse perspective in Reconn-exion, cannot guarantee complete coverage of a system, since it may not be possible to design a set of test cases that exercises all code paths (even for a chosen, constrained domain). In terms of Reconn-exion, this means that for a chosen set of features, that it is not possible to guarantee exercising all the source code responsible for implementing those features. For example, the systems analysed by the case studies in this thesis, JIT/S, AIM and Workplace, are all highly configurable enterprise systems. The number of possible configuration variations could be extremely large, therefore making it infeasible to profile all subtle configurations of the system that would ensure complete source code coverage. However, this problem is partly mitigated in the studies in this thesis for two reasons:
•Since the basis of the reuse perspective is in the relationship between features, it is the feature coverage that is more important than source code coverage.
• Furthermore, the set of features being selected is usually for only a selected part of the system, therefore one is rarely trying to achieve complete source code coverage when creating a reuse perspective, but rather, focussing on functionality of interest.
The identification of features, which is a necessary part of the Reconn-exion process, is a highly subjective task. The decision as to what features exist, and the definition of those features are almost completely left to the discretion of the user. This can lead to different interpretations of what constitutes a feature, and what can be classed as a comprehensive feature set for the domain. For example, in the example of the creation of the reuse perspective in section 7.4, five features are identified. Three of these are rotate, translate and scale. However, another person choosing features of the application may have understood rotating and scaling as types of translation, therefore classing those three different behaviors as the same feature. In the case studies in this thesis,
this problem is partly mitigated in certain cases, due to the fact that the participants had extensive experience in the application domain. However, regardless of experience, conflicting and equally valid interpretations of feature sets can be formed, even by separate experts.
Also, assuming the features have already been chosen, the design and implementation of test cases that exhibit those features correctly is a task left open to error. This is again due to the subjective nature in which a person can create a test case. For example, given two people designing a test case to exhibit a feature, both may design different test cases due to either different interpretations of that feature or their understanding of what the designed test case implements.
Up to 10% of all source code in a system is often cloned (Baxter et al., 1998). Given such circumstances it is possible to have what appears to be a single feature in operation, implemented in several places in the source code, unbeknownst to the user. This can have an adverse affect on the implementation of the software reconnaissance technique, and will result in none of the cloned source code being identified for that feature. For
example, give two profiles gathered from two test cases:
•Test-Case-Profile-1= {method1,method2,method3}
•Test-Case-Profile-2= {method1,method4,method5}
Both test cases are profiles representing the same feature. Method 1 is initialisation code and does not implement the feature. The methods 4 and 5 are clones of 2 and 3 respectively and are responsible for implementing the feature being profiled. If we now calculate the set of involved software elements we get:
IELEMS = {method1}
The methods 2,3,4 and 5, which are the ones we want to locate, are lost in the calculation. Users, generating a reuse perspective should be aware of this limitation when applying the technique.
Finally, Reconn-exion is an encapsulation technique, which means that the component is simply modelled and not extracted as with component recovery. This has been adequately highlighted in this thesis, however it still remains as a limitation that further work will need to be performed if one wishes to extract and reuse the component.
[Back to Home Page] [Previous Chapter] [Next Chapter]
Component Reconn-exion by Andrew Le Gear 2006