Chapter 8 Evaluating the Basis Techniques of Reconn-exion
A Ph.D. Thesis by Andrew Le Gear
[Back to Home Page] [Previous Chapter] [Next Chapter]
“Human beings, who are almost unique in having the ability to learn from
the experience of others, are also remarkable for their apparent disinclination
to do so.”
-Douglas Adams.
The following case study takes the reuse perspective in isolation and evaluates it in an industrial scale context. It should be noted that the entire process is not being evaluated here, rather the hypothesis that the reuse perspective contains software elements of a system that are reused, core and / or generic is being evaluated. To clarify, by reused we mean that the software element is used more than once within the context of the features
under examination in the reuse perspective. By generic we mean that the potential exists for the elements to be reused in a wider context (i.e. reusable). Finally, by architecturally core, we are again alluding to the idea that the software elements are boilerplate or framework code, that may also be generic. The evaluation is intended to
answer that the following questions:
1. Are some elements of the reuse perspective generic?
2. Are some elements of the reuse perspective architecturally core to the system?
3. Do all elements of the reuse perspective fall into at least one of the categories mentioned (reused, generic, core)?
The subject system for this case study is a commercially available, just in time, just in sequence application called JIT/S that extends an ERP (Enterprise Resource Planning) application called MFGPRO (Ltd., 2006). Both of these are products of a company called QAD, who specialise in producing ERP software products and service, catering specifically to the automotive
Table 8.1: JIT/S Summary.
domain vertical. QAD was founded in the mid 1970’s in California and quickly became a market leader in its product space. We specifically examine the Shipping module of JIT/S in this case study. The combined application was in excess of 2 MLOC. A summary of details for the shipping module can be found in table 8.1.
Progress 4GL, which is the language used to develop JIT/S, is a fourth generation language that contains many domain specific features for the manufacturing sector. The language is largely a procedural language that also contains many language features that closely tie it to database implementations. For example, database SELECT and UPDATE statements are actual keyword language constructs.
Two participants were involved in this case study:
1. An ex-employee of QAD Ltd., familiar with the behavior of the shipping module, participated to identify the features of the shipping module and gather the profiles necessary to compute the reuse perspective. For the purposes of this study she will be simply referred to as “the participant.”
2. An original architect and implementor of the shipping module participated in an interview to evaluate the contents of the reuse perspective with respect to the research questions posed in section 8.1.1. For the purposes of this study he will simply be referred to as “the architect.”
Details of both participants can be found in table 8.2.
Table 8.2: Particulars of the participants of the JIT/S study.
The tools used to enable this study were:
•The Progress 4GL tracing tool, Pro*Spy Plus (McIntosh, 2003). This tool allows program traces to be gathered from an executing Progress 4GL program.
•A tool developed by the author at the University of Limerick called “ReconCalc” was used to calculate the results of the Software Reconnaissance and the reuse perspective from the gathered profiles / traces (Le Gear, 2006).
The following protocol is adopted for the study:
1. All aspects of the study will take place in the working environment of both the participant and the architect.
2. Based on her domain knowledge of the shipping module and her perusal of the module’s documentation, the participant will be asked to create a comprehensive list of features for the shipping module.
3. The participant will then gather traces for each of these features by running the JIT/S system.
4. The participant’s role in the study will then be finished.
5. The profiles will be taken away by the coordinator of the study and the reuse perspective calculated using ReconCalc.
6. Next, in a separate session, the reuse perspective will be presented to the architect for evaluation.
7. This evaluation takes the form of an interview and is audio recorded. It is hoped that the evidence gathered here will answer the research questions in section 8.1.1.
8. In the interview the architect will have a printout of the reuse perspective and will be asked about each element in it.
The participant identified 19 features, based upon the user manual for the shipping module and her domain knowledge of the shipping module. The list should be seen as fairly comprehensive for the module because of both the extensive domain experience of the participant and the knowledge they gathered from browsing the component’s documentation. This list is summarised in table 8.3. She proceeded to design appropriate test
cases to exhibit these features and subsequently retrieved traces using the Progress 4GL Pro*Spy tracing application (McIntosh, 2003). Using ReconCalc the reuse perspective was then calculated. The process, from identification of features to the creation of the reuse perspective, was undertaken in approximately three hours. The resulting reuse perspective produced contained 14 software elements (see next section). In this case a
software element corresponds to a Progress 4GL subprocedure.
Table 8.3: Features identified in JIT/S.
Next, we presented the reuse perspective to the architect to evaluate the contents of the reuse perspective. It is important to note, at this point that the architect, in this case, is evaluating the contents of the reuse perspective and not its ability to serve as a prompt for component abstractions. An evaluation of the latter will be carried out in the next chapter, first we must confirm that the contents of the reuse perspective are relevant to the task. Therefore it is appropriate, in this case, to use the architect to evaluate the contents. The research questions outlined in section 8.1.1 are addressed by the evaluation in this section. He explained to us the use, role and function of each of the fourteen elements of the reuse perspective. The following is a paraphasing of the architects comments, with actual phrases from his summary being included in quotations. Note that the file names have been obfuscated for copyright reasons. Also, it is important to note that the architect was prompted to comment explicitly on the genericity, reusability and core-ness of the element of the reuse perspective:
1. tiuecpeftcp3.w, CreateBOL -A “general purpose” procedure that creates a place holder for bill of lading (BOL) infomation. Used by “almost every feature” in the shipping module.
2. tiuicpifgvm3p.w, getObjectType -“Framework” code and thus by definition “generic.” This is “architecturally core” to all Progress 4GL applications, not just the shipping module.
3. shtdbodeful2o.w, getObjectType -“Framework” code and thus by definition “generic.” This is “architecturally core” to all Progress 4GL applications, not just the shipping module.
4. tiuecpeftcp3.w, GetRuleGroup -This is a “generic” routine that should be “used by any procedure” involving packing slips. User specific customisations are stored in a rule group. Evertime an operation on a packing slip occurs, the rule group is referred to via this procedure.
5. tiuecpeftcp3.w, RetrieveObjectHandles -This procedure is “architecturally core” and is intended to be used by customers who wish to write their own extensions to JIT/S. The procedure is used as a “form of API” and returns the internal data structures of the shipping module to a programmer who wishes to retrieve data from the shipping module programatically.
6. tiuecpeftcp3.w, AddPackage -“Functionally core” to the behaviour of shipping.
7. tiuecpeftcp3.w, IdentifyCustomer -A “general purpose” routine that retrieves a customer based upon an ID from a packing slip. It is “used application wide.”
8. tiuecpeftcp3.w, AddPackageObj -Stores data for a package and is “used application wide.”
9. tiuecpeftcp3.w, RefreshQuery -An “architecturally core” procedure. The procedure is used for synchronisation and manages updating between the database and Progress dynamics framework (McIntosh, 2003).
10. tiuecpeftcp3.w, FinalizeShipment -This is the “common step” for all business processes and workflows through the shipping module.
11. tiuecpeftcp3.w, RunRuleGroup -This instantiates the rule group retrieved earlier. This is another “highly reused” procedure.
12. tiuecpeftcp3.w, RemovePackage -This is “used by all deletion” operations -Not just the removal of packages.
13. tiuecpeftcp3.w, serverCommitTransaction -“Framework” code and by definition is generic. This is architecturally core” to all Progress 4GL applications, not just the shipping module.
14. tiuecpeftcp3.w, getObjectType -“Framework code” and thus by definition “generic.” This is “architecturally core” to all Progress 4GL applications, not just the shipping module.
Table 8.4: Summary of reuse perspective.
The results of the examination are summarised in table 8.4. This table, details the percentage of the 14 summaries, made by the architect, that contained the descriptive key phrases of “reuse,” “genericity,” and “architecturally core.” Importantly, the entire set was confirmed as being reused by many features by the architect. Large proportions of the remainder of the set were cited as architecturally core or generic. Furthermore,
large proportions of the set also had two or three of these attributes. This evidence appears to support the research questions posed in section 8.1.1.
Of the fourteen entries, twelve originated from the same procedure file, “tiuecpeftcp3.w.” This was an interesting correlation and when the architect was asked regarding the significance of this file he stated that,
“That’s the GOD class / the controller -That’s the API class. That’s your way into shipping.”
“tiuecpeftcp3.w” acts as an API for all other modules of the system to communicate with the shipping module. One would expect this file to be used by a multitude of features and this is evidenced as true, given the output of the reuse perspective. Again this is evidence in support of the research questions posed in section 8.1.1.
This finding also raises the question “was there more than just 14 reused components?” Almost certainly the answer is yes. Reasons that the reuse perspective did not capture all of them could include:
•Not all the features may have been elucidated.
•The other reused elements may not be subject to the same type of reuse that the reuse perspective captures (refer back to the section on reuse types in section 3.3).
•It is possible that a test case did not correctly trace the right feature. For example, a test case may not have been correctly designed, therefore the test case may not actually have exhibited the desired feature.
From the analysis of the architect’s evaluation of the reuse perspective, 100% of the contents of the reuse perspective were marked as being reused. This result should come as no surprise since the formal definition of the set in section 7.1 is the set of reuse across features. Nonetheless, empirical evidence in support of that fact is comforting.
At the beginning of this section three specific evaluation questions were posed to be answered during the case study:
1. Are some elements of the reuse perspective generic (i.e. reusable)?
•64% of the elements of the reuse perspective were marked as generic. This suggests that more than just “some” of the contents are generic, but potentially a “large proportion” or even a “majority” are generic.
2. Are some elements of the reuse perspective core to the system (i.e. elements of the systems that could be considered framework, infrastructural or rarely changing)?
• 57% of the contents of the reuse perpective were marked as “core.” This result is similar to the result for the “generic” property. Adding further weight to this stream of evidence was the evaluators comment regarding the
“God class,” with reference to the fact that almost the entire contents of the reuse perspective belonged to a single file that represented the core gateway to that module.
3. Do all elements of the reuse perspective fall into at least one of the categories mentioned (reused, generic, core)?
•This question yielded a conclusive yes, with most of the contents having two or even three of these attributes.
However, the extent to which these results can be generalised to other systems is difficult to gauge. The ecological validity of the experiment is strong; The system was a real commercial system of substantial size. The use of the original architect provided an evaluation separate to the creator of the reuse perspective, thus introducing a degree of objectivity. However, as with ecologically valid studies of this nature (Basili, 1996), population size is small and control of variables is difficult (and of limited value, given that they typically generate one data point). Their value is in having high ecological validity and of richness in the data that invites and supports deeper analysis. In an effort to address these issues an evaluation of the reuse perspective is revisited during the AIM
case study in section 9.
In the section 7.2 we described a plausible technique for component encapsulation. This section takes the proposed technique and actuates it in an industrial setting under realistic circumstances, on a large commercial system. Two software engineers, who currently maintain that system volunteered as participants. They performed their component encapsulation tasks in their normal working environment, with the exception that the Reflexion tool plug-in (jRMTool) was installed on their IDE. Both the participants’ processes and products were subsequently evaluated to assess:
1. Did the process seem to support programmers in component encapsulation?
2. Do the participants adhere to the process, or can the case study reveal improvements/refinements to our guidelines?
3. Is the product of the process a valid component?
4. Is this a better means of component encapsulation than the current approach employed in the company?
It is important to note that this case study, unlike the previous study, does not examine the reuse perspective and examines the Reflexion Modelling-based portion of Reconn-exion only.
The subject system in this case study is the Learning Management System (LMS), currently being developed and maintained at IBM’s Dublin Software Laboratory, Ireland.
It provides collaborative services for administering eLearning to the employees of a company. Specifically, it makes learning content available to employees as courses, allowing them to enroll in those courses, tracking their progress through courses and reporting on mandatory training compliance.
The LMS is part of a larger application called “Workplace Collaborative Learning” (WCL). WCL adds collaborative features to the LMS that allows employees and instructors to discuss and share information and resources around their courses, and skills management features, which can be used to assist in the automatic provisioning
of eLearning to employees.
The current LMS was originally developed from a legacy stand-alone product that was altered to become part of WCL. The WCL in turn also forms part of yet a larger application called “Workplace Collaborative Services” (WCS) or simply Workplace. WCS aims to provide employees with a new way to perform their work, where all their
applications (email, calendaring, document management) are integrated with each other and with the other employees in the company through collaborative applications like instant messaging (IM), Web Conferencing and presence awareness. WCL is intended to be the part of Workplace that administers learning to the company’s employees through Workplace. The sub application, in toto, comprises of approximately half a million lines of code.
Two volunteer participants, who are developers of the LMS, participated in the study. Also, two architects of the LMS particpated as evaluators of the output produced by the participants. The details of both the particpants and the architects are summarised in table 8.5
Table 8.5: Workplace case study participant details.
The task of the first participant was to arrive at a better architectural understanding of one of the components of the system (in the work context of arriving at a better architectural understanding of the system). This was one of his work tasks, and although not his highest priority, the task was ecologically valid. The second participant’s task
was a legitimate maintenance task that the support team had tried before, but had cancelled on realising the associated difficulty (it was a GUI and business logic separation project). The participant also undertook some refactoring of the system, prompted by the technique.
To enable the case study to proceed we used the jRMTool reflexion modelling plug- in (Murphy, 2003), provided by Gail Murphy, for the Eclipse Java IDE (Eclipse IDE Homepage, 2005). This tool provides automated abilities for creating and viewing high-level models, for mapping software elements to high-level model elements, for
displaying the resulting Reflexion models and displaying summary information regarding the edges of the model and unmapped values.
A screenshot of the tool can be seen in figure 8.1. As can be seen in the figure, the “mappings” of the source code to the high level model are expressed in a basic XML language. Also note that by clicking on an edge the user gets a list of the relationships that make up that list (“edge information”). Also note, in the Reflexion Model in figure 8.1 unexpected edges in the model are displayed with a dashed line in accordance with the description of the technique in chapter 5. It is worth noting how much this tool was a natural fit for the participants. As an Eclipse plugin the tool integrated with the normal work tool of the participants.
Figure 8.1: A screenshot of the jRMTool eclipse plug-in.
The following protocol will be adopted for the study:
•An existing piece of work, that is also a suitable application of component recovery, will be assigned to the participant. This task will not necessarily be the same for each participant.
•The component encapsulation task will take place at the participant’s work desk and machine that the participant uses (i.e. his natural work environment). The situation will be kept as ecologically valid as possible as a result of this measure.
•The option to undertake his assigned task over several sessions will also be afforded to the participant. In this way the task will assume further ecological validity by making it easier for the task to slot in with the participant’s natural working schedule.
•Prior to the case study each participant will be trained in the use of the jRMTool tool.
•Once the case study begins each participant will perform the component encapsulation task in a separate session (to each other).
•The session will be be video or audio recorded, if possible. Furthermore, each iteration of the Reflexion Model generated will be saved by the user. It is hoped that this evidence, with other data sources, will be sufficient to answer process-based questions posed in section 8.2.1.
•The study will begin with the participant being instructed to think-aloud for the duration of the study. This instruction along with the chosen encapsulation task will be the only direction given.
•When the participant decides that the task is completed this portion of the study will finish.
•Next the encapsulated components will be presented to original architects of the system for evaluation.
•It is hoped that this portion of the evaluation will provide evidence to answer product-based research questions posed in 8.2.1. The degree of freedom given to the participant by this protocol is important to maintain the ecological validity of an in-vivo study (O’Brien et al., 2005; Basili, 1996).
8.2.7.1 Participant 1
The first participant chose to encapsulate a Helpers Data Transfer Objects component in the LMS. The participant was aware of this logical entity as a component and its use, but was not certain of its makeup or how it was connected in the system. The Helper objects are data transfer objects (DTOs, also known as Value Objects) that get passed between different layers of the system. The helper DTOs attempt to encapsulate some business object, such as a course or educational offering retrieved from or persisted to the underlying database or visual output. For instance, when a HTML form is submitted to the presentation layer, the java Struts components build a helper DTO and fill it with all the data from the form, before passing it on to the service layer for further processing. From there, the helper DTO may be passed to the persistence layer, or back up the stack again, as required. It is a tidier method of passing data around than having interfaces with loads of individual parameters, but does incur some overhead.
Using the Eclipse plug-in, described in the tool support section, the participant spent 2 hours in a single sitting encapsulating the Helpers DTO’s component, using our prescribed guidelines. Elements were added to the high-level model through ten model iterations, in the following sequence (Note that the steps of the proposed Reflexion-adapted process for each iteration are placed in square brackets.):
•Iteration 1: Added RestOfLearning and Helpers. [Steps 1 and 2]
•Iteration 2: Added Struts. [Step 3,4,5]
•Iteration 3: Replaced Struts with StrutsActionHanders, StrutsActionForms. [Step 3,4,5]
•Iteration 4: Added Services. [Step 3,4,5]
•Iteration 5: Replaced Services with ServiceInterfaces, ServiceImpls. [Step 3,4,5]
•Iteration 6: Added PersistenceMgrs. [Step 3,4,5]
•Iteration 7: Added JSPTags and WebserviceAPIs. [Step 3,4,5]
•Iteration 8: Added DSPersistenceBeans and RemoteProcessingCommands. [Step 3,4,5]
•Iteration 9: Added LMMPersistenceBeans. [Step 3,4,5]
•Iteration 10: No further additions -Finalised Map. [Step 6]
In creating his first Reflexion model (see point 1 above) the participant followed step one and two of the prescribed process by separating out the helper DTO component from the remainder of the system, based on the naming conventions of the code base. He felt, after this first model, that he had correctly identified the contents of the helper DTO component. Figure 8.2 shows the initial reflexion model and corresponding map. However, he did comment that he would not be absolutely certain of this until he separated out the “RestOfLearning” node.
Over the course of a further nine iterations 12 high-level elements were added to the model and two replacements were made, thus iterating through steps three, four and five of our process repeatedly. The models produced can be found in the appendices. Replacements were typically made in situations where a high-level element needed to be subdivided into further sub-architectures. For example, in iteration five, the participant replaced the services component with two components -one which provided interfaces to the services and another which provided the implementation of the services.
Figure 8.2: First iteration by the first participant.
By the ninth iteration in the process (figure 8.3) the participant was satisfied that important interactions with the helper DTO had been isolated, therefore identifying its many interfaces with the system. This was confirmed when the participant finalised his map in the tenth reflexion model, thus completing step five of our process.
8.2.7.2 Participant 2
The second participant chose to encapsulate the web administration user interface (UI) of the LMS. This component forms part of a larger web interface component. The web component is the collection of Java classes that comprises the front end of the legacy web application of LMS, prior to its integration with Workplace (LMS was originally a standalone J2EE-based web application but it had since been integrated into Workplace.). As the LMS is now being integrated into Workplace, it was important to partition the system into its legacy web administration element and the rest of the system (for inclusion into workplace). Hence this was an outstanding task that the participant had been assigned.
The participant proceeded to encapsulate the web administration UI component over the course of 11 iterations. Elements were added to the high-level model in the following sequence, finishing with the model in figure 8.4:
Figure 8.3: Last iteration by the first participant.
•Iteration 1: [Steps 1 and 2]
–The web administration UI component -Added Delivery, Servelet, Actions.
–Remainder of system -Added RestOfSystem.
•Iteration 2: The web administration UI component -Added View. [Steps 3,4,5]
•Iteration 3: The web administration UI component -Added OddBalls. [Steps3,4,5]
•Iteration 4: The web administration UI component -Removed OddBalls, Added Navigation. [Steps 3,4,5]
•Iteration 5: The web administration UI component -Removed Navigation, Added WebUtil. [Steps 3,4,5]
•Iteration 6-9: The high-level model remained the same for these iterations. The participant only made alterations to the map over the course of several refactorings to the code base. [Steps 3,4,5]
•Iteration 10: The web administration UI component -Removed Delivery. [Steps 3,4,5]
•Iteration 11: The web administration UI component -Removed WebUtil. [Step 6]
The participant immediately subdivided the web administration UI component (in his first iteration) into four major sub components, again highlighting the need for flexibility during the process. Also note that in this particular case (because it is a UI separation task) the participant decided that there was a single interface, therefore removing
the need to split up the RestOfSystem component. This took five minutes.
Figure 8.4: Web administration user interface component. (Note the bolded black box is not part of the tool’s visual output.)
Based on previous domain knowledge, several more sub components of the web administration UI component were added and deleted over the course of the 11 iterations. In iteration three the OddBalls component was added by the participant to hold software elements that he had yet to find a place for. However, as it transpired, the participant never had a need to use this placeholder. The high-level model stabilised in iterations six through nine. At this point only changes to the map were made as refactorings to the system were implemented. After the eleventh iteration, the participant was confident that he had encapsulated the web administration UI component and successfully decoupled it from the system, both in the logical reflexion model and through the associated
refactorings he made in the code of the system. The participant chose only to identify a single interface with the rest of the system. Again, the participant did not adhere strictly to the prescribed process and instead demonstrated some potential improvements. This observation addresses research question 2 in section 8.2.1.
We are interested in evaluating two aspects of our technique:
1. The process.
2. The product as a result of the process.
In evaluating the process we used as our data:
•Concurrent think-aloud [TA] generated by the participants as they performed their task. The participants were asked to state everything that came into their mind as they proceeded. This is in line with best practice use of the technique (Ericsson and Simon, 1993). While speaking aloud is not the most natural way for a person to behave, it has been shown to provide the richest insight into the mental state of the participant at a point in time (Russo et al., 1989), when the information gathered is concurrent, unfiltered and comments on state only, not process.
•Diaries [D] kept by the participants as they proceeded with the process which contained their thoughts on the tool, the technique and details of each iteration of the process.
•Interviews [I] with the participants after the case study.
•A fourth stream of evidence is found in the form of contextual knowledge [CK]. We define contextual knowledge, in this instance, as knowledge we had from the LMS team about the process and its impact. This is explored in greater detail in the section entitled “Contextual Knowledge.”
In evaluating the product we relied on data generated by an architect of the system. This architect viewed the models produced by both participants and evaluated them while being recorded by an MP3 player. In addition, a complimentary analysis of the encapsulated component’s quality was attained by applying coupling and cohesion
metrics to the product, as suggested in section 2.5.2.
Apart from the coupling and cohesion metrics, the methods reported here will not be suited to quantitative, statistical analysis due to the qualitative nature of the data gathered. Indeed, given the high ecological validity in this in-vivo study (Basili, 1996) and the range of uncontrolled variables that this suggests, this evaluation work is more in line with the approach suggested by Seaman (Seaman, 1999) or O’Brien (O’Brien et al., 2005) in providing qualitative evidence of high ecological validity to complement other studies of a quantitative nature.
8.2.8.1 Process -Did the process seem to support programmers in component encapsulation and do the participants adhere to the process, or can the case study reveal improvement to our guidelines? (Question 1 and 2)
The evidence gathered in this section shows support in favour of the first and second research questions in section 8.2.1. Both the participants found the process very helpful in identifying the component abstractions and stated so on several occasions,
“That’s interesting, looks like this package should be catered for in another node.” [referring to view node] [TA]
“Reflexion great for spotting where dependencies were nonetheless . . . ”
[D]
“That’s fairly revealing actually.” [TA]
“A really good tool for getting a high-level idea of a big amount of source code.” [I]
“I certainly intend to continue to use it myself.” [I]
By the end of the process the participant 2 even self-evaluated his own results:
“Delivery is back with the ‘RestOfSystem.’ No unwanted links to RestOfSystem, job done basically . . . Now I have just what I always wanted: the LMS doesn’t even know the web component exists. This is how it should be. It’s good to know that the list of modifications is actually quite short(summed up in 7 bullet points in the notes) which I’m taking as a good sign. The two aren’t nearly as highly interwoven as I had feared, or as I thought from having
attempted this separation task a few months back. There’s been a few hacks along the way, but Reflexion allowed me to see the big picture, safe in the knowledge that this was, at least, the minimum number of hacks in the just the right places.” [D]
In a previous attempt to isolate the web administration UI component participant 2 had relied upon compiler errors generated as a result of changes he manually introduced to the source code when trying to isolate the component. However, he noted that using our process changed all that for him. He could now work at the modelling level and not be faced with many hundreds of dependency errors after each compile when working at the source code level:
“[the] compiler is like a final check, reflexion should be your first check.”
For each participant, evidence of the value of the iterative “hypothesis > test with tool > new hypothesis” nature of the process is seen throughout the gathered data, with many comments such as,
“I’d be interested to know if the action forms are using helpers, because I’d totally expect the action handler to use them . . . [calculates reflexion model] so there are some calls . . . most of the calls from struts are going through
the action handlers . . . expected.”[TA]
“I would expect the service tier to use it, . . . so I’ll add another node . . . and I would be very, very surprised if there was an arc from the services to the helpers . . . it shouldn’t happen, these guys shouldn’t have any actual
processing logic . . . [calculates reflexion model] . . . so there are three calls from the helpers to the services, which suggests there might be a problem there, sounds like something is being violated there.” [TA]
“There is a call from helpers to struts . . . didn’t expect that . . . It’s always been unclear to me whether the validation occurs in the action handle or if it validates itself . . . ” [TA]
“‘Reflexion Model Information’ showed that I should include an extra package, ‘views,’ in my model, as part of the web component. This only moves the problem however, reducing the unwanted links to ‘Actions’ but adding
new ones into ‘Views’.” [D]
“It also looked like navigation needed to go into the web component. In fact it does but since its a component that requires initialisation from the LMS (and is not just a bunch of ‘dumb’ action classes) I think it belongs in the ‘RestOfSystem’ after all.” [D]
“maybe I’ll split services up into the implementation and interface, I’d be interested in knowing how it’s using helpers.” [TA]
“Looks like there’s a few links alright, from rest of system back to view, . . . I was hoping there would be none, so that’s a shame.” [TA]
Only one comment of warning was made regarding the process by the first participant. In this case he highlighted the importance of mnemonics in undertaking the process,
“You’d have a real problem doing this [process] if you didn’t have a con sistent naming scheme.”
Two worthwhile points are illustrated by this comment. Firstly, without a consistent naming scheme for a subject system, the map for the source model would become very large, since users of the jRMTool would not be able to catch a large set of software elements in a single regular expression. Secondly, if the naming scheme was inconsistent and the names held little clue as to meanings of the classes and methods then the participant would find the creation of the map very difficult (as it would not be in any way obvious as to which node a source code element should go). This suggests that there is scope for combining techniques other than mnemonics with our process as part of the component encapsulation process. In our proposal the alternate search technique
would be the software reconnaissance technique, which localises its search for elements of software components based upon the behavior of the program rather than the naming conventions of the source code (Wilde and Scully, 1995).
Also, while undertaking his encapsulation task, participant one deviated slightly from the proposed process by breaking up the RestOfSystem component before completely deciding on the contents of the component he was encapsulating. This suggests
that, in practice, a definite list of contents for the component to be encapsulated would not be decided upon until later iterations of the process. Thus a variant of the process here allows the participant to break down the encapsulated component or the “RestOfSystem” node further to become sure of its contents. This was also evidenced in observations of participant two. These observations are crucial in answering question two of the research questions in section 8.2.1 (Do the participant adhere to the process, or can the participants reveal improvements to the guidelines?). The participant did not stick rigidly to the prescribed process highlighting the need for additional flexibility. This additional flexibility brings the encapsulation process closer to the original Reflexion Modelling process.
8.2.8.2 Product: Architect Assessments -Is the product of the process a valid component? (Question 3)
In order to gather evidence in support of the third research question, which assesses the product of the process, we presented the resultant reflexion models to an architect of the LMS in order to gain his evaluation of the component abstractions produced. This evaluation concentrated on:
•The quality of the component.
•The correctness of the mappings to the component.
The architect agreed with the component encapsulated by participant 1 and the mappings made to it, with the exception of one disagreement over the pattern to which the Helpers DTO component belonged to. In figure 8.5, we see a portion of participant one’s final model. It was the hypothesis of the participant that communication from
JSPTags to DBPersistence beans was a breach of a design pattern implemented in the system and that all communication should be routed through the Helpers component.
The architect initially saw the model as currently implementing a data transfer assembly pattern which permits this type of communication. After further examination of the Reflexion model and discussion with the participant, the architect amended his view saying,
“looks like helpers are being relied upon to do any manipulation.”[I]
“Seems to be doing two patterns . . . object assembler and view helper pat tern.”[I]
He summed up his review of the component stating that the component and interfaces
were,
“valid and interesting . . . I would have never thought of this [helpers] as a view holder. Didn’t know the extent of use.”[I]
“This is not just [Participant 1’s] perception . . . it is quite a common pattern.”[I]
Indeed, the architect noted that 90% of the calls to the DB persistance beans were through the helpers DTO component showing a strong indication of the view helper pattern. This prompted a validating review by a second architect of the LMS, who was an original architect of the system. He indicated that the actual interpretation was a
mixture of both the first architect’s and the participant’s opinions. The helpers component is for gathering information for the UI (like a View Helper pattern), but the UI is not necessarily restricted to this. More importantly, the architect also agreed with the participant’s model and mappings produced.
For participant 2’s web administration UI component decoupling task the first architect completely agreed with the high-level model and the contents of his model. Not only could he appreciate the validity of the component encapsulation and its interface but the motivation also. Once correctly encapsulated he could see that:
Figure 8.5: A design pattern in the LMS.
“all communication could potentially happen over a webservice.” [With the refactored system] [I] ,which was the ultimate intention of participant 2 too. With respect to the correctness of the model he, the architect, stated,
“there is unlikely to be missing of extra edges here because we’ve done the work . . . that is what he wanted it to be . . . that’s good, well done [Participant 2] . . . there’s not an awful lot else to say . . . it’s right.” [I]
To summarise, the qualitative evidence gathered through interviews with the original architects of the LMS would seem to lend support to the third research question, which questions the quality of component recovered.
8.2.8.3 Product: Metrics of Coupling and Cohesion -Is the product of the process a valid component (Question 3)
In section 2.5.2 metrics for coupling and cohesion were described. With, encapsulation, and thus replacibility in mind, we calculated the coupling and cohesion for the components encapsulated by our participants. This provides another, more objective stream of evidence in support of the third research question for this study. We presented the scales of coupling and cohesion to both participants and asked them to assess their respective components. Participant 1 had trouble stating exactly which type of coupling his component exhibited,
“Certainly it is not 0; x and y do communicate. However I don’t know if Helpers really communicates with the outside system (calls out) in any meaningful way. In my opinion, it could be 1 or 2.” [I]
We decided to remain conservative in our calculations and accepted a coupling value of two (participant 1’s worst case coupling) for the helpers DTO component’s coupling with the rest of the system. The following coupling measure for the helpers component was calculated:
Participant two faced a similar dilemma to participant one when assessing the web administration UI component on the coupling scale,
“Coupling between the ‘rest of system’ (ROS) and the web module is either 2 or 3 . . . probably 3, since there is, for example, sometimes some (very very basic) business logic contained in the web module. There’s a subtle
difference (parameters as flags) between 2 and 3 . . . but I think it’s closer to 3.” [I]
Again we remained conservative and calculated the coupling for the web administration UI using a coupling value of 3 from the scale:
Based upon Fenton’s studies, values above 4 are tightly coupled and values below 2 are loosely coupled (Fenton, 1991). Both components in this case lie below the tightly coupled threshold. We can compare this to the self coupling of the components. Participant 1 assessed the self coupling of the helpers DTO component as a four on the coupling scale,
“I would probably concur that it is level 4 coupling -same global data.” [I] which works out to be,
Participant 2 stated that the internal coupling measure for his component also resided at level four on the scale:
This suggests that the component itself is tightly coupled internally. As intra component coupling is higher than inter component coupling in both cases, this suggests that the components have some coupling / cohesion merit.
In assessing the cohesion of the components the first participant noted the type of cohesion exhibited by the helpers DTO component was communicational cohesion,
“I would actually say that Helpers exhibits cohesion level 3 -Communicational. Based on the difficulty [Architect] and I had on agreeing on the true role of the Helpers component, I certainly don’t think it could be said to have a single well defined function (level 1 -Functional). In fact, both [Architect] and I think that it is used as both a DTO (Data Transfer Object or Value Object) and as a view helper. But in the case of either function, the same body of data is acted upon. I think this qualifies it as communicational. (Though the original intent may have been for it to perform a single well defined function, probably DTO or model bean.)” [I]
This suggests reasonably good cohesion for the helpers component. Participant 2 assessed the web administration UI component as having functional cohesion,
“For functional cohesiveness, yes 1 (functional cohesion) is appropriate I think.” [I]
This suggests an extremely high cohesion measure for the web administration UI.
8.2.8.4 Contextual Knowledge: Is this a better means of component encapsulation than the current approach in the company? (Question 4)
For us to make the statement that our process is “better” or “useful,” our process must be “better than” or “useful compared to” something else. Unfortunately, this is difficult to establish definitely using industrial case studies. Contextual knowledge, which is existing data from projects and systems prior to the beginning of the study, allows us to make this comparison.
The most obvious contextual evidence supporting the benefits of this approach over current practice has emerged primarily from subjective statements about the process. For example, in reference to his normal, ad hoc, approach to partitioning the system into components, the first participant said,
“Normally it would take so long because you only get to see small pieces of the code when developing.” [I]
Another example from the first participant’s think-aloud was when he had hypothesised no dependency between helpers and struts when in fact there was. When asked if he thought that this was unexpected he said:
“yeah, yeah I do . . . I wrote that code myself.” [TA]
The fact that he was only made aware of the dependency when using our approach, even in code he wrote himself, is evidence that our approach or the Reflexion model can improve current practice. Another example from the same model arose where validation services for the Helper DTO’s was revealed to the participant. Using his traditional methods he had said that
“It was always unclear where this type of validation occurred.” [TA]
Only with the help of the process was he now able to explicitly document this interface, by identifying the dependencies the component had with the remainder of the system.
Further contextual evidence can be identified for our second participant. A previous project had attempted the same task over the course of two weeks to encapsulate the web administration UI component and was cancelled.
“i did try this same job about two months ago and gave up after two weeks.” [TA]
With the variation on Reflexion, the same task was accomplished by our second participant in four hours (0.5 days). Using the original attempt as a base, our technique allowed the encapsulation and refactoring of the same component 17.5 times faster than the approach normally used. In the organisation it appeared that the standard means of carrying out this kind of refactoring separation of components was by repeatedly making changes and checking subsequent compile errors. Using Reflexion appeared to be far better solution,
“The alternative to this, maybe, is to make the changes to the code. But if I could use the model ... again, not touching the compiler, until I really have to. I could gradually put stuff into this oddball node. Then at the end
when I wanna start actually changing code you can, am, actually start the refactoring proper and keep rebuilding my reflexion model, and eventually all the stuff in my odd ball node will go down to zero and I’ll have no stray links, so I’m gonna try that.” [TA]
Importantly, it should be noted that this evidence shows that the technique is better that state-of-the-practice for the participant not the state-of-the-art in general.
An even better solution would be to create automatic refactorings, triggered by the Reflexion model. Work underway at the University of Bremen is currently investigating this goal (Christl et al., 2005).
As a final stream of contextual knowledge, the two participants stated that they would actively use the technique during their ongoing work. The architects who assessed the product of the process also expressed an interest in using the approach. Four more programmers in the department have also requested to use the technique as a result of the review provided by the participant and another architect/developer is actively using the approach in his ongoing work. This seems to indicate proliferation of the technique in the organisation.
Close to the beginning of this section, four evaluation questions were posed to be evaluated by this case study:
1. Did the process seem to support programmers in component encapsulation?
•The documented think-aloud, interviews and diaries all seem to indicate that the process was directly contributing to the programmers ability to encapsulate their respective components. Participant 2 was even able to encapsulate a component that could not be encapsulated using his standard practice. In addition, the two participants stated how they would actively use the technique during their ongoing work and the architects who assessed the product of the process also expressed an interest in using the approach. Four more programmers in the department have also requested to use the technique and another architect/developer is actively using the approach in his ongoing work. This seems to indicate technology transfer of the technique in the organisation.
2. Do the participants adhere to the process, or can the case study reveal improvements
to our guidelines?
•One useful change became apparent during the first study. We later incorporated the change when undertaking the second study. It involved allowing the participant to break down the component under scrutiny into sub architecturesto enhance his certainty that the contents of his chosen component is correct and to help break down the interfaces.
3. Is the product of the process a valid component?
•We assessed the products of both studies using two independent architects of the system and an objective measure of the products (components) using software metrics of coupling and cohesion was applied. Both assessments indicated that the components were valid.
4. Is this a better means of component encapsulation than the current approach in
the company?
•From the evidence we gathered it appears so. Compared to the contextual data gathered the encapsulation guidelines allowed the encapsulation of components up to 17.5 times faster, to reach the same level of usefulness,
than with the traditional approach within the company. Of course, this is only a single quantitative data point, however other qualitative contextual data gathered also suggests that the process is better. The evaluation indicates that our process afforded significant benefits to the participants. The ecological validity of this study is also quite high. The industry setting, a large commercial system, actual maintainers, objective metric measures and original architects for assessment made for a highly ecologically valid case study, with many streams of evidence to support the evaluation. These streams included, think-aloud, diaries, metrics, interviews and original architects. A common shortcoming of experiments of this nature is the small population size. This is partially tackled because two participants were used in this case study, as opposed to one. However this still falls quite short of a valid population size. As a result, a statistical analysis was not possible, hence the strong bias in the usage of rich qualitative data. Furthermore, the reliability of these results will be further assessed as two further participants are added in the next case study, where the entire Reconn-exion process is examined.
[Back to Home Page] [Previous Chapter] [Next Chapter]
Component Reconn-exion by Andrew Le Gear 2006