Chapter 9 Evaluating Reconn-exion: The AIM Case Study
A Ph.D. Thesis by Andrew Le Gear
[Back to Home Page] [Previous Chapter] [Next Chapter]
“A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools.”
-Douglas Adams.
The purpose of this case study is to examine the entire Reconn-exion process as a single unit. Central to the combined process is whether the reuse perspective is a useful means of prompting the initial Reflexion models of the user, and this is an evaluation not present in the previous case studies. The following evaluation questions are thus examined by this case study:
1. Does the reuse perspective influence the prompting of component abstractions and subsequently influence iterations of Reflexion models?
2. Do the participants adhere to the process, or can the case study reveal improvements to the guidelines?
3. Do participants find the process useful?
4. Is the produced component of high quality?
5. Is this a better means of encapsulating reusable components than the current approach in the company?
The subject system in this case study is a warehousing management application called Advanced Inventory Management or AIM, and is a commercial product of QAD Ltd. (Ltd., 2006). AIM as an application is used as an extension to MFGPRO, which is an enterprise resource planning application and the flagship product of QAD. AIM consists of approximately 250KLOC and over 2572 procedure files. The application is used to provide warehousing, picking and placing algorithms for large storage spaces, as a feed to a manufacturing environment. The application was acquired by QAD Ltd. and more recently the responsibility for maintenance was shifted to QAD Ltd.’s Irish development facility. As a result several software developers, who had little or no experience with
the application (but had experience in the domain) were faced with the task of learning AIM and maintaining it.
Table 9.1: Participant details.
Two participants demonstrated the Reconn-exion process in two different sessions. Their details are summarised in table 9.1.
Also, an original architect of the AIM application also took part in the study in order to assess the output of the Reconn-exion process. The architect had worked on the application since it had been acquired by QAD Ltd. four years previously.
Tool support necessary to enable Reconn-exion for this study included:
•The Progress 4GL profiling option. This was enabled to create code coverage profiles from the running code (Progress Software, 2003).
•An open source tool, developed by the author at the University of Limerick called “ReconCalc”: This was used to calculate the results of the Software Reconnaissance from the gathered profiles (Le Gear, 2006).
•A static analysis tool called “XREF” (Progress Software, 2003) was used to derive the source model necessary to perform Reflexion Modelling.
• A Perl-based analysis tool call “RMTool”: This was used to calculate the Results of Reflexion Modelling (Murphy, 2003).
•A graph layout and display tool call GraphViz: This was used to display the results of Reflexion Modelling to the user (AT and T, 2005).
The protocol for the study is planned to execute as follows:
•Prior to a monitored session with the participant, the participant will choose features of the application to analyse.
•The person running the study will profile the chosen features and calculate the reuse perspective.
•In the company the following situation is used:
–The participant’s own computer and workplace (i.e. the participant’s natural working environment).
–The AIM source code, the RMTool and the reuse perspective are open and ready to browse or use on the computer.
–A video camera is set up to record the study.
•The participant will first be invited to browse the reuse perspective to see if it may prompt component abstractions to him. The participant will be asked to talk aloud throughout the experiment.
•If the reuse perspective prompts a component abstraction to him, he will sketch the initial software elements belonging to the component(s) and places them in an initial map file and computes a Relflexion Model.
•The participant will then continue to add to and refine his Reflexion Model until both the component and the roles it plays in the systems are captured by the Reflexion Model. Each iteration of the Reflexion Model, map and high-level model are saved for later perusal.
•When the participant decides that he is finished he will inform the study supervisor and is free to go.
•The time will be noted and recordings and saved model stored for analysis later.
•Next, the resulting encapsulated components will be presented to the original architects of the system for appraisal.
This section describes how the case study unfolded with respect to the protocol of the study and the Reconn-exion process. In particular the section can help to answer the following research questions posed in section 9.1:
•Do the participants adhere to the process, or can the case study reveal improvements to the guidelines?
Table 9.2: Feature set examined by participant 1 during case study.
9.6.1.1 Creating the reuse perspective
Participant 1 identified a rich feature set totaling twenty one features. The maintenance agenda of the AIM evolution team was used as a seed set (Robillard and Murphy, 2002) in this instance. Table 9.2 lists these features.
Using the profiling option in Progress4GL (Progress Software, 2003), coverage profiles for one or more test cases exhibiting each of the features (table 9.2) were retrieved. Using tool automation developed by the author at the University of Limerick, the reuse perspective was calculated. A reuse perspective 99 procedures in size was produced (Note: The listing of the reuse perspective cannot be included due to copyright reasons).
9.6.1.2 Using the reuse perspective to prompt component abstractions
The participant was first asked if, without a prompt, he could identify any components of AIM. The participant could not. The participant was then presented with the reuse perspective and asked to sketch components of the system, prompted by the list of software elements in the reuse perspective. The reuse perspective served as a sufficient prompt and allowed the participant to sketch what he considered worthwhile components, as shown in figure 9.1. In the sketch, consecutive lines represent a cluster. The participant circled the best candidate clusters.
9.6.1.3 Encapsulation with Reflexion
Based upon the components prompted to the participant by the reuse perspective, the participant chose to encapsulate a utility component called ‘AIM_Util.’ The participant created a simple high-level model consisting of ‘AIM_Util’ and the remainder of the system, ‘RestofAIM.’ The participant mapped all procedures with the naming convention ‘whgp*’ to ‘AIM_Util’ and everthing else to ‘RestofAIM’ and calculated the Reflexion model. The result of this is in figure 9.2.
Upon examining the resulting Reflexion model the participant was satisfied that he had encapsulated the utilities component.
9.6.1.4 Identifying Multiple Interfaces
The participant next began to divide out the ’RestofAIM’ component in order to reveal the multiple interfaces that the component had. Over the course of nine further iterations, nodes were added to the model (dividing out RestOfAIM) in the following order:
Figure 9.1: Procedure clusters prompted by the reuse perspective (participant 1). File names are blurred for copyright reasons.
Figure 9.2: Participant one’s first Reflexion model and map.
•Iteration 1: Added AIM_Util and RestofAIM. [Steps 1 and 2]
•Iteration 2: Added Engine. [Steps 3,4 and 5]
•Iteration 3: Added UIBrowsers. [Steps 3,4 and 5]
•Iteration 4: No changes to the model only changes to the mappings were made. [Steps 3,4 and 5]
•Iteration 5: No changes to the model only changes to the mappings were made. [Steps 3,4 and 5]
•Iteration 6: Added AIMDB_Triggers. [Steps 3,4 and 5]
•Iteration 7: Added Printing and Algorithms. [Steps 3,4 and 5]
•Iteration 8: Added Reporting. [Steps 3,4 and 5]
•Iteration 9: Added MFGPRO and ControlFiles. [Steps 3,4 and 5]
•Iteration 10: Added RFClient, Picking, DOSessionTriggers, InventoryIC, OrderMGT and Item. [Step 6]
By the tenth iteration the participant felt he had adequately encapsulated the utilities component and identified the important interfaces it had with the remainder of the system. Not all the other components divided from RestofAIM represented a new interface. For example, the UIBrowses component had no communication with AIM_Util,
therefore AIM_Util had no role to play in services UIBrowses. The final Reflexion model of participant one is in figure 9.3. To view the other models produced, please refer to appendix E.
Figure 9.3: Participant one’s final Reflexion model.
9.6 Recounting the Process
9.6.2 Participant 2
Based upon a request from the participant, the source model for this study not only included
call dependencies, but also database dependencies. For more information about
alternate source models please refer section 11.11.
9.6.2.1 Creating the reuse perspective
The participant identified 13 features in the AIM application and coverage profiles were
retrieved in the same manner as the first participant. The participant’s own maintenance
agenda was used to seed the choice of features, and this feature set differed significantly
for participant 2. These are shown in table 9.3.
The participant in conjunction with the study coordinator then proceeded to gather
profiles for each of the features. The reuse perspective was calculated using ReconCalc
and resulted in a set of 58 procedures2.
9.6.2.2 Using the reuse perspective to prompt component abstractions
The participant, when asked, could not initially think of any components and their contents
in the application without a prompt. The reuse perspective was then presented
to the participant where he was asked to examine it and attempt sketching component
clusters from AIM, prompted by viewing the reuse perspective. The participant’s sketch
is shown in figure 9.4. Each circled regular expression or list represents the contents of
a component. The name of the component is written either above or below the component.
2Note: The listing cannot be included in the appendix due to copyright reasons.
Table 9.3: The set of features chosen by the second participant.
Figure 9.4: Procedure clusters prompted by the reuse perspective (participant
2). Note that the file names are blurred for copyright reasons.
9.6.2.3 Encapsulation with Reflexion
The participant, upon examining the reuse perspective and resulting sketch, chose to encapsulate the ‘RF_Screens’ component. This component was an alternate interface to the AIM system for people using the system in conjunction with a radio frequency gun. For example, as deliveries arrive at the wareshouse an employee with a scanner can scan palates as they enter the warehouse, instead of the normal access via the main
computer.
The participant spent the first three iterations encapsulating this component, using a simple model consisting of ‘RF_SCREENS’ and ‘EVERYTHINGELSE.’ In the participant’s view the ‘RF_SCREENS’ component should exist with a one way interface on ‘EVERYTHINGELSE.’ In this way ‘EVERYTHINGELSE’ is unaware of the existence of the ‘RF_SCREENS’ component and the ‘RF_SCREENS’ component is purely a user of the services of ‘EVERYTHINGELSE,’ similar to a pull model for user interfaces / system interaction. By the third model the participant had finalised this encapsulation as shown in figure 9.5.
To see other Reflexion models please refer to appendix F.
9.6.2.4 Identifying Multiple Interfaces
The participant next began to divide the ‘EVERYTHINGELSE’ node in order to identify the multiple interfaces for the ‘RF_SCREENS’ component. Six further iterations ensued after iteration 3, adding and deleting nodes in the following order:
•Iteration 1: Added RF_SCREENS and EVERYTHINGELSE.
•Iteration 2: Only changes to the map were made.
•Iteration 3: Only changes to the map were made.
•Iteration 4: Added WAVE, MFG_DATA and AIM_DATA.
•Iteration 5: Added CONF and MENU (removed WAVE).
•Iteration 6: The list is too numerous to mention but, during this iteration the participant experimented, by creating a node for every table in the database. (Removed AIM_DATA and MFG_DATA.).
•Iteration 7: Only changes to the map were made.
•Iteration 8: The nodes added during iteration six were conglomerated. Added AIM_RF_DATA, MFG_DATA, AIM_DATA, AIM_SEQUENCE, AIM_RF_SEQUENCE. (Removed CONF).
•Iteration 9: Added START, CONF, RF_TASKS, RF_BPICK and RF_CONT. During the final iteration the participant, began to divide up the ‘RF_SCREENS’ component to possible identify a subarchitecture. This subarchitecture is represented ‘RF_SCREENS’, ‘RF_TASKS’, ‘RF_BPICK’ and ‘RF_CONT’. This creation of a subarchitecture is similar to the behavior of participant 2 in the Workplace case study in the previous chapter. This is an example of a deviation from the process, addressed in research question 2. This final Reflexion model is shown in figure 9.6.
Figure 9.5: Participant two’s third Reflexion model.
Figure 9.6: Participant two’s final Reflexion model.
As with previous case studies, several streams of evidence were used to assess Reconnexion.The results were examined from the point of view of both the process and the product of that process. Data sources included, diaries, think-aloud, interviews, videoed observations, email conversation and metrics. The opportunity was also taken to revisit the evaluation of the reuse perspective.
This section presents evidence regarding the following research question from section 9.1:
•Does the reuse perspective influence the prompting of component abstractions and subsequently influence iterations of Reflexion models? Both participants were first asked, without a prompt, could they identify reusable
components of the system. Both replied they could not, and it was not until prompted with the reuse perspective that they could identify components. This suggests that the prompt is relevant to the task.
In recovering components, one of the core criteria is that the resultant components are reusable. Several quotes here show that the software engineer anticipated that the components would be reusable,
“. . . you now know that if I am doing some new functionality that I should be using that [he said pointing to a reusable element in set] . . . you reuse what should be reused.”
“here’s thirteen files that are doing a lot of generic work . . . and the next stage would be ‘lets go and see what going on here’ . . . you are getting some information on what I am working with . . . understanding what services are provided.”
The original architect of the system was also consulted and confirmed that the reuse perspective contained core and reusable elements of the AIM system:
“Absolutely agree. Some of the procedures in the list are the most important procedures in AIM.”
“Yes. Some of them are key in AIM.”
”Yes indeed. Radio Frequency, Picking, AIM creation tasks, Wave Functionality.” [with reference to their reusability]
The benefits of clustering core functionality by the view was also noted,
“. . . you are building what are the core players.”
“xijd*, i think they’re fairly core.”
and several comments were made highlighting how participants actively clustered the reuse perspective elements into clusters,
“All of the trigger code to do with deletes and writes is in that block up to.”
“the ‘fo’ programs are all to do with the ‘engine W’ table.”
“‘tpqlbmm’ and ‘ng’, I know they’re part of a separate application.”
“the last four, they’re to do with ‘wave planning’.”
Some of the best evidence with respect to the reuse perspective as an effective prompt to component abstractions are the component sketches produced as a result of viewing the reuse perspective. These are shown earlier in figures 9.1 and 9.4. The sketches were also presented to an original architect of the system for evaluation,
“I agree with [Participant 1] comments. Obviously there are a large amount of programs involves in those categories but I could see that it is the right way to classify these programs.”
The prompt provided by the reuse perspective also had a reaching influence through many iterations of Reflexion model, not simply the first. For example, the first participant continued to refer to the reuse perspective for the first nine Reflexion model iterations.
This section provides evidence regarding the following research question from section 9.1:
•Do participants find the process useful? It should be noted that the value of the reuse perspective portion of the process is implicit in the quotes from the previous subsection, thus the majority of quotes here refer to the Reflexion portion of Reconn-exion.
As previously mentioned, both participants were videoed talking aloud as they performed the case study. This provided a wealth of evidence that demonstrates the success of the process as a means of component encapsulation. Throughout the case study many statements can be cited where we can actively see the process support of Reconn-exion contributing to the encapsulation of components. These are some examples (Note: File names have been obfuscated for copyright reasons):
“We’ve got 19 calls between aim_util and ‘rest of aim,’ which is interesting, right, we need to figure that out because . . . we’d need to find out what those 19 things are going back here, because they theoretically should belong in there [pointing to ‘aim_util’] as well. Because you’d make a call to utility,
it should really be encapsulated, . . . give you back an answer, . . . unless its a call back thing.”
“Let’s put all these engine files, let’s put all these ‘when’ files somewhere, . . . so let’s create a thing [high level node] with engine in it.”
“UIbrowsers, there’s no way it should be calling back, they’ll all call it, . . . actually engine mightn’t call it . . . [calculates Reflexion model] . . . so, yeah look there’s no call back between engine and browsers. There’s one
call between AIM_Util and browsers . . . <EXPLETIVE> thats interesting, we gotta find out why that is.”
“So he’s making one call to aim_util. We need to find out what that is, OK? . . . basically this belongs in the rest of aim, actually, yunno what we’ll do?, we’ll put all that into a triggers [node].”
“I think that there shouldn’t be. The RF should be . . . it should go from the RF to the rest, I don’t see the RF going back into the rest. It could happen but I don’t think and if it does it should be small. [computes Reflexion
model] now that’s interesting. Pretty much on the mark about the one way direction. There’s only four going back into the RF_SCREENS, curious to know what they would be [checks edge information] OK, this makes sense. These make sense. These two programs are actually still RF, they’re exceptions to the rule about the ‘xisg’ naming convention. I’ve seen them before. So that actually does make sense. In actual fact to clear that up
we’d need to add those to that group [RF_SCREENS] as well . . . That’s nice actually. There’s sort of a -you can clearly see that there’s definitely only one way.”
“So there’s one [call back from RF_SCREENS], that’s interesting, let’s have a look and see what that is [examines edge] OK yes there was one other, . . . so we’re gradually pulling them all in [in to the RF_SCREENS component]. The next step would be to pull that one in and see if that clears it up.”
The participants also openly commented on the usefulness of Reconn-exion, “<EXPLETIVE> this is great craic ! very powerful, I must admit!” (“Craic” is an Irish colloquialism meaning good fun, worthwhile or of value.)
“Gee, that’s an interesting tool you have here, I must say.”
As before, this is only a fraction of the available think-aloud in support of the process.
This section provides evidence regarding the following research question from section 9.1:
•Is the produced component of high quality?
To gain an independent assessment of the product of Reconn-exion (the component) an original architect of the system, situated in Spain, was consulted (the same architect from the previous section). Due to the distance factor this assessment was carried out over an e-mail dialogue. The architect was posed the following questions with respect to the product of Reconn-exion, for both participants:
“Encapsulating a component: I have included a list of procedures from AIM that the participant grouped together. He felt that these files could be grouped as a [Component type] in AIM. I have two questions:
1: do you agree that these files constitute a [Component type]? In your opinion is anything omitted?
2: The participant grouped the files together on the premise that they form a cohesive and independent component of the AIM system. Do you agree?”
In both cases a positive response was attained:
“If you are just talking about procedures (*.p) I could say that in this list are all the important ones.”
“Yes. RF functionality is a very independent module from a programming point of view.”
With respect to the multiple interfaces for the utilities component encapsulated by the first participant, the architect agreed with the interface categories chosen saying,
“Yes indeed.”
This section provides evidence regarding the following research question from section 9.1:
• Is the produced component of high quality?
This section provides another evaluation of the components produced by the participants, this time using software metrics to assess quality. As with the previous case study, metrics of coupling and cohesion, described in section 2.5.2 are used to assess the components encapsulated using Reconn-exion. The first participant assessed the
utilities component as having a coupling rating of 3. The resulting coupling measure then works out as
Likewise the second participant assessed his component as being coupled at level 4 coupling, providing a coupling measure of,
With respect to cohesion the participants assessed their components as having a cohesion measure of 7 for the utilities component and a measure of 3 for the RF_SCREENS component.
Fenton states that components with a coupling measure above 4 are tightly coupled. The utilities component appears not to be tightly coupled, but based on the cohesion measure does not perform a single well defined function. That is, it’s cohesion is coincidental. This, however, may be considered satisfactory considering the purpose of the component. As a utilities component it will provide many services to the remainder of the system, similar to a library of reusable routines. Therefore, one could say that the component is of reasonable quality from a metrics point of view.
The RF_SCREENS component appears to be tightly coupled to the system because it operates upon the same global data as the rest of the system. This may prevent reuse. However, within the context of the system, the component appears to be reasonably cohesive as a single radio frequency (RF) user interface provider. Therefore this component, though probably not reuseable beyond the context of this system, is a cohesive component within the system. Furthermore the unidirectional dependency between the RF_SCREENS component and the remainder of the system make the component ”pluggable” in AIM. That is, the remainder of the system is oblivious to the RF_SCREENS component’s and could potentially be replaced with any RF UI component without any disruption to the remainder of the system.
This section provides evidence regarding the following research question from section 9.1:
•Is this a better means of encapsulating reusable components than the current approach in the company?
Contextual knowledge was far more scarce than in the previous study, however some was gathered, enabling us to address the first research question for this study. Upon interviewing one of the participants in the organisation one of the standard practices for architectural recovery involved examining the call stack for various scenarios of
execution of the system. A drawback of this approach is the inability to ever get an overall, summarized view, as with Reconn-exion,
“The other way is top down, which is where you start tracing through the program, and you could spend weeks doing that and you might never get to where you want to be. Whereas here you can just look up and say ‘why is
this happening?’ and go in . . . its a different approach.”
As before, it should be noted that this evidence demonstrates that the technique is better than the state-of-the-practice for the participant and not the state-of-the-art in general.
According to the participants, Reconn-exion is far better that this approach and vocal comments of support were made throughout the case studies, for example, as reported earlier,
“<EXPLETIVE> this is great craic! very powerful, I must admit!”
Furthermore the process itself could be seen as actively contributing to the formation of components (see section 9.7.2).
Also, as noted earlier, each participant was asked to list components of AIM without a prompt prior to being presented with the reuse perspective. Neither could think of components without a prompt. The participants were asked if he could identify components in the AIM module without prompting from existing documentation. The
participants could not. Participant 1 replied:
“No, can’t off the top of my head.”
At the beginning of this section, several evaluation questions we proposed to be addressed by this case study:
Does the reuse perspective have an influence in prompting component abstractions and subsequently influence iterations of Reflexion models? From viewing the reuse perspectives generated, both participants were able to sketch 17 components (see figures 9.1 and 9.4). Neither were able to sketch potential components without the prompt. In each case one of these components was the basis for the targeted component refinement
using the variation on Reflexion Modelling. The architect of the system was able to confirm that the identified abstractions were correct. Video evidence from the case study also shows how both participants then based
their component choice and formation on the component abstractions sketched. With participant one, the reuse perspective continued to influence the resulting models and maps for a further eight iterations.
Do the participants adhere to the process, or can the case study reveal improvements to the guidelines? By and large the participant’s adhered to the process. One notable improvement emerged from the the first participant, which was to include database accesses in the source model to reveal component dependencies on the data model. The language being examined was Progress 4GL. In this language the database and language are tightly integrated, so much so that database fields are treated almost identically to variables. The importance of this was reflected in the improved source model. Also notice how participant one also broke his component down into a sub architecture similar to the participants from the previous chapter.
Do participants find the process useful? Each of the participants voiced their satisfaction with the process on a number of occasions. These were discusses in section 9.7.2. Is the produced component of high quality? Based upon the evaluations from both the original architect and the interpretation of the results in light of objective metrics, the components seem valid. The reusability of the utilities component was higher than the RF_SCREENS component, but both seemed cohesive within the system.
Is this a better means of component encapsulation than the current approach in the company? This, as always, is difficult to assess using case studies. Evidence to support this question was scarce, however some contextual knowledge did exist demonstrating that the technique was better than what already existed in the organisation as shown in section 9.7.5.
Do all elements of the reuse perspective fall into at least one of the categories, “reused,” “generic,” “core”)? The original architect was able to confirm these properties as correct.
[Back to Home Page] [Previous Chapter] [Next Chapter]
Component Reconn-exion by Andrew Le Gear 2006