Semantic-Enhanced Automatic Refinement of Architecture Recovery Results Using LLMs

Understanding the architecture is crucial for effectively maintaining and managing large software systems. However, discrepancies often exist between the designed and implemented architectures, which can pose significant risks. To identify these discrepancies, architects need to extract the architecture from the system implementation, which is both time-consuming and error-prone. To simplify this procedure, many automatic architecture recovery techniques have been developed. Yet, their accuracy is often limited. Architects must still invest significant effort in refining recovery results to ensure they accurately reflect the implemented architecture.

To reduce such manual effort, we introduce SemRef, a framework that combines LLMs with dependency analysis to automatically refine architectures recovered by existing architecture recovery tools. By leveraging the LLM's semantic understanding capabilities and integrating structural dependencies, SemRef enhances both the accuracy and the comprehension of recovered architectures. To evaluate SemRef, we tested on 9 projects with published ground-truth architectures and 10 state-of-the-art architecture recovery tools. 5 commonly used metrics are adopted to evaluate the effectiveness of SemRef. The result shows that SemRef improves the accuracy by 17.72% to 43.35% across different metrics, as measured by normalized gain. Specifically, when measured by MoJoFM and a2a_adj metric, SemRef increases accuracy by an average of 118.57% and 100.41%, respectively. Moreover, SemRef is highly scalable. It maintains stable performance across projects ranging from thousands to trillions of lines of code with the cost scale linearly with project size. Further, we test SemRef on various LLMs to demonstrate its generalizability across different models. Beyond improving accuracy, the integration of LLMs enables SemRef to provide a structured module hierarchy and hierarchical module summaries, which further enhance the comprehensibility of recovered architectures.

• RQ1 (Effectiveness Analysis): How effective is SemRef in increasing the accuracy of the SAR tools?

• RQ2 (Ablation Study): How does each part of SemRef contribute to the increment of accuracy?

• RQ3 (Cost Analysis): How many tokens are required to refine an architecture? How does it scale with project size?

• RQ4 (Model Comparison): How does using different LLMs affect the performance of SemRef?

• We propose SemRef, the first LLM-based SAR result refinement framework that integrates dependency analysis with the semantic understanding of LLMs.

• We evaluate SemRef using 10 existing techniques across 9 projects with known ground truth architectures. 5 metrics are utilized to evaluate the improvement. The results show that SemRef improves the accuracy by 17.72% to 43.35% across different metrics, as measured by normalized gain.

• We conduct a comprehensive study on SemRef, confirming its scalability and robustness. The data and code related to our study is publicly available on this website.