Study Design
Embedded Files
The goal of this study is to assess (i) the precision of CONAN in detecting connectivity issues in Android apps, and (ii) the relevance of the identified issues from the practitioners’ perspective. The context is represented by (i) 44 open source apps on which we run CONAN looking for connectivity issues (ii) six interviews conducted with practitioners working with Android apps; and (iii) 84 issues we opened in 27 open source apps to report the connectivity issues identified by CONAN.
Data collection
Data collection
To answer our research questions, we selected 44 Android open source apps as subject systems. As a starting point we adopted the two following datasets:
Geiger et al: Composed of 8,431 real-world open-source Android apps. It combines source and history information from GitHub with metadata from Google Play store.
Coppola et al: Composed of 1,232 Android apps. The authors of this dataset mined all projects hosted on F-Droid, a repository for free and open source Android apps.
After removing duplicates among both datasets we obtained 8,157 unique open source Android apps, from which we excluded the ones no longer available on GitHub, leaving 8,003 apps. Then, we filtered out those not written in Java and/or Kotlin, obtaining 6,847 apps. This new dataset was then analyzed to identify repositories that could support an automated running of CONAN. This basically translates in the requirement that the repositories must use Gradle as their project-building tool. Thus, we identified repositories that: (1) contain the source code of more than one Android app (70); (2) do not use Gradle (1,902); (3) use Gradle, without however adopting the default name (i.e., app) for the main module to build (3,522); and (4) use Gradle, by adopting a customized name for the main module (1,353). We discarded repositories from (1) and (2) , since they do not support the automated running of CONAN. Also, we excluded all repositories not using a version of Gradle equal or newer than 6.7.1. Indeed, accordingly to the Android Lint documentation, the version in which we integrated CONAN (30.0.0-alpha10) works properly on projects adopting those Gradle versions.
After these filters, we obtained a sample of 62 repositories belonging to (3) and 160 repositories belonging to (4). For the projects in (4) (i.e., not using the default name for the main module), we manually identified such a name. The final set of 44 open-source Android apps was obtained as the set of projects successfully compiling and notifying at least one incident (i.e., a warning raised by CONAN) when running CONAN on their latest snapshot (as of July 19th, 2021).
RQ-1: What is the precision of CONAN in identifying Connectivity issues in Android apps?
RQ-1: What is the precision of CONAN in identifying Connectivity issues in Android apps?
Two authors performed a manual validation on a sample of the 898 candidate instances. We performed a stratified sampling: For each connectivity issues type Ti we targeted the random selection of 50 of its instances by considering the analyzed apps as strata. This means that the number of Ti instances we selected from each app is proportional to their prevalence in the app.
Overall, we manually analyzed 389 instances, ensuring a significance interval (margin of error) of ±5% with a confidence level of 99%. The two authors involved in the manual validation independently inspected each instance, classifying it as a true positive (TP) or false positive (FP). This was done by inspecting the source code of the apps. Once completed, the two authors together inspected the 44 cases of conflict (11.31%), solving them through open discussion. See in Table I the number of candidate connectivity issues identified by CONAN in the 44 subject apps.
Table I: Number of candidate connectivity issues identified by CONAN in the 44 subject apps
RQ-2: What is the relevance of the connectivity issues identified for Android practitioners?
RQ-2: What is the relevance of the connectivity issues identified for Android practitioners?
We answer RQ-2 in two steps. First, we conduct six interviews with Android practitioners in our contact network asking them about the relevance of the issue types currently supported in CONAN. The first part of the interview asked participants a few demographics questions including (i) the country in which they were located; (ii) their role in the company (e.g., developer, tester); (iii) their programming experience and Android programming experience (in years). Then, we asked a few questions related to whether (i) they have experienced connectivity issues as app users; (ii) they adopt connectivity strategies (i.e., code aimed at ensuring the proper working on the app in different connectivity scenarios) in their apps; (iii) they think connectivity strategies are part of the prioritized tasks when developing an app. After that, we asked the participants to assess their perceived relevance of the 16 connectivity issues that CONAN is able to detect. For each issue type, we showed the participant a description of the issue and asked their perceived relevance of this issue on a fivepoint scale: Very Irrelevant, Irrelevant, Neutral, Relevant, Very Relevant. The last question of the interview was: What kind of support would you like to have from an automated tool that helps to detect these issues?
Second, using the true positive instances (i.e., actual connectivity issues) manually identified in RQ-1, we opened 84 issues in 27 of the 44 apps involved in our study (some were excluded due to the lack of true positive instances to report). When the same issue type affected the same app in multiple locations (e.g., several methods of the app used network connectivity without first checking for the status of the network), we only opened one issue summarizing all of them. For this reason, the 84 reports concern a total of 150 connectivity issues detected by CONAN.
Page updated
Google Sites
Report abuse