Total Combined Evaluation Results

Summary of Evaluations

The Content, Usability, and Performance Evaluation (CUPE) Criteria were suggested to find which digital libraries are well-designed digital libraries among existing digital libraries in fifteen subject areas.

The well-designed digital libraries in various subject areas will consist of the proposed International Open Public Digital Library (IOPDL). With the CUPE criteria, three main evaluations are performed with seven sub evaluation criteria. First, to find candidates of well-designed digital libraries, Professor McDonough and I evaluated content’s quality of existing digital libraries in fifteen subject area.

We investigated: whether existing digital libraries’ content is accuracy in wide and deep scopes; whether they have authority; and whether they give subjective Satisfaction. Through the content evaluation, sixty two digital libraries were recommended as candidates of well-designed digital libraries. Next, sixty two candidate digital libraries were evaluated again with four Usability evaluations: Accessibility evaluation with seven accessibility evaluation tools, and Interface Usability evaluations with Convenience/Ease of Use, Interfaces’ Consistency, and Visible Design and Aesthetic Appeal Evaluation Criteria. The interface usability evaluations were done by heuristic evaluation method. Lastly, to evaluate performance of sixty two candidate digital libraries, two computer programs were implemented to measure the Link and Search Response Times, and Relevance of obtained results.

Total Combined Evaluation of Seven Evaluations

To put together all scored values (scores) of six evaluations with the CUPE criteria, an average is calculated for each digital library. I call it as ‘final average.’ In here, the content evaluation value is excluded, because through the evaluation, the candidate, sixty two digital libraries, are recommended.

1. Six scored values of six evaluations are calculated into a final average with Equation 8.
2. Then, a total average is calculated by all final averages of sixty two digital libraries. I call it as ‘total average.’ As Equation 9 shows, it is calculated by dividing sums of Final Averages of the candidate by 62.
3. Then, the digital libraries that have higher average values than the total average of all sixty two candidate digital libraries are defined as well-designed digital libraries, as Equation 10 shows.

Total Result and Analyses

As a result, the total average of all candidate digital libraries is 3.2003 in 5-point scale. It is higher than a half of 5 points. It may mean generally that the candidate digital libraries may provide enough quality usability and performance. They are recommended by content evaluation in their subject domains. Ultimately, thirty four digital libraries out of sixty two digital libraries show higher final averages than the total average, 3.2003. That is, about half of the candidate digital libraries are turned out as well-designed digital libraries in their subject domains.

NASA's Visible Earth (4.452381) and Census Atlas of the United States (4.263889) in Geography subject area show the highest averages by seven evaluations with the CUPE criteria. Third, U.S. Department of Health & Human Services shows 4.261905 in medicine subject domain. Fourth, GPO Access in political science and law shows 4.142857. Fifth, National Science Foundation in science shows 4.138889. Appendix A in the Evaluation paper show more detail results based on scores.

Especially, in Geography subject domain, all three candidates turn out as WDDLs: NASA's Visible Earth, Census Atlas of the United States, and David Rumsey Map Collection. Moreover, the candidate digital libraries in History of the America subject area are seven, and six digital libraries are turned out as well-designed digital libraries except Digital Past which has problems in accessing the website. According to the results based on averages of subject areas, the average of Geography subject area is the highest (4.1316). Then, the averages of Medicine (3.5571), Military Science (3.496), Political Science and Law (3.4524), Science (3.4454), Agriculture (3.3849), and Art (3.3577) are high. Ten subject areas, 66.67% of fifteen subject areas, have higher averages than the total average. On top of that, all averages of fifteen subject areas are higher than 2 points out of five points. The social science subject area is the lowest, 2.2341 average. The results show the qualities of sixty two candidate digital libraries in fifteen subject areas are good in Content quality, Usability quality, and Performance quality overall.

Finally, I can conclude that the U.S. has strongly specialized digital libraries in Geography subject area (4.1316), Medicine (3.5571), Military Science (3.496), Political Science and Law (3.4524), and Science (3.4454).

Future Evaluations

One more thing I would like to point out is that the experiment is mainly done with the U.S. digital libraries except few other country libraries such as The National Archives, Education Resources UK, British Library Online Gallery, International children’s Digital Library, Chinese Philosophical Etext archive. However, for the International Open Public Digital Library (IOPDL), we should include and evaluate more national digital libraries of many countries.