General Information

Problem Statement

  • Deployment and configuration of the test environment in virtualization context is complex and time consuming
  • Deployment and configuration of usage scenarios is critical for end-to-end testing and evaluation
  • Automatic control and execution is critical for efficient testing and evaluation tasks
  • A generic and extensible framework for further reuse


  • A framework aims to provide an integrated environment for software testers from test environment configuration, usage scenario deployment to automatic test and evaluation execution, that helps to achieve the functional and quality evaluation on virtualization technologies efficiently

Framework Architecture

Framework Deployment

Open Source Software Description URL 
Testopia Testopia is a test case management extension for Bugzilla. It is designed to be a generic tool for tracking test cases, allowing for testing organizations to integrate bug reporting with their test case run results
Testopia Plugin This plug-in integrates Jenkins with Testopia and generates reports on automated test execution. With this plug-in you can manage your tests in Testopia, schedule, and control in Jenkins, and execute using your favorite test execution tool (TestPartner, Selenium, TestNG, Perl modules, prove, PHPUnit, among others) 
Bugzilla Bugzilla is a "Defect Tracking System" or "Bug-Tracking System". Defect Tracking Systems allow individual or groups of developers to keep track of outstanding bugs in their product effectively
Jenkins Jenkins is an automation engine with an unparalleled plugin ecosystem to support different tools in delivery pipelines, whether your goal is continuous integration, automated testing, or continuous delivery
Expect Expect is a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc.
JMeter Apache JMeter may be used to test performance both on static and dynamic resources (Webservices (SOAP/REST), Web dynamic languages - PHP, Java, ASP.NET, Files, etc. -, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, group of servers, network or object to test its strength or to analyze overall performance under different load types
Selenium Primarily, it is for automating web applications for testing purpose but is certainly not limited to just that. Web-based administration tasks can also be automated as well
Cacti Cacti is a complete network graphing solution designed to harness the power of RRDTool's data storage and graphing functionality. Cacti provides a fast poller, advanced graph templating, multiple data acquisition methods, and user management features out of the box. All of this is wrapped in an intuitive, easy to use interface that makes sense for LAN-sized installations up to complex networks with hundreds of devices
libvirt Libvirt is the collection of software that provides a convenient way to manage virtual machines and other virtualization functionality, such as storage and network interface management. These software pieces include an API library, a daemon (libvirtd), and a command line utility (virsh). A primary goal of libvirt is to provide a single way to manage multiple different virtualization providers/hypervisors. For example, the command 'virsh list --all' can be used to list the existing virtual machines for any supported hypervisor (KVM, Xen, VMWare ESX, etc.)

Supplementary Materials


Related Publications

  1. C. H. Kao, "Testing and Evaluation Methods for Cloud Environments: A Review," accepted by 2017 6th International Conference on Software and Computing Technologies, Apr. 2017.
  2. C. H. Kao, "Testing and Evaluation Framework for Virtualization Technologies," accepted by Computing, 2016.
  3. C. H. Kao, C. C. Lin and H. T. Lu, "Toward Automatic Performance Testing for REST-based Web Applications," Proceedings of the Eleventh International Conference on Software Engineering Advances, pp. 68-71, Aug. 2016.