Introduction

General Information

Problem Statement

  • Deployment and configuration of the test environment in virtualization context is complex and time consuming
  • Deployment and configuration of usage scenarios is critical for end-to-end testing and evaluation
  • Automatic control and execution is critical for efficient testing and evaluation tasks
  • A generic and extensible framework for further reuse

Solution

  • A framework aims to provide an integrated environment for software testers from test environment configuration, usage scenario deployment to automatic test and evaluation execution, that helps to achieve the functional and quality evaluation on virtualization technologies efficiently

Framework Architecture

Framework Deployment



Open Source Software Description URL 
Testopia Testopia is a test case management extension for Bugzilla. It is designed to be a generic tool for tracking test cases, allowing for testing organizations to integrate bug reporting with their test case run results https://developer.mozilla.org/zh-TW/docs/Mozilla/Bugzilla/Testopia
Testopia Plugin This plug-in integrates Jenkins with Testopia and generates reports on automated test execution. With this plug-in you can manage your tests in Testopia, schedule, and control in Jenkins, and execute using your favorite test execution tool (TestPartner, Selenium, TestNG, Perl modules, prove, PHPUnit, among others) https://wiki.jenkins-ci.org/display/JENKINS/Testopia+Plugin 
Bugzilla Bugzilla is a "Defect Tracking System" or "Bug-Tracking System". Defect Tracking Systems allow individual or groups of developers to keep track of outstanding bugs in their product effectively https://www.bugzilla.org/
Jenkins Jenkins is an automation engine with an unparalleled plugin ecosystem to support different tools in delivery pipelines, whether your goal is continuous integration, automated testing, or continuous delivery https://jenkins.io/
Expect Expect is a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc. http://expect.sourceforge.net/
JMeter Apache JMeter may be used to test performance both on static and dynamic resources (Webservices (SOAP/REST), Web dynamic languages - PHP, Java, ASP.NET, Files, etc. -, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, group of servers, network or object to test its strength or to analyze overall performance under different load types http://jmeter.apache.org/
Selenium Primarily, it is for automating web applications for testing purpose but is certainly not limited to just that. Web-based administration tasks can also be automated as well http://www.seleniumhq.org/
Cacti Cacti is a complete network graphing solution designed to harness the power of RRDTool's data storage and graphing functionality. Cacti provides a fast poller, advanced graph templating, multiple data acquisition methods, and user management features out of the box. All of this is wrapped in an intuitive, easy to use interface that makes sense for LAN-sized installations up to complex networks with hundreds of devices http://www.cacti.net/
libvirt Libvirt is the collection of software that provides a convenient way to manage virtual machines and other virtualization functionality, such as storage and network interface management. These software pieces include an API library, a daemon (libvirtd), and a command line utility (virsh). A primary goal of libvirt is to provide a single way to manage multiple different virtualization providers/hypervisors. For example, the command 'virsh list --all' can be used to list the existing virtual machines for any supported hypervisor (KVM, Xen, VMWare ESX, etc.) https://libvirt.org/

Supplementary Materials

  1. http://www.slideshare.net/walkerchang/automated-testing-environment-by-bugzilla-testopia-and-jenkins
  2. https://wiki.jenkins-ci.org/display/JENKINS/Testopia+Plugin

Related Publications

  1. C. H. Kao, "Testing and Evaluation Methods for Cloud Environments: A Review," accepted by 2017 6th International Conference on Software and Computing Technologies, Apr. 2017.
  2. C. H. Kao, "Testing and Evaluation Framework for Virtualization Technologies," accepted by Computing, 2016.
  3. C. H. Kao, C. C. Lin and H. T. Lu, "Toward Automatic Performance Testing for REST-based Web Applications," Proceedings of the Eleventh International Conference on Software Engineering Advances, pp. 68-71, Aug. 2016.