Improving Automation in Developer Testing

PROJECT SUMMARY

Developer testing, a common step in software development, involves generating sufficient test inputs and checking the behavior of the program under test during the execution of the test inputs. Complicated logics inside a method make generating appropriate arguments difficult. In testing object-oriented programs, generating method sequences to put the receiver object or argument objects into appropriate states further complicates test-input generation. After the generated test inputs are executed, program crashes or uncaught exceptions can be used to indicate program problems, especially robustness problems. However, some program problems such as producing wrong program outputs do not crash the program. This project improves automation in developer testing in collaboration between the NCSU Automated Software Engineering Research Group and Microsoft Research, developing novel ideas and tools on Pex, an automatic unit testing tool from Microsoft Research. The developed tools are released as open source at the Pex Extensions project web.

RELEVANCE TO MILITARY/DOD

• In 2009 Summer, our undergraduate student, Justin Gorham, is working as a summer intern at a Fort Hood Army team in applying Pex and our extensions on Army code bases.
  
• In 2009 Summer, our PhD student, Kunal Taneja, is working as a summer intern at FDA in applying Pex and our extensions on a DoD code base for regulatory purposes.
  

PEOPLE

Faculty
    Tao Xie

Graduate Students
    Nuo Li
    Madhuri R Marri
    Kunal Taneja
    Suresh Thummalapenta

Undergraduate Students
    Justin Gorham

Collaborators
   Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte (Microsoft Research)

PUBLICATIONS

1. Suresh Thummalapenta, Tao Xie, Nikolai Tillmann, Peli de Halleux, and Wolfram Schulte. MSeqGen: Object-Oriented Unit-Test Generation via Mining Source Code. In Proceedings of the 7th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE 2009), Amsterdam, the Netherlands, August 2009. [PDF][BibTeX]
2. Tao Xie, Nikolai Tillmann, Peli de Halleux, and Wolfram Schulte. Fitness-Guided Path Exploration in Dynamic Symbolic Execution. To appear in Proceedings of the 39th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2009), Lisbon, Portugal, June-July 2009. [PDF][BibTeX]
3. Kunal Taneja, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. Guided Path Exploration for Regression Test Generation. To appear in Proceedings of the 31st International Conference on Software Engineering (ICSE 2009), New Ideas and Emerging Results, Vancouver, Canada, May 2009. [PDF][BibTeX]
4. Madhuri R Marri, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. An Empirical Study of Testing File-System-Dependent Software with Mock Objects. To appear in Proceedings of the 4th International Workshop on Automation of Software Test (AST 2009), Business and Industry Case Studies, Vancouver, Canada, May 2009. [PDF][BibTeX]
5. Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. Mutation Analysis of Parameterized Unit Tests. To appear in Proceedings of the 4th International Workshop on Mutation Analysis (Mutation 2009), Denver, Colorado, April 2009.  [PDF][BibTeX]
6. Tao Xie, Nikolai Tillmann, Jonathan de Halleux, Wolfram Schulte. Method-Sequence Exploration for Automated Unit Testing of Object-Oriented Programs. Presented at Workshop on State-Space Exploration for Automated Testing (SSEAT 2008), Seattle, WA, July 2008. [BibTeX]
7. Tao Xie, Darko Marinov, Wolfram Schulte, and David Notkin. Symstra: A Framework for Generating Object-Oriented Unit Tests using Symbolic Execution. In Proceedings of the 11th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS 2005), Edinburgh, U.K., pp. 365-381, April 2005. [PDF][BibTeX][Slides]

TUTORIALS

1. Nikolai Tillmann, Jonathan de Halleux, Tao Xie, and Wolfram Schulte. Parameterized Unit Testing: Principles, Techniques, and Applications in Practice. To be presented at ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA 2009), Tutorials, Orlando, Florida, October 2009. [BibTeX]
2. Nikolai Tillmann, Jonathan de Halleux, Tao Xie, and Wolfram Schulte. Parameterized Unit Testing: Principles, Techniques, and Applications in Practice. Presented at the 31st International Conference on Software Engineering (ICSE 2009), Tutorials, Vancouver, Canada, May 2009. [BibTeX]

PRESENTATIONS

1. Tao Xie. Automated Testing of System Software. Presented at the NCSU SOSI Year 1 review, Raleigh, NC. [Slides]

SUBPROJECTS

• Wiki site for Teaching Parameterized Unit Testing/Pex
  
• Fitnex: Fitness-guided path exploration 
• MSeqGen: Method sequence generation based on static code mining
• Regtestgen: Guided path exploration for regression test generation
• Reggae: Automated Test Generation for Programs using Complex Regular Expressions

SPONSORS

National Science Foundation Award CCF-0725190, Science of Design Program (01/01/2008-12/31/2010)

Army Research Office Award W911NF-08-1-0443 (09/08/2008-08/30/2011)

NCSU Secure Open Systems Initiative (SOSI) (08/15/2008-)