depex

Improving Automation in Developer Testing: Cooperative Developer Testing

PROJECT SUMMARY

Developer testing, a common step in software development, involves generating sufficient test inputs and checking the behavior of the program under test during the execution of the test inputs. Complicated logics inside a method make generating appropriate arguments difficult. In testing object-oriented programs, generating method sequences to put the receiver object or argument objects into appropriate states further complicates test-input generation. After the generated test inputs are executed, program crashes or uncaught exceptions can be used to indicate program problems, especially robustness problems. However, some program problems such as producing wrong program outputs do not crash the program. This project improves automation in developer testing in collaboration between the NCSU Automated Software Engineering Research Group and Microsoft Research, developing novel ideas and tools on Pex, an automatic unit testing tool from Microsoft Research. The developed tools are released as open source at the Pex Extensions project web. Our teaching/learning materials around paramterized unit testing/Pex can be found at our Wiki site for Teaching Parameterized Unit Testing/Pex. Our Pex community building activities are summarized in the Pex community web.

COLLABORATION VISITS with MICROSOFT RESEARCH

• May 24-June 8, 2010: Tao Xie worked as a visiting researcher at the Pex group, Microsoft Research, Redmond
• Sept 14-18, 2009: Nikolai Tillmann from the Pex group, Microsoft Research visited the NCSU ASE group
• May-August, 2009: Suresh Thummalapenta worked as a research summer intern at the Pex group, Microsoft Research, Redmond
• July-August, 2009: Tao Xie worked as a visiting researcher at the Pex group, Microsoft Research, Redmond
• June-July, 2008: Tao Xie worked as a visiting researcher at the Pex group, Microsoft Research, Redmond
• May-June, 2007: Tao Xie worked as a visiting researcher at the Pex group, Microsoft Research, Redmond

INTERNATIONAL or K-12 OUTREACH

• July 19-23 2010, Tao Xie will teach a one-week summer course in Shanghai within the Dragon Star program, where Pex, Code Contract, CHESS, etc. will be used to teach students from China on the area of software testing and analysis
• Summers of 2010 and 2009, Tao Xie has mentored two high school students on learning and applying Parametrized Unit Testing and Pex.

OUTREACH to MILITARY/DOD, INDUSTRY

• In 2010 Summer, our PhD student TO-BE, Justin Gorham, is working as a summer intern at AllScripts, a leading vendor of healthcare IT solutions in applying Pex on their code bases.
• In 2009 Summer, our undergraduate student, Justin Gorham, worked as a summer intern at a Fort Hood Army team in applying Pex and our extensions on Army code bases.
• In 2009 Summer, our PhD student, Kunal Taneja, worked as a summer intern at FDA in applying Pex and our extensions on a DoD code base for regulatory purposes.

PEOPLE

Faculty

Tao Xie

Microsoft Research Collaborators

Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte

NCSU ASE Graduate Students

Justin Gorham

Madhuri R Marri

Rahul Pandita

Yoonki Song

Kunal Taneja

Suresh Thummalapenta

Xusheng Xiao

Kiran Shakya

Ge Xi

Collaborating Students/Professors from China

Yitao Ni (Peking University, China)

Lin Yan (Peking University, China)

Lingming Zhang (Peking University, China)

Lu Zhang (Peking University, China)

Hao Zhong (Chinese Academy of Sciences)

Undergraduate Students

Khiry Arnold

Justin Gorham

Dennis Rankin

Travis Weston

Former Members

Nuo Li (former PhD student)

PUBLICATIONS

1. Linghao Zhang, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, Xiaoxing Ma, and Jian Lv. Environment Modeling for Automated Testing of Cloud Applications. IEEE Software, Special Issue on Software Engineering for Cloud Computing, March/April 2012. [PDF][BibTex]
2. Kai Pan, Xintao Wu, and Tao Xie. Generating Program Inputs for Database Application Testing. In Proceedings of the 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011), Lawrence, Kansas, November 2011. [PDF][BibTeX]
3. Suresh Thummalapenta, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Zhendong Su. Synthesizing Method Sequences for High-Coverage Testing. In Proceedings of ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications
4. (OOPSLA 2011), Portland, Oregon, October 2011. [PDF][BibTeX]
5. Kunal Taneja, Tao Xie, Nikolai Tillmann, and Jonathan de Halleux. eXpress: Guided Path Exploration for Efficient Regression Test Generation.
6. In Proceedings of the 2011 International Symposium on Software Testing and Analysis (ISSTA 2011), Toronto, Canada, July 2011. [PDF][BibTeX]
7. A previous version appeared in Proceedings of ICSE 2009 New Ideas and Emerging Results track.
8. Xusheng Xiao, Tao Xie, Nikolai Tillmann, and Jonathan de Halleux. Precise Identification of Problems for Structural Test Generation.
9. In Proceedings of the 33rd International Conference on Software Engineering (ICSE 2011), Honolulu, Hawaii, pp. 611-620, May 2011. [PDF][BibTeX]
10. Suresh Thummalapenta, Madhuri Marri, Tao Xie, Nikolai Tillmann, and Jonathan de Halleux. Retrofitting Unit Tests for Parameterized Unit Testing. In Proceedings of International Conference on Fundamental Approaches to Software Engineering (FASE 2011), Saarbrücken, Germany, March-April 2011. [PDF][BibTeX]
11. Judith Bishop, Jonathan de Halleux, Nikolai Tillmann, Nigel Horspool, Don Syme, and Tao Xie. Browser-based Software for Technology Transfer. In Proceedings of the 2011 Annual Research Conference of the South African Institute for Computer Scientists and Information Technologists (SAICSIT 2011), Industry Oriented Paper, Cape Town, South Africa, October 2011. [PDF][BibTeX]
12. Xi Ge, Kunal Taneja, Tao Xie, and Nikolai Tillmann. DyTa: Dynamic Symbolic Execution Guided with Static Verification Results. In Proceedings of the 33rd International Conference on Software Engineering (ICSE 2011), Demonstration, Honolulu, Hawaii, pp. 992-994, May 2011. [PDF][BibTeX]
13. Nikolai Tillmann, Jonathan de Halleux, and Tao Xie. Pex4Fun: Teaching and Learning Computer Science via Social Gaming. In Proceedings of the 24th IEEE-CS Conference on Software Engineering Education and Training (CSEE&T 2011), Practice and Methods Presentations, & Tutorials (PMP&T), Honolulu, Hawaii, May 2011. [PDF][Tutorial Web][BibTeX]
14. Nikolai Tillmann, Jonathan de Halleux, and Tao Xie. Pex for Fun: Engineering an Automated Testing Tool for Serious Games in Computer Science. Microsoft Research, Technical report, no. MSR-TR-2011-41, March 2011. [PDF][BibTex]
15. Kai Pan, Xintao Wu, and Tao Xie. Database State Generation via Dynamic Symbolic Execution for Coverage Criteria. In Proceedings of the 4th International Workshop on Testing Database Systems (DBTest11), Athens, Greece, June 2011. [PDF][BibTeX]
16. Tao Xie, Jonathan de Halleux, Nikolai Tillmann, and Wolfram Schulte. Teaching and Training Developer-Testing Techniques and Tool Support. Proceedings of the 25th Annual ACM Conference on Systems, Programming, Languages, and Applications: Software for Humanity (SPLASH 2010), Educators' and Trainers' Symposium, Reno/Tahoe Nevada, October 2010.[PDF][BibTeX]
17. Kunal Taneja, Yi Zhang, and Tao Xie. MODA: Automated Test Generation for Database Applications via Mock Objects. In Proceedings of the 25th IEEE/ACM International Conference on Automated Software Engineering (ASE 2010), Short Paper, Antwerp, Belgium, September 2010. [BibTeX]
18. Kunal Taneja, Nuo Li, Madhuri Marri, Tao Xie, and Nikolai Tillmann. MiTV: Multiple-Implementation Testing of User-Input Validators for Web Applications. In Proceedings of the 25th IEEE/ACM International Conference on Automated Software Engineering (ASE 2010), Short Paper, Antwerp, Belgium, September 2010. [BibTeX]
19. Wujie Zheng, Qirun Zhang, Michael Lyu, and Tao Xie. Random Unit-Test Generation with MUT-aware Sequence Recommendation. In Proceedings of the 25th IEEE/ACM International Conference on Automated Software Engineering (ASE 2010), Short Paper, Antwerp, Belgium, September 2010. [BibTeX]
20. Rahul Pandita, Tao Xie, Nikolai Tillmann, and Jonathan de Halleux. Guided Test Generation for Coverage Criteria. In Proceedings of the 26th IEEE International Conference on Software Maintenance (ICSM 2010), Timișoara, Romania, September 2010. [BibTeX]
21. Lingming Zhang, Tao Xie, Lu Zhang, Nikolai Tillmann, Jonathan de Halleux, and Hong Mei. Test Generation via Dynamic Symbolic Execution for Mutation Testing. In Proceedings of the 26th IEEE International Conference on Software Maintenance (ICSM 2010), Timișoara, Romania, September 2010. [BibTeX]
22. Madhuri R Marri, Suresh Thummalapenta, Tao Xie, Nikolai Tillmann, Jonathan de Halleux. Retrofitting Unit Tests for Parameterized Unit Testing. North Carolina State University Department of Computer Science Technical report TR-2010-9, March 23, 2010. Download: [PDF][BibTex]
23. Xusheng Xiao, Tao Xie, Nikolai Tillmann, and Peli de Halleux. Issue Analysis for Residual Structural Coverage in Dynamic Symbolic Execution. North Carolina State University Department of Computer Science Technical report TR-2010-7, March 23, 2010. Download: [PDF][BibTex]
24. Nuo Li, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. Reggae: Automated Test Generation for Programs using Complex Regular Expressions. In Proceedings of the 24th IEEE/ACM International Conference on Automated Software Engineering (ASE 2009), Short Paper, Auckland, New Zealand, November 2009. [PDF][BibTeX]
25. Suresh Thummalapenta, Tao Xie, Nikolai Tillmann, Peli de Halleux, and Wolfram Schulte. MSeqGen: Object-Oriented Unit-Test Generation via Mining Source Code. In Proceedings of the 7th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE 2009), Amsterdam, the Netherlands, August 2009. [PDF][BibTeX]
26. Tao Xie, Nikolai Tillmann, Peli de Halleux, and Wolfram Schulte. Fitness-Guided Path Exploration in Dynamic Symbolic Execution. In Proceedings of the 39th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2009), Lisbon, Portugal, June-July 2009. [PDF][BibTeX]
27. Kunal Taneja, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. Guided Path Exploration for Regression Test Generation. In Proceedings of the 31st International Conference on Software Engineering (ICSE 2009), New Ideas and Emerging Results, Vancouver, Canada, May 2009. [PDF][BibTeX]
28. Madhuri R Marri, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. An Empirical Study of Testing File-System-Dependent Software with Mock Objects. In Proceedings of the 4th International Workshop on Automation of Software Test (AST 2009), Business and Industry Case Studies, Vancouver, Canada, May 2009. [PDF][BibTeX]
29. Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. Mutation Analysis of Parameterized Unit Tests. In Proceedings of the 4th International Workshop on Mutation Analysis (Mutation 2009), Denver, Colorado, April 2009. [PDF][BibTeX]
30. Tao Xie, Nikolai Tillmann, Jonathan de Halleux, Wolfram Schulte. Method-Sequence Exploration for Automated Unit Testing of Object-Oriented Programs. Presented at Workshop on State-Space Exploration for Automated Testing (SSEAT 2008), Seattle, WA, July 2008. [BibTeX]
31. Tao Xie, Darko Marinov, Wolfram Schulte, and David Notkin. Symstra: A Framework for Generating Object-Oriented Unit Tests using Symbolic Execution. In Proceedings of the 11th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS 2005), Edinburgh, U.K., pp. 365-381, April 2005. [PDF][BibTeX][Slides]

TUTORIALS

1. Nikolai Tillmann, Jonathan de Halleux, and Tao Xie. Parameterized Unit Testing: Theory and Practice. In Proceedings of the 32nd International Conference on Software Engineering(ICSE 2010), Tutorials, Cape Town, South Africa, May 2010. [Tutorial Web][BibTeX]
2. Nikolai Tillmann, Jonathan de Halleux, Tao Xie, and Wolfram Schulte. Parameterized Unit Testing: Principles, Techniques, and Applications in Practice. To be presented at ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA 2009), Tutorials, Orlando, Florida, October 2009. [Tutorial Web][BibTeX]
3. Nikolai Tillmann, Jonathan de Halleux, Tao Xie, and Wolfram Schulte. Parameterized Unit Testing: Principles, Techniques, and Applications in Practice. Presented at the 31st International Conference on Software Engineering (ICSE 2009), Tutorials, Vancouver, Canada, May 2009. [Tutorial Web][BibTeX]

COURSES

1. Tao Xie created Wiki site for Teaching/Learning Parameterized Unit Testing/Pex and Parameterized Unit Testing presentation/tutorial.
2. Tao Xie taught CSC 712 Software Testing and Reliability, Graduate Course, Fall 2009, where Microsoft Research Pex, Code Contracts, and NModel were used as the primary tools and materials of the course.
3. Tao Xie taught CSC 712 Software Testing and Reliability, Graduate Course, Fall 2008, where Microsoft Research Pex and NModel were used as the primary tools and materials of the course.
4. Tao Xie taught CSC510 Software Engineering, Graduate Course, Spring 2009, where where Microsoft Research Pex and CHESS were used as tools and materials for covering testing topics in the course.

SELECTED PRESENTATIONS

1. Tao Xie. Automated Developer Testing: Achievements and Challenges. Invited talk, Department of Computer Science, Columbia University, NY, March 2010. [Slides]
2. Tao Xie. Automated Developer Testing: Achievements and Challenges. Keynote talk, the 2nd CREST Open Workshop Software Testing (COW), CREST Centre, King's College London , December 2009.
1. Automated Developer Testing: Achievements and Challenges. Invited talk, Department of Computer Science, Virginia Polytechnic Institute and State University, Blacksburg, VA, October 2009. [Slides]
2. Automated Developer Testing: Achievements and Challenges. Invited talk, Department of Computer and Information Science, Indiana University/Purdue University at Indianapolis, Indianapolis, IN, October 2009
3. Parameterized Unit Testing: Principles, Techniques, and Applications in Practice. Invited talk, Protocol Engineering Team, Microsoft, Beijing, June 2009.
4. Fitness-Guided Path Exploration in Automated Test Generation. Invited talk, Department of Computer Science and Engineering, Michigan State University, November 2008. [Slides]
5. Fitness-Guided Path Exploration in Automated Test Generation. Invited talk, Department of Computer Science, the College of William and Mary, October 2008.
6. Fitness-Guided Path Exploration in Dynamic Symbolic Execution. Invited talk, Foundation of Software Engineering Group, Microsoft Research, July 2008.
7. Tao Xie. Automated Testing of System Software. Presented at the NCSU SOSI Year 1 review, Raleigh, NC. [Slides]

SUBPROJECTS (Pex community web)

• Wiki site for Teaching Parameterized Unit Testing/Pex
  • Fitnex: Fitness-guided path exploration
• MSeqGen: Method sequence generation based on static code mining
• eXpress: Guided path exploration for regression test generation
• Reggae: Automated Test Generation for Programs using Complex Regular Expressions
• Visco: Visualization to Aid Issue Analysis for Residual Structural Coverage

SPONSORS

National Science Foundation Award CCF-0915400, SHF Program (09/01/2009-08/31/2012), CCF-0845272, CAREER SHF Program (08/01/2009-07/31/2014), CCF-0725190, SoD Program (01/01/2008-12/31/2010)

Army Research Office Award W911NF-08-1-0443 (09/08/2008-08/30/2011)