Testing Embedded Software

Overview

Koopman, P., "Avoiding the top 43 embedded software risks," Embedded Systems Conference Silicon Valley, May 2, 2011. (Talk slides)
Gabor Karsai, Fabio Massacci, Leon J. Osterweil, Ina Schieferdecker: Evolving Embedded Systems. IEEE Computer 43(5): 34-40 (2010)
Gian Pietro Picco. Software engineering and wireless sensor networks: happy marriage or consensual divorce?. FoSER 2010.
Peter Liggesmeyer and Mario Trapp. Trends in Embedded Software Engineering. IEEE Softw 2009. [Slides]
Reinhard Wilhelm, Jakob Engblom, Andreas Ermedahl, Niklas Holsti, Stephan Thesing, David Whalley, Guillem Bernat, Christian Ferdinand, Reinhold Heckmann, Tulika Mitra, Frank Mueller, Isabelle Puaut, Peter Puschner, Jan Staschulat, and Per Stenstrm. 2008. The worst-case execution-time problem —overview of methods and survey of tools. ACM Trans. Embed. Comput. Syst. 7, 3, Article 36 (May 2008)
David Jeff Jackson and Paul Caspi. 2005. Embedded systems education: future directions, initiatives, and cooperation. SIGBED Rev. 2, 4 (October 2005)
Koopman, P., "Avoiding the top 43 embedded software risks," Embedded Systems Conference Silicon Valley, May 2, 2011. (Talk slides)
- Beatty, S., Where Testing Fails, Part 1, Part 2 Embedded Systems Programming, 2003
Patrick Cousot and Radhia Cousot. Verification of Embedded Software: Problems and Perspectives. Embedded SoftwareLecture Notes in Computer Science, 2001, Volume 2211/2001
Umansky, Studs and Duds, The Washington Monthly, December 2001
Ganssle, "Proactive Debugging," Embedded Systems Programming,, January 2001.
Beatty, "Sensible software testing", Embedded Systems Programming, August 2000
Edward A. Lee. 2000. What's Ahead for Embedded Software?. Computer 33, 9 (September 2000),
Edward A. Lee. Embedded Software — An Agenda for Research. UCB ERL Memorandum M, 1999
Nancy G. Leveson. Software safety in embedded computer systems. Commun. ACM 34, 2 (February 1991),

Testing Infrastructures/Testability

Markku Pollari. A Software Framework for Improving the Testability of Embedded Real-time Systems, Thesis 2009
Suvarshi Bhadra, Alexander Conrad, Charles Hurkes, Brian Kirklin, and Gregory M. Kapfhammer. An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments. AST 2009.
Teemu Kanstrn. A Study on Design for Testability in Component-Based Embedded Software. SERA 2008.
Gregory M. Kapfhammer, Mary Lou Soffa, and Daniel Mosse. Testing in resource constrained execution environments. ASE 2005.
- F. Vargas, E. Bezerra, A. Terroso. Testability Verification of Embedded Systems Based on Weak Mutation Analysis. TECS 1999.

Testing Processes

Jennifer L. Davidson and Guy-Alain Amoussou. Testing to certify an embedded software system. J. Comput. Sci. Coll. 25, 4 (April 2010)
Jooyoung Seo, Yuhoon Ki, Byoungju Choi, Kwanghyun La, Which Spot Should I Test for Effective Embedded Software Testing? SSIRI 2008
Falk Salewski, Stefan Kowalewski: Achieving Highly Reliable Embedded Software: An Empirical Evaluation of Different Approaches. SAFECOMP 2007
Wei-Tek Tsai, Lian Yu, Feng Zhu, and Ray Paul. Rapid Embedded System Testing Using Verification Patterns. IEEE Softw. 22, 4 (July 2005).
Nancy Van Schooenderwoert and Ron Morsicato. Taming the Embedded Tiger - Agile Test Techniques for Embedded Software. ADC 2004.
Coulter, A.C.. Graybox software testing methodology: embedded software testing technique. Digital Avionics Systems Conference, 1999

Coverage Criteria/Fault Models

N. Jones. A taxonomy of bug types in embedded systems,Oct. 2009. http://embeddedgurus.com/stack-overflow/-2009/10/a-taxonomy-of-bug-types-in-embedded-systems.
Lai zhifeng, S.C. Cheung, W.K.Chan. Inter-Context Control-Flow and Data-Flow Test Adequacy Criteria for nesC Applications. FSE 2008.
Nguyet T. M. Nguyen and Mary Lou Soffa. Program representations for testing wireless sensor network applications. DoSTA 2007.
Jing Guan, Jeff Offutt, and Paul Ammann. An industrial case study of structural testing applied to safety-critical embedded software. ISESE 2006.
M. Krichen and S. Tripakis. Real-time testing with timed-automata testers and coverage criteria. Technical Report TR-2004-15, 2004-15, Verimag Technical Report, 2004.
José C. Costa, Srinivas Devadas, and José C. Monteiro. Observability analysis of embedded software for coverage-directed validation. ICCAD 2000.
F. Vargas, E. Bezerra, A. Terroso. Testability Verification of Embedded Systems Based on Weak Mutation Analysis. TECS 1999.

Test Oracles

T. Yu, A. Sung, W. Srisa-an, and G. Rothermel, Using Property-Based Oracles when Testing Embedded System Applications, ICST 2011.
Yafeng Wu, Krasimira Kapitanova, Jingyuan Li, John A. Stankovic, Sang H. Son, and Kamin Whitehouse. Run time assurance of application-level requirements in wireless sensor networks. IPSN 2010.
W. K. Chan, T. Y. Chen, S. C. Cheung, T. H. Tse, and Zhenyu Zhang. Towards the testing of power-aware software applications for wireless sensor networks. Ada-Europe 2007.

Code-Based Testing

Y.Kim and M.Kim, SCORE: a Scalable Concolic Testing Tool for Reliable Embedded Software, ESEC/FSE 2011 Demo
Y.Kim, M.Kim and Y.Jang, Concolic Testing on Embedded Software - Case Studies on Mobile Platform Programs. ESEC/FSE 2011 Industry Track

Sanjit A. Seshia and Jonathan Kotker. GameTime: A Toolkit for Timing Analysis of Software. TACAS 2011.
A. Sung, W. Srisa-an, G. Rothermel, T. Yu, Testing Inter-Layer and Inter-Task Interactions in RTES Applications, APSEC 2010.
Peng Li and John Regehr. T-Check: Bug Finding for Sensor Networks. ISPN 2010.
Makoto Higashi, Tetsuo Yamamoto, Yasuhiro Hayase, Takashi Ishio, and Katsuro Inoue. An effective method to control interrupt handler for data race detection. AST 2010. [Slides]
Rupak Majumdar, Indranil Saha, and Zilong Wang. Systematic Testing for Control Applications. Memocode 2010.
Rupak Majumdar, Indranil Saha. Symbolic Robustness Analysis, RTSS 2009. [Slides]
- Nupur Kothari, Todd Millstein, Ramesh Govindan. Deriving State Machines from TinyOS Programs Using Symbolic Execution. IPSN 2008. [Slides]
Marouane Tlili, Stefan Wappler, and Harmen Sthamer. Improving evolutionary real-time testing. GECCO 2006
John Regehr. Random testing of interrupt-driven software. EMSOFT 2005. [Slides]
Songtao Xia, Ben Di Vito, and Cesar Munoz. Automated test generation for engineering applications. ASE 2005.
Guillaume Brat, Doron Drusinsky, Dimitra Giannakopoulou, Allen Goldberg, Klaus Havelund, Mike Lowry, Corina Pasareanu, Arnaud Venet, Willem Visser, and Rich Washington. Experimental Evaluation of Verification and Validation Tools on Martian Rover Software. Form. Methods Syst. Des. 25, 2-3 (September 2004),
Joachim Wegener, Kerstin Buhr, and Hartmut Pohlheim. Automatic Test Data Generation For Structural Testing Of Embedded Software Systems By Evolutionary Testing. GECCO 2002.
Joachim Wegener and Frank Mueller. A Comparison of Static Analysis and Evolutionary Testing for the Verification of Timing Constraints. Real-Time Syst. 21, 3 (November 2001),
Joachim Wegener, Hartmut Pohlheim, and Harmen Sthamer. Testing the Temporal Behavior of Real-Time Tasks Using Extended Evolutionary Algorithms. RTSS 1999.
Joachim Wegener, Harmen Sthamer, Bryan F. Jones, and David E. Eyres. Testing real-time systems using genetic algorithms. Software Quality Control 1997.

Model-based Testing

Ina Schieferdecker: Test Automation with TTCN-3 - State of the Art and a Future Perspective. ICTSS 2010. [Slides]
Sastry J.K.R. Chandra Prakash V.A Formal Framework for Verification and Validation of External Behavioral Models of Embedded Systems Represented Through Black Box Structures IACC 2010.
Peter M. Kruse, Joachim Wegener, and Stefan Wappler. A highly configurable test system for evolutionary black-box testing of embedded systems. GECCO 2009
Juergen Grossmann, Diana Serbanescu, and Ina Schieferdecker. Testing Embedded Real Time Systems with TTCN-3. ICST 2009.
- Aditya Kanade, Rajeev Alur, Franjo Ivan\&\#269;i\&\#263;, S. Ramesh, Sriram Sankaranarayanan, and K. C. Shashidhar. Generating and Analyzing Symbolic Traces of Simulink/Stateflow Models. CAV 2009.
Rajeev Alur, Aditya Kanade, S. Ramesh, and K. C. Shashidhar. Symbolic analysis for improving simulation coverage of Simulink/Stateflow models. EMSOFT 2008.
Manoranjan Satpathy, Anand Yeolekar, and S. Ramesh. Randomized directed testing (REDIRECT) for Simulink/Stateflow models. EMSOFT 2008. [Slides]
Ambar A. Gadkari, Anand Yeolekar, J. Suresh, S. Ramesh, Swarup Mohalik, K. C. Shashidhar: AutoMOTGen: Automatic Model Oriented Test Generator for Embedded Control Systems. CAV 2008.
Benjamin Hummel and Peter Braun. Towards an integrated system model for testing and verification of automation machines. MiSE 2008.
Justyna Zander-Nowicka, Xuezheng Xiong, Ina Schieferdecker: Systematic Test Data Generation for Embedded Software. SERP 2008.
Ahyoung Sung, Byoungju Choi, and Seokkyoo Shin. An interface test model for hardware-dependent software and embedded OS API of the embedded system. Comput. Stand. Interfaces 29, 4 (May 2007),
M. Esser and P. Struss. Fault-model-based test generation for embedded software. IJCAI 2007.
Justyna Zander-Nowicka, Abel Marrero Pérez, Ina Schieferdecker: From Functional Requirements through Test Evaluation Design to Automatic Test Data Patterns Retrieval - a Concept for Testing of Software Dedicated for Hybrid Embedded Systems. SERP 2007.
Justyna Zander-Nowicka, Ina Schieferdecker, Tibor Farkas: Derivation of Executable Test Models from Embedded System Models using Model Driven Architecture Artefacts - Automotive Domain -. MBEES 2006.
J. P. Bodeveix, R. Bouaziz, and O. Kone. Test method for embedded real-time systems. WSIDES 2005.
Kim G. Larsen, Marius Mikucionis, Brian Nielsen, and Arne Skou. Testing real-time embedded software using UPPAAL-TRON: an industrial case study. EMSOFT 2005.
Mirko Conrad, Ines Fey, and Sadegh Sadeghipour. Systematic Model-Based Testing of Embedded Automotive Software. Electron. Notes Theor. Comput. Sci. 111 (January 2005),
Saddek Bensalem, Marius Bozga, Moez Krichen, and Stavros Tripakis. Testing Conformance of Real-Time Applications by Automatic Generation of Observers. Electron. Notes Theor. Comput. Sci. 113 (January 2005),
Moez Krichen , Stavros Tripakis. Black-box conformance testing for real-time systems. SPIN 2004.
Kim G. Larsen , Marius Mikucionis , Brian Nielsen, Paul Pettersson, and Arne Skou. Online Testing of Real-Time Systems Using uppaal. FATES 2004. [Slides]
Marius Mikucionis, Kim G. Larsen, and Brian Nielsen. T-UPPAAL: Online Model-based Testing of Real-Time Systems. ASE 2004.
M. Conrad, I. Fey, and S. Sadeghipour. Systematic model-based testing of embedded control software: the MB3T approach. EE Digest / Volume 2004 / Issue 914
Abdeslam En-Nouaary, Rachida Dssouli, and Ferhat Khendek. Timed Wp-Method: Testing Real-Time Systems. IEEE Trans. Softw. Eng. 28, 11 (November 2002),
V. Braberman, M. Felder, and M. Marre. Testing timing behavior of real-timesoftware. In International Software Quality Week, 1997.
Duncan Clarke and Insup Lee. Automatic Test Generation for the Analysis of a Real-Time System: Case Study. In Proceedings of the 3rd IEEE Real-Time Technology and Applications Symposium (RTAS '97) (RTAS '97).

Regression Testing

Matt Netkow and Dennis Brylow. Xest: An Automated Framework for Regression Testing of Embedded Software. WESE 2010.

Environment Modeling

Wenchao Li, Lili Dworkin, and Sanjit A. Seshia. Mining Assumptions for Synthesis. MEMOCODE 2011
Sanjit A. Seshia and Alexander Rakhlin. Quantitative Analysis of Systems Using Game-Theoretic Learning. ACM TECS 2011
Sanjit A. Seshia. Quantitative Analysis of Software: Challenges and Recent Advances. FACS 2010
Sanjit A. Seshia. Environment Modeling: A Usability Challenge for Verifying Cyber-Physical Systems Workshop on Usable Verification 2010.
Jewgenij Botaschanjan and Benjamin Hummel. Specifying the worst case: orthogonal modeling of hardware errors. ISSTA 2009.
Jakob Engblom, Bengt Werner, and Guillaume Girard. Testing Embedded Software using Simulated Hardware, ERTS 2006
- Marco A.A. Sanvido, Vaclav CechtickyWalte. Testing embedded control systems using hardware-in-the-loop simulation and temporal logic. 15th Triennial World Congress 2002. [Slides]

Debugging

Padma Iyenghar, Clemens Westerkamp, Juergen Wuebbelmann, and Elke Pulvermueller. A model based approach for debugging embedded systems in real-time. EMSOFT 2010.
Vinaitheerthan Sundaram, Patrick Eugster, and Xiangyu Zhang. Lightweight tracing for wireless sensor networks debugging. MidSens 2009.
Jing Yang, Mary Lou Soffa, Leo Selavo, and Kamin Whitehouse. Clairvoyant: a comprehensive source-level debugger for wireless sensor networks. SenSys 2007. [Slides]
Peter Zoeteweij, Rui Abreu, Rob Golsteijn, and Arjan J. C. van Gemund. Diagnosis of Embedded Software Using Program Spectra. ECBS 2007

Resources