6-minute read · Embedded systems and resilience
Imagine an embedded controller that samples a sensor, updates an internal estimate, and commands an actuator. During one update, a bit in a working register changes. The controller might overwrite the affected value before using it. It might detect an inconsistency and recover. Or it might continue running while issuing an incorrect command.
These are hypothetical outcomes, but they expose a practical engineering question: what evidence would distinguish a resilient system from one that merely keeps executing?
Fault injection makes that question experimentally accessible. By deliberately introducing a specified disturbance and observing the response, engineers can investigate how errors propagate and whether protective mechanisms work. The value of the experiment depends on the connection between the injected disturbance, the system model, and the service being evaluated.
The dependability taxonomy developed by Avižienis and colleagues distinguishes a fault, its resulting error, and a service failure. A fault is an attributed cause of an error; an error is an incorrect part of system state that may lead to failure; a failure occurs when delivered service departs from correct service. The distinctions matter because an internal error need not become externally visible. [1]
In the controller example, an injected register corruption represents an erroneous state associated with a hypothesized hardware fault. If the value is overwritten before affecting computation, the disturbance may leave no observable consequence. If it changes an actuator command beyond the allowed tolerance, the service has failed—even if the processor never crashes.
This makes the service requirement the starting point. Correctness may include numerical accuracy, timely delivery, and an acceptable recovery path. “The program completed” is too weak an acceptance criterion when the application must also act correctly and on time.
Different injection techniques answer different questions. LLFI, for example, injects at LLVM’s intermediate representation, allowing experiments at selected program points. Its published evaluation showed that choices such as instruction type and register target affected observed outcomes. Those findings apply to the evaluated programs and model; they are a reason to document injection choices carefully. [2]
An emulator offers another perspective. Höller and colleagues demonstrated a QEMU-based framework for examining software countermeasures using modeled faults affecting memory, registers, and instruction execution on embedded processors. This enables controlled system-level investigation without requiring a detailed hardware design. [3]
However, an abstraction can omit behavior that matters. In their REFINE study, Georgakoudis and colleagues showed how high-level compiler representations can miss machine-level injection opportunities. Their compiler-backend approach was evaluated against binary-level injection in high-performance computing programs. The relevant lesson for embedded research is methodological: agreement at one abstraction level should be demonstrated, rather than assumed to transfer to another. [4]
Physical experiments help test the assumptions beneath those models. Moro and colleagues investigated electromagnetic disturbances on a particular 32-bit microcontroller and analyzed effects on instruction and data transfers. Their results support developing fault models from observed device behavior; they do not establish a universal model for every processor or disturbance. [5]
A useful research strategy follows from these studies: use controllable models to explore mechanisms, then investigate whether the important modeled effects correspond to the target hardware. A convenient bit flip is a hypothesis about a disturbance’s consequences, not a complete description of its physics.
An alarm is an intermediate event. A controller can detect corruption and still miss its deadline, restore damaged state, or repeatedly restart. Conversely, some internal disturbances may never affect the required service. Reporting only detection counts hides these differences.
Arlat and colleagues’ fault-injection methodology explicitly connects the selected faults and activation conditions to observations and derived measures. That structure remains useful: measurements should be designed around the question the experiment is meant to answer. [6]
For a controller study, I would record several dimensions separately: whether the disturbance was successfully introduced; whether detection occurred; whether outputs remained acceptable; whether timing requirements were met; and whether recovery restored useful service. These are proposed reporting dimensions, not a claim that the cited studies used this exact scheme.
This separation also clarifies silent data corruption: an incorrect result can escape the system’s own detection while an external checker identifies it. The checker therefore needs an explicit acceptance rule. Depending on the application, that might compare exact outputs, numerical tolerances, or a required sequence of actions.
Every percentage needs a denominator. A detection fraction among successfully injected experiments differs from a fraction among all attempted injections. Neither directly gives the probability of failure per operating hour. Connecting experimental outcomes to field reliability requires additional assumptions and evidence about actual fault occurrence and operating conditions.
The following workflow is a practical synthesis for an engineering study, rather than a standardized qualification procedure.
Begin with a falsifiable question. For example: under a stated corruption model, does a proposed check prevent unacceptable actuator commands within the required response time? Specify what evidence would count against the mechanism.
Establish the baseline. Record expected behavior without injection across representative workloads. Retain the program version, build settings, inputs, and acceptance criteria. The baseline should reveal normal variation before that variation is mistaken for a fault effect.
Describe the sampled universe. State the eligible locations, execution phases, disturbance duration, and selection method. Uniform sampling over executed instructions is a different experiment from uniform sampling over elapsed time. Include workload boundaries and recovery paths when they matter to the question.
Preserve each experiment’s identity. Save the actual injection location and event, configuration, workload, observations, and classification. Record random seeds where applicable, but also preserve the selected events: a seed alone cannot compensate for a changed execution environment.
Make uncertainty visible. Report sample sizes and appropriate uncertainty intervals for estimated proportions. Keep failed injections and incomplete observations visible. Investigate unusual cases instead of hiding them within a single average, and reserve additional workloads to test whether a proposed improvement generalizes.
A useful campaign should change a design decision. If corruption reaches an output before a periodic check executes, an earlier check may be worth evaluating. If detection works but recovery violates the response requirement, recovery deserves attention. If errors appear harmless, determine whether they were actually exercised and observed before crediting the protection.
One flipped bit can illuminate a surprisingly long chain: computation, detection, recovery, timing, and the service delivered to the outside world. Following that chain carefully turns fault injection into a way to build—and justify—more resilient embedded systems.
Avižienis, A., Laprie, J.-C., Randell, B., & Landwehr, C. (2004). “Basic Concepts and Taxonomy of Dependable and Secure Computing.” IEEE Transactions on Dependable and Secure Computing, 1(1), 11–33. DOI: 10.1109/TDSC.2004.2
Lu, Q., Farahani, M., Wei, J., Thomas, A., & Pattabiraman, K. (2015). “LLFI: An Intermediate Code-Level Fault Injection Tool for Hardware Faults.” IEEE International Conference on Software Quality, Reliability and Security (QRS), 11–16. DOI: 10.1109/QRS.2015.13
Höller, A., Krieg, A., Rauter, T., Iber, J., & Kreiner, C. (2015). “QEMU-Based Fault Injection for a System-Level Analysis of Software Countermeasures Against Fault Attacks.” 18th Euromicro Conference on Digital System Design (DSD), 530–533. DOI: 10.1109/DSD.2015.79
Georgakoudis, G., Laguna, I., Nikolopoulos, D. S., & Schulz, M. (2017). “REFINE: Realistic Fault Injection via Compiler-Based Instrumentation for Accuracy, Portability and Speed.” International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’17), Article 29. DOI: 10.1145/3126908.3126972
Moro, N., Dehbaoui, A., Heydemann, K., Robisson, B., & Encrenaz, E. (2013). “Electromagnetic Fault Injection: Towards a Fault Model on a 32-bit Microcontroller.” Workshop on Fault Diagnosis and Tolerance in Cryptography (FDTC), 77–88. DOI: 10.1109/FDTC.2013.9
Arlat, J., Aguera, M., Amat, L., Crouzet, Y., Fabre, J.-C., Laprie, J.-C., Martins, E., & Powell, D. (1990). “Fault Injection for Dependability Validation: A Methodology and Some Applications.” IEEE Transactions on Software Engineering, 16(2), 166–182. DOI: 10.1109/32.44380