Motivation

We first investigate whether abnormal behaviors can be distinguished from normal behaviors in the high-dimensional hidden space. We collected three sets of samples, each consisting of 400 pairs, corresponding to three different threat models: Jailbreak (Normal vs. Attack), Hallucination (Correct vs. Incorrect), and Backdoor (Clean vs. Poisoned). We extracted hidden states from the Attention and MLP sublayers of the last transformer block and employed t-Distributed Stochastic Neighbor Embedding (t-SNE) to visualize the manifold structure. Additionally, we calculated Representational Similarity Analysis (RSA) metrics to quantify the correlation distances between different classes.

The primary observation is that abnormal inputs (red points) generally map to distinct regions of the manifold compared to normal inputs (blue points). This geometric separability is consistently supported by the RSA metrics across the majority of tasks, where the inter-class distance (e.g., Normal-Jailbreak) is measurably larger than the intra-class distance (e.g., Normal-Normal). For instance, in the Jailbreak scenario, the distance between normal and attack queries (0.333 ± 0.053) is roughly double the distance within normal queries (0.167 ± 0.055), indicating a clear decision boundary in the high-dimensional space.

Secondarily, we observe variations in separability between different component types. While both Attention and MLP layers distinguish strong adversarial patterns effectively, MLP layers tend to exhibit more compact clustering and reduced intra-class variance. This is particularly noticeable in the Hallucination task, where the MLP representations provide a clearer distinction between correct and incorrect answers compared to the Attention layers.

While Study I confirms separability, it treats the model as a black box regarding layer contribution. To understand where this divergence occurs, we draw inspiration from coverage metrics in deep learning security testing. The core premise is that abnormal and normal inputs trigger distinct processing pathways, effectively covering different regions of the model's internal state space. Aligning with metrics such as Neuron Coverage and Top-K Neuron Patterns, we utilize the count of active neurons as a proxy for pathway activation. We collected 100 normal and 100 attack queries and calculated the activation ratio (Attack/Normal) for each layer to map how processing dynamics shift across the model architecture.

Figure illustrates the activation ratios across the model's depth. The data reveals a stark heterogeneity: the model's response to anomalies is not uniformly distributed. While many layers (shown in blue) maintain stable activation levels regardless of input type, specific blocks (highlighted in orange) exhibit a dramatic surge in activity when processing attacks, with ratios spiking significantly. This indicates that the "abnormality" is not diffused evenly but is localized in specific processing stages. These layers with high activation ratios are likely where the model reacts most strongly to attacks, such as by bypassing safety checks.