Partially Deepfake Audio Localization

Introduction

Recent advances in speech synthesis and voice conversion have enabled the generation of increasingly realistic human-like speech. However, the growing accessibility of these technologies also raises serious security concerns, as malicious attackers can manipulate semantically critical content while leaving most of the original audio unchanged. Such partially deepfake audio is more covert than fully synthesized speech and can substantially alter the intended meaning of an utterance, posing significant threats to information authenticity and real-world security. Therefore, partially deepfake audio localization aims to determine the authenticity of each frame in an input utterance and precisely identify the manipulated regions.

Method

• Dual-Relation Graph Neural Network

The Dual-Relation Graph Neural Network (DRGNN) models relationships among speech frames from two distinct perspectives. The class-relation branch aggregates information from frames that share the same authenticity class, whereas the segment-relation branch captures the temporal coherence among consecutive frames belonging to the same segment.

• Segment-Level Contrastive Loss

The Segment-Level Contrastive Loss (SCL) constrains frame representations from a segment-level perspective. It first aggregates frames belonging to the same segment and then pulls segments with the same authenticity class closer while pushing segments with different classes farther apart. By constructing contrastive relationships across utterances, SCL further improves the discriminability and generalization of the learned representations.

Experiment Result

The results on the PS dataset under the 20 ms frame-level setting demonstrate that the proposed DGRL achieves the best overall performance among all compared methods.

Publication