Documentation

To generate our FakeAVCeleb, we gathered real videos from the VoxCeleb2[1] dataset, where VoxCeleb2 consists of real YouTube videos of 6,112 celebrities. And these videos contain interviews of celebrities and the speech audio spoken in the video. We chose 500 videos from VoxCeleb2, one video for each celebrity. 

We selected videos based on gender, ethnicity, and age. The individuals in the real video set belong to four different ethnic groups, Caucasian, Black, Asian (Southern), and Asian (Eastern). Each ethnic group contains 100 real videos of 100 celebrities. The male and female ratio of each ethnic group is 50%, i.e., 50 videos of men and 50 videos of women out of 100 videos.

And then, we incorporated 500 unique real videos to our FakeAVCeleb as a real baseline-video set, each belonging to a single individual, with an average of 6.4 seconds duration. Since we are focusing on a specific and practical usage of deepfakes, each video was selected based on some specific criteria, there is a single person in the video with a clear and centered face, and he or she is not wearing any hat, glasses, mask or anything that might cover the face.

Overall

We used face-swapping methods, Faceswap[2], and FSGAN [4], to generate swapped deepfake videos. To generate cloned audios, we used a transfer learning-based real-time voice cloning tool (SV2TTS) [5] (see Figure 2). After generating fake videos and audios, we apply Wav2Lip [6] on generated deepfake videos to reenact the videos based on generated fake audios. On the other hand, the Real-Time Voice Cloning tool (RTVC) [5] was used for synthetic cloned voice generation.

This type of FakeAVCeleb dataset is minacious as an attacker can generate fake video and fake audio and impersonate any potential target person. This type of deepfake dataset can be used for training a detector for deepfake video and deepfake audio datasets.

Synthesis methods

we use a facial recognition service called Face++ [7], which measures the similarity between two faces. The similarity score helps us find the most similar source and target pairs, resulting in more realistic deepfakes. We selected the top 5 videos with the highest similarity score. After calculating the similarities, each video was synthesized with the chosen five videos by applying synthesis methods to produce high-quality realistic deepfakes.

Generation

While inspecting the generated videos, we filter the videos based on these criteria: 

1) The resulting fake video must be of good quality and realistic, i.e., hard to detect through the human eye.

2) The synthesized cloned audio should also be good.

3) The video and corresponding audio should be lip-synced.

Since we apply the manual screening process on synthesized videos, the final video count is more than 20,000.

We found that some of the synthesis methods, FSGAN and Wav2Lip, resulted in many fake videos with excellent and realistic quality. Meanwhile, FaceSwap resulted in several defective videos, since they are sensitive to different lightning conditions and require excessive time and resources to train.

Since we are generating cloned voices along with the fake video, we can create four possible combinations of audio-video pairs (see Table).

• Real-Video with Real-Audio : the samples from our base dataset, which contains real videos with real audio

• Real-Video with Fake-Audio : developed using the cloned speech of the target speaker

• Fake-Video with Real-Audio : representing the existing deepfakes benchmark datasets such as FF++

• Fake-Video with Fake-Audio : The main contribution of this work