Dataset

Figure 1: A scheme that shows the process of the creation of CONAN-MT-SP.

As we can see in Figure 1, to build CONAN-MT-SP, we use the hate speech of the English MultiTarget CONAN (CONAN-MT) corpus (Fanton et al. 2021) that collected its HS-CN pairs by niche sourcing from two different NGOs and subsequently used these pairs to generate more HS-CN with GPT-4 with human review integrated into the process. Due to the fact that the hate speech message is in English in CONAN-MT, we translate it into Spanish using the DeepL API. All translations were reviewed by our annotators, and in those pairs where the translations were erroneous, they were edited. The associated counternarrative (CN) to each hate-speech message (HS) is generated by the GPT-4 model using a prompt strategy. The strategy used consisted in a Few Shot Learning Strategy, where the model was prompted with a task description and 8 examples of HS-CN pairs (one for each target). In addition, the counternarrative generated by GPT-4 has been evaluated by human experts using different metrics: 

• Offensiveness:

  • 0 (not sure)

  • 1 (not offensive)

  • 2 (maybe offensive)

  • 3 (completely offensive)

• Stance:

  • 0 (irrelevant)

  • 1 (strongly agree)

  • 2 (slightly agree/disagree)

  • 3 (strongly disagree)

• Informativeness:

  • 0 (irrelevant)

  • 1 (not informative)

  • 2 (generic and uninformative statement)

  • 3 (specific and informative)

• Truthfulness:

  • 0 (not sure)

  • 1 (not true)

  • 2 (partially true)

  • 3 (completely true)

• Editing required:

  • 0 (no editing)

  • 1 (yes editing)

• Comparison between H-M:

  • 0 (both CN are equally valid)

  • 1 (human generates a better CN)

  • 2 (machine generates a better CN)

  • 3 (neither CN is good)

In RefutES, we selected from this corpus the “perfect” counter-narratives, i.e., those that are non-offensive, in complete disagreement, specific and informative, compellingly truthful, do not need editing, and are equal to or better than the initial CONAN-MT counter-narrative. The corpus is divided into three subsets, each related to a different part of the competition:

• Train split: contains 2496 HS-CN pairs.

• Dev split: contains 279 HS-CN pairs.

• Test split: contains 156 pairs HS-CN. 78 HS-CN pairs are generated by GPT-4 and manually annotated by humans and the others 78 HS-CN pairs generated by humans.

The refutES corpus is composed by the followig features that are the columns in the provided CSVs:

• Hate-speech: contains the hate speech message.

• Counter-narrative: contains the counternarrative associated to the hate-speech message that is generated by GPT-4.

• Target: contains the collective affected by the hate message. It can be disabled, Jews, LGBT+, migrants, Muslims, people of colour, women and other groups.

The data associated for this task will be provided to the participants following the established dates.

Now, the dev, train and TEST splits are available at https://github.com/sinai-uja/RefutES.git