Final Project: Debiasing Word Embeddings


Katya Rebholz & Olivia Ritz

Results From The Original Paper

The original paper, “Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings,” discusses a method for reducing gender stereotypes in text data through modification of word embeddings (1). Note: a word embedding is numerical vector representation of a word, that can describe semantic meaning and provide information on similarity between words.

The authors describe their method, a hard-debiasing algorithm, and they conclude from empirical results that the algorithm significantly reduces gender bias in word embeddings trained on Google News articles. Additionally, they conclude that the algorithm preserves the embedding’s ability to group relationships and accurately form analogies.

The main procedure for the experiment was as follows:

  • Use a word embedding to create analogies that solve the following task: 'she' is to x as 'he' is to y
  • Run the hard-debiasing algorithm to remove gender bias in the word embedding
  • Use the modified word embedding to solve the same analogy task, then analyze the results

The results of the algorithm were demonstrated by comparing crowd-worker evaluation of analogies formed before and after debiasing. The proportion of analogies that were judged by a majority of the workers as showing gender stereotypes was reduced from 19% to 6%.

The associated source code is provided in the Github repository at reference (4). The main files of interest, and the ones that were used in our reproduction discussed below, are we.py and debias.py.


Procedure to Reproduce Results

We use the same software utilized by the original research paper to create and debias analogies from a word embedding text file (4). However, we replace the Google News corpus dataset used in the original experiment with word embeddings trained by Stanford’s GloVe algorithm on data aggregated from 2014 Wikipedia articles (2)(5).

The Wikipedia dataset was trimmed to contain only non-numerical English words by running our subset_data.py script (7). This script loads the desired word embedding text file and an English json dictionary file and then writes each word to a new text file only if it is not a number and it exists in the json dictionary (6). We then run our experiment.py script which loads the trimmed embedding file, generates the analogies as done in the original experiment, runs the debiasing algorithm, and then generates a new list of analogies (7). The script then randomly shuffles the biased and debiased analogies and writes them to two textfiles: labeled.txt and not_labeled.txt. The labeled.txt file contains the list of shuffled anaologies along with their label: "b" for biased or "db" for debiased. The non_labeled.txt file contains the same list of shuffled analogies without their corresponding label. This file will be used for us to objectively cast a vote on whether or not an analogy contains gender bias. Finally, after we label each analogy in the non_labeled.txt file, we run analogy_counter.py which reads in both labeled.txt and the modified non_labeled.txt to calculate how many analogies from the biased and debiased sets were voted as containing gender bias (7).