Academia is based on merit —
And women are just not good enough
Research
Submitting a paper to conferences or journals or getting prompted based on publication records, being male seems to be an advantage. While older studies did not find many differences in acceptance for journal publications by gender, newer studies try to account for the quality of the research. Thus, studies published in the last years seem to point into the direction that women are held to higher standards than men when assessing their research. When it comes to citations and self-citations whether gender differences exist or not is difficult to conclude from the state of the art literature.
Peer review process & promtions based on research
International study on an experiment testing single vs. double-blind submissions to an academic conference in Web Search and Data Mining with an 15.6% acceptance rate. 500 submissions were reviewed by about 1,000 double-blind reviewers and 1,000 single-blind reviewers. Each submission is simultaneously scored by two single-blind and two double-blind reviewers. The analysis shows that single-blind reviewing confers a significant advantage to papers with famous authors and authors from high-prestige institutions. The influence of author gender on reviewing behavior is not statistically significant. However, the estimated effect size is non-negligible.
International study of paper submission to four economics journals with in total 29,890 submissions during the period 2003-2013. About half of the submitted papers (15,147) were not desk-rejected and were assigned to at least two referees. First, the authors compare the relative recommendations of female and male referees in assessing female — versus male — authored papers. The difference-in-differences is precisely estimated and close to zero, suggesting that the differential biases of male referees are negligible. The authors furthermore use different regression models to test whether referee recommendations and editorial decisions are gender-biased, in particular, holding women to higher standards. Therefore the authors compare citations for female and male-authored papers, holding constant the referee evaluations and other characteristics, including prior publications of the authors. The authors find that female-authored papers receive about 25% more citations than observably similar male-authored papers. Referees of both genders appear to set a higher bar for female-authored papers. Editors largely follow the referees, resulting in a 7 percentage points lower probability of a revise and resubmit verdict for female-authored papers, relative to a citation-maximizing benchmark. In their desk rejection decisions editors treat female authors more favorably, though they still impose a higher bar than would be expected under citation-maximization. Also, no significant differences in the time that referees or editors take in reaching decisions about female-authored versus male-authored papers were found.
International three week summer conference in economics where papers have to be submitted and accepted . Using anonymized data on submissions (6,867 papers and 17,474 authors and discussants), the authors examine several channels potentially affecting female representation including gender differences in acceptance and submissions rates, institutional rank, NBER affiliation, faculty seniority and the role of organizers. The authors find that no difference in acceptance rates by gender, but there seems to be gender differences in submission rates, which drive overall representation.
International study of about 9,000 economic article abstracts in the top 4 journals. As a measure of quality, the author uses the five most widely used readability scores assessing simple vocabulary and short sentences which are easier to understand. Conditional on this measure of quality of a paper, women are held to higher writing standards in academic peer review: (i) female-authored papers are 1–6 percent better written than equivalent papers by men; (ii) the gap widens during peer review; (iii) women improve their writing as they publish more papers (but men do not). Furthermore, the author documents evidence that the cost to women of publishing a paper is much higher than it is for men: female authors spend 3–6 months longer under review. The author concludes that women gradually anticipate higher writing standards in peer review by writing in a more readable manner, suggesting they probably respond to biased treatment in ways that inadvertently disguise it as a voluntary choice.
The study employs a probit regression making use of top U.S. research universities' tenure decisions (n=574) in economics based on publication records of the candidates. The idea is to investigate whether credit for group work allocated when individual contributions are not perfectly observed; and whether it differs by gender. By using data from academic economists’ CVs, the author tests if coauthored and solo-authored publications matter differently for tenure for men and women. Because coauthors are listed alphabetically in economics, coauthored papers do not provide specific information about each contributor’s skills or ability. Solo-authored papers, on the other hand, provide a relatively clear signal of ability. The study finds that men are tenured at roughly the same rate regardless of whether they coauthor or solo-author. Women, however, become less likely to receive tenure the more they coauthor. The result is most pronounced for women coauthoring with men and less pronounced among women who coauthor with other women. Women who write all of their papers alone have similar tenure rates to men. However, women who coauthor all of their papers have an approximately 40% tenure rate, substantially lower than that of men who coauthor all of their papers (75%). The slope for women is 0.37 and is statistically significant at the 5% level. To ensure to not pick up on ability differences between men and women, the author controls for the quality of papers using both journal rankings and citations, allowing for a comparison of men and women with similar research portfolios. The results are also robust to including other individual-level controls such as length of time to tenure and the seniority of one’s coauthors, as well as tenure year, tenure institution, and primary field fixed effects. In sociology, a discipline in which coauthors are listed in order of contribution, and find that when contributions are made clear, men and women receive equal credit for coauthored papers.
Study using a probit model to test if the acceptance of papers to three European and Spanish economic conferences (n=16,154) is biased by gender. The authors find that all-female-authored papers are 3.2 p.p. (6.8%) less likely to be accepted than all-male-authored papers. This gap is present after controlling for (i) number of authors, (ii) referee fixed effects, (iii) field, (iv) cites of the paper at submission year, (v) previous publication record of the authors, and (vi) the quality of the affiliations of the authors. The authors also find that the gap is entirely driven by male referees—female referees evaluate male and female-authored papers similarly, but male referees are more favorable towards papers written by men.
Survey study amongst early-career economists (n=562), investigating whether the gender gap in economic publications can be explained by different submission strategies of male and female economists. The online survey among early-career economics faculty of top 50 institutions focused on the submission trajectories of job market papers as well as personal and institutional characteristics. Results suggest that there are no significant differences in submission strategies for this early-career sample.
US study uses data (n= 798 assistant professor faculty members in economics departments ) from Academic Analytics to examine gender differences in promotion to associate professor in economics. The authors found that women in economics were 15% less likely to be promoted to associate professor after controlling for cumulative publications, citations, grants and grant dollars. In contrast, they found no significant gender differences in promotion in other fields including biomedicalscience, physical science, political science, mathematics and statistics, and engineering. They separated the sample by the research intensity of institutions and found suggestive evidence that these resultswere being driven by less research-intensive institutions.
Self citations
The research on the number of self-citations is rather new and still developing rapidly. Starting with a seminal paper with a large scale dataset of King et al. in 2017 found indications that men do cite their own works more often than women do (similar findings in smaller studies in tourism by Nunkoo (2019) et al and Deschat & Maes (2017) in management journals). However, the authors are very careful with their conclusion because for their main dataset they could not control appropriately for the total number of papers the researchers had published before. Mishra et al. (2018) used a dataset in biomedicine and came to contradicting findings when accounting for prior publication count. Similarly, Azoulay & Lynn (2020) did not find any differences in self-citation of male and female life science researchers when accounting for their past productivity and the likelihood of self-citing intellectually distant material .
With a different focus Lerchenmueller et al. (2019) focus on the positive presentation of author's own work in life science publications. The authors find that especially in high impact journals men do use more positive words to describe their own research.
King, M. M., Bergstrom, C. T., Correll, S. J., Jacquet, J., & West, J. D. (2017). Men set their own cites high: Gender and self-citation across fields and over time. Socius, 3, 2378023117738903.
The authors use a data set of 1.5 million research papers in the scholarly database JSTOR published between 1779 and 2011 and find that nearly 10 percent of references are self-citations by a paper’s authors. The findings also show that between 1779 and 2011, men cited their own papers 56 percent more than did women. In the last two decades of data, men self-cited 70 percent more than women. Women are also more than 10 percentage points more likely than men to not cite their own previous work at all. While these patterns could result from differences in the number of papers that men and women authors have published rather than gender-specific patterns of self-citation behavior, this gender gap in self-citation rates has remained stable over the last 50 years, despite increased representation of women in academia. The authors break down self-citation patterns by academic field and number of authors and comment on potential mechanisms behind these observations.
The study assesses cross‐cultural differences in self‐promotion by comparing the self‐citation behaviour of scholarly authors originating from individualist and collectivist cultures, using original data on 1,346 journal articles published between 2009 and 2014 in the fields of Management and Business. The main finding is that articles by authors from individualist cultures are about twice as likely to contain many self‐citations. Results confirm the presence of a gender gap in self‐citations, but this effect is smaller than the cultural effect and that the effect appears to be stable across cultures. These findings show that the structure of rewards and costs associated with particular self‐promotion tactics differ from culture to culture.
In this study the authors explore gender gaps and differences in citation practices of scholars in the top-cited articles in tourism research (n= 431 articles and n = 1031 authors). The results suggest that male researchers dominate the authorship of those articles and are more likely to engage in self-citation than females. The study also finds a disparity in citation counts between male- and female-authored articles. Controlling for other factors, author gender is an important determinant of citation counts. The authors conclude that a more gender conscious citation practices and provides potential gender-based interventions to reduce the citation gap. The research raises awareness about the dangers of the perfunctory use of citations and paves the way for further debates on the politics and embedded inequalities of citations in tourism research.
This study replicates the study of King et al. (2017) with a sample of 1.6 million papers from Author-ity, a version of PubMed (mainly covering research in biomedicine) with computationally disambiguated author names. the authors show that the gender effect largely disappears when accounting for prior publication count in a multidimensional statistical model. Gender has the weakest effect on the probability of self-citation among an extensive set of features tested, including byline position, affiliation, ethnicity, collaboration size, time lag, subject-matter novelty, reference/citation counts, publication type, language, and venue. The authors conclude that self-citation is the hallmark of productive authors, of any gender, who cite their novel journal publications early and in similar venues, and more often cross citation-barriers such as language and indexing. As a result, papers by authors with short, disrupted, or diverse careers miss out on the initial boost in visibility gained from self-citations. The data of this study further suggest that this disproportionately affects women because of attrition and not because of disciplinary under-specialization.
This study focussing on life science researchers (n= 3,667) extends the findings of King et al. (2017) and Mishra et al. (2018) which arrived at conflicting conclusions. The authors state that in science, self-citation is often interpreted as an act of self-promotion that (artificially) boosts the visibility of one’s prior work in the short term, which could then inflate professional authority in the long term. In this study 36 cohorts of life scientists (1970–2005) followed through 2015 (or death or retirement) by tracking the rate of self-citation per unit of past productivity and the likelihood of self-citing intellectually distant material and the rate of return on self-citations with respect to a host of major career outcomes, including grants, future citations, and job changes. With this comprehensive, longitudinal data, the authors do not find evidence whatsoever of a gender gap in self-citation practices or returns. The authors conclude that men may very well be more aggressive self-promoters than women, but this dynamic does not manifest in our sample with respect to self-citation practices.
This paper uses a retrospective observational study looking at titles and abstracts from 101 720 clinical research articles and approximately 6.2 million general life science articles indexed in PubMed. Articles in which both the first and last author were women used at least one of the 25 positive terms to describe the own work in 10.9% of titles or abstracts versus 12.2% for articles involving a male first or last author, corresponding to a 12.3% relative difference (95% CI 5.7% to 18.9%). Gender differences in positive presentation were greatest in high impact clinical journals (impact factor >10), in which women were 21.4% less likely to present research positively. Across all clinical journals, positive presentation was associated with 9.4% (6.6% to 12.2%) higher subsequent citations, and in high impact clinical journals 13.0% (9.5% to 16.5%) higher citations. Results were similar when broadened to general life science articles published in journals indexed by PubMed, suggesting that gender differences in positive word use generalize to broader samples. The authors conclude that clinical articles involving a male first or last author were more likely to present research findings positively in titles and abstracts compared with articles in which both the first and last author were women, particularly in the highest impact journals. Positive presentation of research findings was associated with higher downstream citations.
Citations
The research on the number of citations per paper for female and male authors is very ambigouse. We present some studies here, but our overview is far form compliete. Some studies find a positive gender bias for female authored papers, some find a negative and some find non. Results seems to differs across diciplines, time and location. A nice overview of the state of the art literature findings is given by Lynn et al. We would further recommend reading Koffis paper as she uses a rather new and interesting approach in measuring how much a paper should have been cited! Also Hengel & Moon on the effect of co-author gender is strongly recommend.
Study starts with a very helpful review of the literature and then analyzing about 10,000 publications in economics, political science, and sociology, of a gender bias in ciations exist. The authors wonder why in academia, women trail men in nearly every major professional reward, such as earnings, publications, and funding. However, bibliometric studie suggest that citations are unique with regard to gender inequality: female penalties have been reported, but gender parity or even female premiums are routinely documented as well. Two questions follow from this puzzle. First, does gender matter for citations in sociology and neighboring social science disciplines? No theoretically informed study of gender and citations exists for the social science core. Thus the authors begin to fill this gap by analyzing roughly 10,000 publications in economics, political science, and sociology. In contrast to many big data studies, this study estimates the effect of author gender on citations alongside other author-, article-, journal-, and (sub)field-level predictors. Results strongly suggest that when male and female authors publish articles that are comparably positioned to receive citations, their publications do in fact accrue citations at the same rate. This finding raises a second question: Why would gender matter “everywhere but here”? The authors hypothesize that the answer is related to the mechanisms (e.g., self-selection, biased assessments of commitment) that are activated in the context of some professional rewards but not citations. Rhey discuss why a null gender finding should not be discarded as an anomaly but rather approached as an analytical opportunity.
Koffi, M. (2019). Innovative Ideas and Gender Equality. Discussion paper
Study on citations in economic journals. Koffi analyzes the recognition of women's innovative ideas by using bibliometric data from research in economics to investigate gender biases in citation patterns. Based on deep learning and machine learning techniques, the paper (1) establishs the similarities between papers and (2) builds a link between articles by identifying the papers citing, cited and that should be cited. The study finds that, on average, a paper omits almost half of related prior papers. There are, however, substantial heterogeneities among the authors. In fact, omitted papers are 15% to 30% more likely to be female-authored than maleauthored. First, the most likely to be omitted are papers written by women (solo, mostly female team) working at mid-tier institutions, publishing in non-top journals. In a group of related papers, the probability of omission of those papers increases by 6 percentage points compared to men in similar affiliation when the citing authors are only males. Overall, for similar papers, having at least one female author reduces the probability of omitting other women's papers by up to 10 percentage points, whereas having only male authors increases the probability of being omitted by almost 4 percentage points. Second, the omission bias is twice as high in theoretical elds that involve mathematical economics than it is in applied elds such as education and health economics. Third, men benefit twice as much as women from publishing in a top journal, in terms of likelihood of being omitted. Lastly, being omitted with respect to past publications a ects future productivity and reduces the probability of getting published in a top journal. Finally, peer effects and more editorial board diversity tend to counteract and reduce the omission bias.
Hengel, E., & Moon, E. (2020). Gender and quality at top economics journals. Discussion Paper
Study focusing on articles (n=10,954) published in “top-five” economics journals by 7,574 unique authors. The study shows that articles authored by men are lower quality— as measured by citations—than articles those same journals publish by women. Additionally, the quality of a man’s paper rises when he co-authors with the opposite sex whereas the quality of a woman’s paper falls. Under strong assumptions, the findings imply top economics journals hold female-authored papers to higher standards and, as a result, fail to publish the very best research. They also suggest that authors of both sexes may forgo co-authoring opportunities with women in order to pursue less productive collaborations with men.
Study on who the number of citations differ for female and male first authors in the top five astronomical publications (n>200.000). The authors control for non-gender specific properties of the papers such as seniority of the first author, number of references, total number of authors, year of publication, publication journal, field of study and region of the first author’s institution. The study finds that papers authored by females receive 10.4±0.9% fewer citations than what would be expected if the papers with the same non-gender specific properties were written by the male authors. Female authors in the sample tend to self-cite more, but this effect disappears when controlled for non-gender specific variables.
In a matched case-control study (n=1,269,542) with papers in selected areas of medicine published between 2008 and 2014 the authors show that papers with female authors are, on average, cited between 6.5 and 12.6% less than papers with male authors. However, the standardized mean differences are very small, and the percentage overlaps between the distributions for male and female authors are extensive. Adjusting for self-citations, number of authors, international collaboration and journal prestige, the authors find near-identical per-paper citation impact for women and men in first and last author positions, with self-citations and journal prestige accounting for most of the small average differences.
Helmer, M., Schottdorf, M., Neef, A., & Battaglia, D. (2017). Gender bias in scholarly peer review. Elife, 6, e21718.
It is difficult to check for any bias in the peer-review process because the identity of peer reviewers generally remains confidential. This study uses public information about the identities of 9000 editors and 43000 reviewers from the Frontiers series of journals (142 journals in Science, Health, Engineering and the Humanities and Social Sciences) between 2007 and the end of 2015. The authors show that women are underrepresented in the peer-review process, that editors of both genders operate with substantial same-gender preference (homophily), and that the mechanisms of this homophily are gender-dependent. Furthermore they show that homophily will persist even if numerical parity between genders is reached, highlighting the need for increased efforts to combat subtler forms of gender bias in scholarly publishing.
The authors conducted an exploratory analysis of peer review outcomes of 23,876 initial submissions and 7,192 full submissions to the biosciences journal eLife between 2012 and 2017. Women and authors from nations outside of North America and Europe were underrepresented both as gatekeepers (editors and peer reviewers) and authors. The study finds evidence of a homophilic relationship between the demographics of the gatekeepers and authors and the outcome of peer review; that is, there were higher rates of acceptance in the case of gender and country homophily. The acceptance rate for manuscripts with male last authors is seven percent, or 3.5 percentage points, greater than for female last authors (95% CI = [0.5, 6.4]); this gender inequity is greatest, at nine percent or about 4.8 percentage points (95% CI = [0.3, 9.1]), when the team of reviewers is all male; this difference is smaller and not significantly different for mixed-gender reviewer teams. Homogeny between countries of the gatekeeper and the corresponding author is also associated with higher acceptance rates for many countries. To test for the persistence of these effects after controlling for potentially confounding variables, the authors conduct a logistic regression including document and author metadata. Disparities in acceptance rates associated with gender and country of affiliation and the homophilic associations remained.
Thelwall, M. (2020). Gender differences in citation impact for 27 fields and six English-speaking countries 1996–2014. Quantitative Science Studies, 1-19.
International study (Australia, Canada, Ireland, New Zealand, the UK, and the USA ) on gender differences in citation impact in different academic fields. Previous research about gender differences in journal article citation impact has found the direction of any difference to vary by country and field, but has usually avoided discussions of the magnitude and wider significance of any differences and has not been systematic in terms of fields and/or time. This study investigates differences in citation impact between male and female first-authored research for 27 broad fields from 1996 to 2014. The results show an overall female first author citation advantage, although in most broad fields it is reversed in all countries for some years. International differences include Medicine having a female first author citation advantage for all years in Australia, but a male citation advantage for most years in Canada. There was no general trend for the gender difference to increase or decrease over time. The average effect size is small, however, and unlikely to have a substantial influence on overall gender differences in researcher careers.
What are the explinations for these results? While a gender discirimination is found in several stages of the academic career it seem odd to not having these issues when it comes to citations. Grossbard, Yilmazer, Zhang, L. (2018) find themself a positive female gender gap in Demographic Economic Journals. They explaination for this might be that female authored papers in these field journals might be of higher qualitiy than male authored papers. Prior research showed that female papers are more likely to be rejected in top-journals (see Card et al. above) and this after benefitting from the review proces in the top-journal are higher quality when submitted to field journals. This explanation is supported by their finding that gender gap in citations favoring women is not found for authors with limited experience past graduate school. Therefore this supports the explanation for the gender gap based on authors’ prior experience with economics journals of higher rank. Lynn at al (2019) who do not find any gender differences in citation hypothises that this null finding just for citations is related to the mechanisms (e.g., self-selection, biased assessments of commitment) that are activated in the context of some professional rewards but not citations. They conclude that a null gender finding for citations should not be discarded as an anomaly but rather approached as an analytical opportunity.
Larivière, V., Ni, C., Gingras, Y., Cronin, B., & Sugimoto, C. R. (2013). Bibliometrics: Global gender disparities in science. Nature News, 504 (7479), 211.
International, cross-disciplinary bibliometric analysis (5,483,841 papers of 27,329,915 authors) of the relationship between gender and research output (proxy was authorship on published papers); the extent of collaboration (proxy was co-authorships); and the scientific impact of all articles published between 2008 and 2012 and indexed in the Thomson Reuters Web of Science databases (proxy was citations). The authors find that in the most productive countries, all articles with women in dominant author positions receive fewer citations than those with men in the same positions. And this citation disadvantage is accentuated by the fact that women's publication portfolios are more domestic than their male colleagues — they profit less from the extra citations that international collaborations accrue. Given that citations now play a central part in the evaluation of researchers, this situation can only worsen gender disparities.
Strumia, A. (2020) Gender issues in fundamental physics: A bibliometric analysis. Quantitative Science Studies. (forthcoming)
Bibliometric study with data about fundamental physics world-wide from 1970 to now extracting quantitative data about gender issues. The author does do not find significant gender differences in hiring rates, hiring timing, career gaps, and slowdowns, abandonment rates, citation, and self-citation patterns. Furthermore, the author finds that various bibliometric indicators (number of fractionally-counted papers, citations, etc) exhibit a productivity gap at hiring moments, at the career level, and without integrating over careers. The gap persists after accounting for confounding factors and manifests as an increasing fraction of male authors going from average to top authors in terms of bibliometric indices, with a quantitative shape that can be fitted by higher male variability.
Reply to Strumia (2020) re-analysing a subsample of solo-authored papers from Strumia's data, adjusting for year and journal of publication, authors’ research age and their lifetime “fame”. The re-analysis suggests that female-authored papers are actually cited more than male-authored papers. This finding is inconsistent with the “greater male variability” hypothesis Strumia (2020) proposes to explain many of his results. In the conclusion to his paper, Strumia states that “..dealing with complex systems, any simple interpretation can easily be incomplete…”. The austhors agree entirely. Strumia’s simple—and, more importantly, simplistic—analysis and interpretation are far from complete.
Invitations to research seminars
Doleac, J., Hengel, E., & Pancotti, E. (2021). Diversity in economics seminars: who gives invited talks? Forthcoming in AEA Papers and Proceedings. American Economic Association.
The paper describes the diversity of economics seminars. It summarizes both the type of research presented and the demographics and institutional background of the presenting economists. The analysis is based on a comprehensive database using a balanced panel of 66 economics and economics-adjacent departments from August 2014 through December 2019. The authors conclude by outlining several actionable steps departments can take to increase the diversity of their own seminar series.
Research Funding
Recent studies on gender effects in successful grant applications are rare and mixed. While one audit study does not find any gender differences in the assessment of the grant proposals, a second one making use of a natural experiment shows a disadvantage for female researchers when the person, rather than the proposed science is evaluated.
U.S. audit study with 48 modified versions of actual proposals for grants representing a broad range of science funded by the National Institute of Health. 421 reviewers were informed that the authors study the National Institute of Health's review process and that the researchers would evaluate modified versions of actual proposals. Each reviewer evaluated a set of three proposals as a primary reviewer. Two of the set of three proposals were presented as been written by white male applicants (one high quality and one moderate quality). The third proposal was either high or moderate quality and, depending on the experimental condition, was presented as been written by a white female, black male or black female applicant. To avoid suspicion as to the purpose of the study, no reviewer was asked to evaluate more than one proposal written by a non-white male applicant. This design was chosen to isolate the causal role of perceived applicants demographic characteristics on scores and written critiques, independent of the characteristics of reviewers or proposals.
The study finds little to no race or gender bias in initial evaluations, and additionally find that any bias that might have been present must be negligible in size. This conclusion was robust to a wide array of statistical model specifications. The authors suggest that important bias may be present in other aspects of the granting process, but the evidence suggests that it is not present in the initial round of the grant reviews.
The Canadian study makes use of a natural experiment in which the Canadian Institute of Health Research introduced two new grant programs. In one the quality of the proposed science was evaluated, in the other the focus is on the evaluation of the researcher. In total 23,918 grant applications from 7,093 unique researchers in a 5-year natural experiment are investigated to shed light on the question of why men often receive more research funding than women. Using Generalized Estimating Equations the authors show that the overall grant success rate across all competitions was 15.8%. After adjusting for age and research domain, the predicted probability of funding success in the program focused on the proposed science, the gap was 0.9 percentage points in favor of male researchers. In the program with an explicit review focus on the researcher, the gap was 4.0 percentage points in favor of male researchers. This study suggests gender gaps in grant funding are attributable to less favorable assessments of women as researchers, not differences in assessments of the quality of science led by women.
Teaching
“The Truth will set you free…but first it will piss you off” Gloria Steinem
Studies on teaching evaluations are rather clear: At different universities, in different countries, in different academic fields, using different research set-ups identifying causal effects, studies find gender bias against women in student evaluations of teaching.
The authors use non-parametric statistical tests and apply them to two datasets: 23,001 student evaluations of 379 instructors by 4,423 students in six mandatory first-year courses in social sciences in a natural experiment at a French university, and 43 student evaluations for four sections of an online course in a randomized, controlled, blind experiment at a U.S. university. Student evaluations are important as they are widely used in academic personnel decisions as a measure of teaching effectiveness. Student evaluations are biased against female instructors by an amount that is large and statistically significant. The bias affects how students rate even supposedly objective aspects of teaching, such as how promptly assignments are graded. The bias varies by discipline and by student gender, among other things. It is not possible to adjust for the bias, because it depends on so many factors. Student evaluations are more sensitive to students’ gender bias and grade expectations than they are to teaching effectiveness. Gender biases can be large enough to cause more effective instructors to get lower evaluations than less effective instructors.
In the French natural experiment, first-year students take the same six mandatory courses: history, macroeconomics, microeconomics, political institutions, political science, and sociology. Each course has one male professor who delivers the lectures to groups of approximately 900 students. Courses are divided into sections of 10–24 students, taught by male and female instructors. The instructors have considerable pedagogical freedom. Students enroll in sections of these courses. The enrollment process does not allow students to select individual instructors. The assignment of instructors to sets of students is as if at random, forming a natural experiment. Section instructors assign interim grades during the semester. Interim grades are known to the students before the students submit the student evaluation. Interim grades are thus a proxy for students' grade expectations. Final exams are written by the course professor, not the section instructors. Students in all sections of a course in a given year take the same final exams. Final exams are graded anonymously. To the extent that the final exam measures appropriate learning outcomes, performance on the final is a measure of the effectiveness of an instructor: in a given course in a given year, students of more effective instructors should do better on the final exam, on average, than students of less effective instructors.
In the U.S. randomized experiment, students in an online course were randomized into six sections of about a dozen students each, two taught by the primary professor, two taught by a female graduate teaching assistant, and two taught by a male teaching assistant. In one of the two sections taught by each assistant, the assistant used her or his true name; in the other, she or he used the other assistant's identity. Thus, in two sections, the students were led to believe they were being taught by a woman, and in two sections, they were led to believe they were being taught by a man. Students had no direct contact with assistants: the primary interactions were through online discussion boards. The assistants credentials presented to the students were comparable; the assistants covered the same material; and assignments were returned at the same time in all sections. Student evaluations included an overall score and questions relating to professionalism, respectfulness, care, enthusiasm, communication, helpfulness, feedback, promptness, consistency, fairness, responsiveness, praise, knowledge, and clarity.
Quasi-experimental dataset of 19,952 student evaluations of their teachers at the Business and Economics Faculty of Maastricht University in The Netherlands. The study can identify causal effects as students are randomly allocated to female or male instructors. The dataset includes students’ subjective evaluations of their teachers’ performance, as well as two objective measures of teachers’ performance: Students’ course grades, which are mostly based on centralized exams usually not graded by the section teacher and students’ self-reported number of hours spent on studying for the course. The results show that female faculty receive systematically lower teaching evaluations than their male colleagues. By contrast, neither students’ grades nor self-reported study hours are affected by the teacher’s gender. This bias is driven by male students’ evaluations, is larger for mathematical courses, and particularly pronounced for junior women. The gender bias in teaching evaluations may have direct as well as indirect effects on the career progression of women by affecting junior women’s confidence and through the reallocation of instructor resources away from research and toward teaching.
U.S. randomized, controlled, blind experiment in which 43 students in an online course filled in student evaluations for four sections. Two sections taught by the primary professor, two taught by a female graduate teaching assistant, and two taught by a male teaching assistant. In one of the two sections taught by each assistant, the assistant used her or his true name; in the other, she or he used the other assistant's identity. Thus, in two sections, the students were led to believe they were being taught by a woman, and in two sections, they were led to believe they were being taught by a man. Students had no direct contact with assistants: the primary interactions were through online discussion boards. The assistants' credentials presented to the students were comparable; the assistants covered the same material; and assignments were returned at the same time in all sections. Student evaluations included an overall score and questions relating to professionalism, respectfulness, care, enthusiasm, communication, helpfulness, feedback, promptness, consistency, fairness, responsiveness, praise, knowledge, and clarity. Students rated the male identity significantly higher than the female identity, regardless of the instructor’s actual gender, demonstrating gender bias. Given the vital role that student ratings play in academic career trajectories, this finding warrants considerable attention. For details on the experiment see also Boring et al. (2017).
Natural experiment at a French university with 20,197 student evaluations in mandatory first-year courses in social sciences. The study uses data from a French university to analyze gender biases in student evaluations of teaching. The results of fixed effects and generalized ordered logit regression analyses show that male students express a bias in favor of male professors. Also, the different teaching dimensions that students value in male and female professors tend to match gender stereotypes. Men are perceived by both male and female students as being more knowledgeable and having stronger class leadership skills, although students appear to learn as much from women as from men. For details on the experiment see also Boring et al. (2017).
The study conducted at a graduate school of social sciences in The Netherlands applying a regression study with fixed and time effects on student evaluations of teaching (n=688) at Erasmus University Rotterdam. The dataset includes student evaluations of mixed teaching teams. Most lecturers teach more than one course and many courses are co-taught by mixed gender and ethnicity teams. This allows the authors to study the impact of gender and ethnicity on student evaluations within the same course. This strategy controls for course heterogeneity and for self-selection of teachers and students into courses, all of which are determinants of evaluations. The authors document a negative effect of being a female teacher on student evaluations of teaching, which amounts to roughly one fourth of the sample standard deviation of teaching scores. Overall women are 11 percentage points less likely to attain the teaching evaluation cut-off for promotion to associate professor compared to men. The effect is robust to a host of co-variates such as course leadership, teacher experience and research quality. The results are also robust to netting out teacher unobservables (such as ability or personality) in an alternative panel model specification. Running a teacher fixed effect models separately for men and women allows estimating the impact of co-teaching with the opposite, as well as same gender relative to teaching a course alone. Women obtain considerably lower teacher evaluations when teaching with men compared to teaching alone or with other women. There is no evidence of a corresponding ethnicity effect. The results are suggestive of a gender bias against female teachers and indicate that the use of teaching evaluations in hiring and promotion decisions may put female lectures at a disadvantage.
Small quasi-experimental study (n=42) in 14 online political science sections at a large U.S. state institution exploring gender, race and ethnicity biases in student evaluations. A different faculty member presided over each section as the instructor of record to supervise grading and broad course-level issues. Although the instructors were considerably visible—listed in the registration interface, on the online course homepage, as the suggested first point of contact for questions or concerns, and featured in a welcome video posted to the course homepage and emailed to all relevant students—their interaction with students ended there. The courses contained identical textbook readings, lecture content, activities, assignments, and student assessments. All welcome videos recorded by the instructors contained an identical script that delivered a brief and basic welcome to the course. All 14 courses had the same course coordinator charged with overseeing day-to-day administrative responsibilities and to ensure consistency across the sections. The course coordinator handled all questions and communications from students. Although all students had a different professor, their interactions remained with the course coordinator, who responded to students across courses with high consistency. In essence, every component of the online courses was identical except for the identity of the professor. This constituted an ideal quasi-experimental situation in which to compare student evaluations to determine the impact of gender and race/ethnicity on evaluations. Findings show that instructors who are female received significantly lower scores on ordinal student evaluations than those who are white males. Results for persons of color were also lower, however, the results were not significant.
The paper combines observational and experimental data to investigate whether scarcity of female instructors affects students' demand thereof. First, the authors exploit variation in the share of female professors across different faculties at a Swiss university to explore gender patterns in student evaluations (n=26,996). They find that female students evaluate female professors more favorably (compared to male students and relative to the gender differences in evaluating male professors) but only in faculties with a relatively low share of female professors (Economics and Computational Science, but not Communication Science). To shed light on scarcity of female professors as a potential channel for the gender gaps in student preferences, the authors design an incentivized instructor-choice experiment on MTurk (n=1,000). They experimentally vary the existing pool of instructor gender and let subjects choose one additional instructor among one male and one female. Female (and only female) subjects are more likely to choose the female instructor when the pool of instructors is male-dominated, suggesting that female students appreciate a more balanced instructor pool if female professors are scarce (as is the case in Economics and Informatics).
New, not yet published work, abstract just copied from the AEA website
Batz-Barbarich, C., & Felkey, A. 2021. Can Women Teach Math (and be promoted)? A Meta-Analysis of Gender Difference Across Student Teaching Evaluations.
"While bias affects academic promotions in several ways, this meta-analysis examines bias within one of the most common performance evaluation instruments: student teaching evaluations (STEs). When analyzing potential bias in STEs, it is important to account for the intersectionality of gender and the context of discipline as gender expectations, experiences and student preconceptions vary by field. The lack of gender balance in Economics and Finance fields for both students and faculty, coupled with stereotypes that women cannot do math, leads students to question female faculty expertise. This results in women’s credibility and authority in the classroom being challenged. Collectively, social expectations lead women in quantitative fields to have different career experiences and outcomes than men and women in other fields – including differences in their STEs. Despite this theoretical foundation and decades of research citing biases of STEs, no study summarizes the magnitude of these biases or explores their nuances. Our systematic review and meta-analysis fills this void. By exploring this question meta-analytically, we engage in a more objective evaluation of existing evidence by combining data from all research on the subject subsequently improving the accuracy of the conclusions as compared traditional independent studies. Our meta-analysis examines gender differences in STEs across the social sciences including Economics and Finance. Based on a large number of effect sizes, the analysis quantifies the field specific difference in teaching by gender. Moderator analyses also examined how the magnitude is impacted by race, type of evaluation, the aspect of the teaching evaluated, among others. Collectively, this analysis provides a summary of the gender bias within STEs,. We discuss the implications of this on women’s success in academia. Recommendations regarding the use of STEs in promotion decisions are made."
Literature survey: The paper analyses the current research regarding student evaluations of courses and teaching. The article argues that student evaluations are influenced by racist, sexist and homophobic prejudices, and are biased against discipline and subject area. The paper’s findings are relevant to policymakers and academics as student evaluations are undertaken in over 16,000 higher education institutions at the end of each teaching period. The article’s purpose is to demonstrate to the higher education sector that the data informing student surveys is flawed and prejudiced against those being assessed. Evaluations have been shown to be heavily influenced by student demographics, the teaching academic’s culture and identity, and other aspects not associated with course quality or teaching effectiveness. Evaluations also include increasingly abusive comments which are mostly directed towards women and those from marginalised groups, and subsequently make student surveys a growing cause of stress and anxiety for these academics. Yet, student evaluations are used as a measure of performance and play a role in hiring, firing and promotional decisions. Student evaluations are openly prejudiced against the sector’s most underrepresented academics and they contribute to further marginalising the same groups universities declare to protect, value and are aiming to increase in their workforces.
Literature survey: Student evaluations of teaching are ubiquitous in the academe as a metric for assessing teaching and frequently used in critical personnel decisions. Yet, there is ample evidence documenting both measurement and equity bias in these assessments. Student Evaluations of Teaching (SETs) have low or no correlation with learning. Furthermore, scholars using different data and different methodologies routinely find that women faculty, faculty of color, and other marginalized groups are subject to a disadvantage in SETs. Extant research on bias on teaching evaluations tend to review only the aspect of the literature most pertinent to that study. In this paper, the autors review a novel dataset of over 100 articles on bias in student evaluations of teaching and provide a nuanced review of this broad but established literature. They find that women and other marginalized groups do face significant biases in standard evaluations of teaching – however, the effect of gender is conditional upon other factors. They conclude with recommendations for the judicious use of SETs and avenues for future research.
A much more encompassing summary of research articles on teaching evaluations can be found on Rebecca Kreitzer's homepage.
Service work
The recent studies on service work in academia are predominantly U.S. based and show that women do face more and different requests to do service work from colleagues and students. Also the service they do seem to be more internal rather than external, and if asked, women are more often requested to volunteer for token services. It also seems as if women do accept more of these service tasks than male professors do.
This paper presents multiple laboratory experiments at a U.S. university (about 6,000 observations in five experiments) testing for gender differences in the allocation of a service task. Such a service task is a task that everyone prefers to be completed by someone else — like writing a report or serving on a committee. The study examines first whether men and women differ in their “supply” of such tasks, and second, whether the “demand” for such tasks differs by gender. The study finds that women, more than men, volunteer, are asked to volunteer, and accept requests to volunteer for such tasks.
The laboratory experiments mirror the incentives that a small group with three participants faces when it is asked to find a volunteer for a task that everyone is reluctant to undertake. An individual is relatively better off if the task is done by someone else because it allows that individual to spend more time on more-promotable tasks. After the request for a volunteer is made to the group, every member waits for a volunteer to step forward, fully aware that an excessive delay increases the likelihood that an inferior outcome will result (such as the task not being completed in time or not completed at all). As no explicit request is made of any one individual, the request is implicit and arises through time pressure. In the study women on average invest 3.4 times, whereas men invest 2.3 times. This 48 percent difference in total investment is statistically significant.
To manipulate beliefs, the authors also conducted a single-sex version of the experiment. That is, they conducted sessions where only men or only women participated. In sharp contrast to the results from the previous experiment, women are not more likely to invest than men. The average number of investments does not differ by gender. There is no evidence that all-female groups fare better than all-male groups.
To study the demand side of task allocation, a fourth group member was added, a requestor. Before the investment round the requestor is asking one of the three members of the investment group to invest. On average females receive 2.5 more requests than males.
Beliefs that women, more than men, say yes to tasks with low promotability appear as an important driver of these differences. If women hold tasks that are less promotable than those held by men, then women will progress more slowly in organizations.
Extending the paper of Babcock et al. (2017) above, this paper explores whether the prospect of being penalized for saying “no” to a request to do a low-promotability task exacerbates gender differences in the allocation. One might expect that penalties for failure to help another person would be harsher on women than men as helping behavior is more normative for women than for men. In particular, the authors ask whether gender differences found in previous research—women receiving more requests than men to do these tasks and women being more likely to accept such requests—amplify by the prospect of penalties for declining the request. The paper replicates prior findings but finds no evidence that penalties increase the gender differences in task allocation. Nevertheless, the authors find that the penalties men impose on others for saying "no" are larger than those imposed by women.
U.S. study on service work in academia exploiting two data sets to investigate the relative amounts of service performed by women and men: The first is specific to a particular university system and contains details regarding the type of service performed by faculty (n=1,401). The data set comprises annual reports that are required and are used in yearly faculty evaluations. Salary increases are predominantly based on a merit evaluation based on the activity reports. The other is a national data set comprised of respondents to a survey of faculty at a large number of institutions of higher education (about 19,000 responses).
The authors find evidence in both data sources that, on average, women faculty perform significantly more service than men, controlling for rank, race/ethnicity, and field or department. Results suggest that the male–female differential is driven more by internal service—i.e., service to the university, campus, or department—than external service—i.e., service to the local, national, and international communities—although significant heterogeneity exists across field and discipline in the way gender differentials play out.
This U.S. study uses a survey (n=88) and an experimental design (n=121) to shed light on the question of whether female professors are subject to different student expectations and treatment. Do students perceive and expect female professors to be more nurturing than male professors? In a survey of professors across the United States, the find that female (versus male) professors reported getting more requests for standard work demands, special favors, and friendship behaviors. Especially the last two resulting in more emotional labor for female professors. The second study utilized an experimental design. Student participants were shown a scenario in which a fictitious female or male professor got special favor requests by other students. The results indicated that academically entitled students (i.e., those who feel deserving of success in college regardless of effort/performance) had stronger expectations that a female (versus male) professor would grant special favor requests. Those expectations consequently increased students’ likelihood of making the requests and of exhibiting negative emotional and behavioral reactions to having those requests denied.
Using a survey experiment with undergraduate students (n=204) of different fields of a US university, the authors examine whether the way in which information is conveyed and who conveys it shapes student rule following. While they find students largely follow rules regardless of whether they are written or unwritten, they find significant gender bias. Male students are less likely to follow instructions given by a female professor than a male professor. Gender bias among student workers is another bias in academia that may influence productivity, but perhaps greater representation could reverse this trend.
U.S. survey study of the 2009 American Political Science Association (n=1,399) analyzing the dual problems of "women don't ask" and "women don't say no" in the academic profession. Testing if "woman don't ask" the authors test if female faculty bargain more or less frequently than male faculty over such resources as salary, research support, clerical support, moving expenses, and spousal accommodation. Results show that women are more likely to ask for resources than men when considering most categories of bargaining issues. This goes against conventional wisdom in the literature on gender and bargaining. Testing if "woman don't say no" when asked to provide service at the department, college, university, or disciplinary levels, the study finds that women are asked to provide more service and that they agree to serve more frequently than their male colleagues. Furthermore, the authors find that the service women provide is more typically “token” service, as women are less likely to be asked by their colleagues to serve as department chair, to chair committees, or to lead academic programs.
Interview study at a US university with with 27 pretenured and tenured faculty. The demand for faculty service has increased substantially in recent years, the workload is not shared equitably among tenure-track faculty (Guarino & Borden, 2017; Pyke, 2011). Women faculty tend to spend more time on service activities than men, and they tend to perform important yet less institutionally recognized forms of service like mentoring, committee work, emotional labor, and organizational climate control (Babcock, Recalde, Vesterlund, & Weingart, 2017; Misra, Lundquist, Holmes, & Agiomavritis, 2011). Drawing from the theory of gendered organizations (Acker, 1990), this interview study examined how institutional gender biases impact the visibility and evaluation of faculty service across the tenure-track career trajectory. The study findings reveal how task-oriented forms of service tend to be more visible and valued than relationally oriented service. In addition to addressing a gap in the literature, it presents practical recommendations to make service more visible, valuable, and equitable across faculty ranks and gender identities.
US cross sectional survey study (n=957) onperception on faculty workload. Faculty workload inequities have important consequences for faculty diversity and inclusion. On average, women faculty spend more time engaging in service, teaching, and mentoring, while men, on average, spend more time on research, with women of color facing particularly high workload burdens. The authors explore how faculty members perceive workload in their departments, identifying mechanisms that can help shape their perceptions of greater equity and fairness. White women perceive that their departments have less equitable workloads and are less committed to workload equity than white men. Women of color perceive that their departments are less likely to credit their important work through departmental rewards systems than white men. Workload transparency and clarity, and consistent approaches to assigning classes, advising, and service, can reduce women’s perceptions of inequitable and unfair workloads. The research suggests that departments can identify and put in place a number of key practices around workload that will improve gendered and racialized perceptions of workload.
Recommendation letters
The recent literature on recommendation letters in academia is very limited. Without being able to look for causal reasons, the study below adds to older literature that women receive less excellent letter than men.
Analysis of an international data set of 1,224 recommendation letters, submitted by recommenders from 54 countries, for postdoctoral fellowships in the geosciences over the period 2007–2012. The authors examine the relationship between applicant gender and two outcomes of interest: letter length and letter tone. Female applicants are only half as likely to receive excellent letters versus good letters compared to male applicants. Furthermore, no evidence that male and female recommenders differ in their likelihood to write stronger letters for male applicants over female applicants.
Academia is based on merit or on looks?
US study (n=752) exploring the role of appearance in the academic labor market in the United Statesbetween 2002 and 2006. The study document appearance effects in the economics profession. Using unique data on PhD graduates from ten of the top economics departments in the United States the authors test whether morea ttractive individuals are more likely to succeed. They find robust evidence that appearance has predictive power for job outcomes and research productivity. Attractive individuals are more likely to study at higher ranked PhD institutions and are more likely to be placed at higher-ranking academic institutions not only for their first job, but also for jobs as many as 15 years after their graduation, even when controlling for the ranking of PhD institution and first job. Appearance also predicts the success of research output: while it does not predict the number of papers anindividual writes, it predicts the number of citations for a given number of papers, again even when controlling for the ranking of the PhD institution and first job. All these effects are robust, statistically significant, and substantial in magnitude.