Building High Performing Teams

and

Mitigating Unconscious Bias in Recruitment and Hiring 

The following are supplemental materials for our #ARRS23 panel on 4/18/23. (as well as out #RSNA19 workshop on Thursday 12/5/19 , and  my talk at the APAMSA national meeting 10/5/19 )

Video for reducing bias for virtual interviews

https://vimeo.com/443088643


Why should I use structured / behavioral interviews?

Can avoid employment lawsuits.

Can get better recruits/avoid having to coach (or even fire) the folks you hired.

Its recommended by AAMC (although I disagree with their recommendation for hypothetical questions (except for technical issues , which is unsporting for resident interviews!)


https://www.aamc.org/services/admission-interview-foundations/residency

https://www.aamc.org/services/admission-interview-foundations/developing-structured-residency-interviews

https://www.aamc.org/services/admission-interview-foundations/developing-structured-residency-interviews#guide

It was promoted at the 2002 AUR

https://www.apdr.org/assets/docs/Resources/Semi-Structured_Conversation.pdf

Its widely accepted in the business world Google, etc. 

https://www.cnbc.com/2017/11/30/google-uses-this-scientifically-proven-method-to-hire-employees.html 

Several articles from Iowa by Franken and associates discussed the utility of a structured accomplishment (behavioral) interview. A recent review from JGME cites this 1992 reference (Radiology seems to be the first specialty to have used this technique!)

https://journals.lww.com/investigativeradiology/Abstract/1992/05000/The_Predictive_Utility_of_Behavior_Based.13.aspx 

Altmaier EM, Smith WL, O'Halloran CM, Franken EA Jr. The predictive utility of behavior-based interviewing compared with traditional interviewing in the selection of radiology residents. Invest Radiol. 1992;27(5):385–389. 


Theory:

Behavioral interviews are based on the assumption that past behavior predicts future behavior.


Method:

First one has to decide what are desirable qualities/behaviors for the company in general and the job in particular.

Then design questions to assess these qualities. 


Historical notes:

If you have read "Thinking Fast and Slow and/or the "Undoing project", you will note that  in 1955, Daniel Kahneman was tasked with designing a selection process for the Israeli Army.

He assessed 6 behaviors, including sociability, responsibility, and pride. He had interviewers grade applicants on a 5 point scale

1 never exhibits behavior to 5 always exhibits behavior (kind of like the ACGME milestones)

Thinking Fast and Slow p 229-233

https://socraticowl.com/post/hire-like-the-israeli-military/ 


Reference:

podcast with Lisa Stern Haynes explaining Google's approach to interviews

https://play.google.com/music/listen#/ps/Iuqrrs4psdoms4eyvof7nsekwjm 

(June 19, 2017)

A review from JGME

https://www.jgme.org/doi/pdf/10.4300/JGME-D-14-00236.1 


Lets review the case for diversity:

While there are many reasons to encourage diversity in the health care workforce, one of the most compelling is:

In 2002 the Institute of Medicine published "Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care". In 2004 the IOM published "In the Nation’s Compelling Interest: Ensuring Diversity in the Health-CareWorkforce". This report noted that one way to reduce racial and ethnic disparities in health care is to increase the diversity of the Health-Care workforce.

The Association of American Medical Colleges (AAMC) is doing its part to improve the diversity of medical students and has a section of its website devoted to the topic https://www.aamc.org/initiatives/diversity/

The ACGME compiles statistics on diversity amongst the various residencies. Across all residencies between 2015-2016, only 9.6% of residents were underrepresented minorities (versus about 20% in the US population) and 44% were women. Within the field of Radiology, only 5.8 % of residents were underrepresented minorities and 27% were women. (These statistics can be found in the ACGME Data Resource Books)

What can PDs do to improve diversity in Radiology?

What can be done during the interview process?

https://www.acr.org/-/media/ACR/Files/Member-Resources/Women-Diversity/Improving-Diversity-using-ERAS.pdf

https://www.acr.org/-/media/ACR/Files/Member-Resources/Women-Diversity/Improving-Diversity-using-ERAS.pdf

https://www.aamc.org/what-we-do/mission-areas/diversity-inclusion/unconscious-bias-training 

(register for the free seminar)

and/or assign this JACR article

https://www.jacr.org/article/S1546-1440(17)30209-0/abstract


This is a short HBR article by Howard Ross

https://hbr.org/2015/04/3-ways-to-make-less-biased-decisions

An article on the history of behavioral interviews (created by Daniel Kahneman who would later win the Nobel Prize for creating the field of behavioral economics)

https://www.inc.com/marcel-schwantes/19-interview-questions-that-cut-through-bs-to-reveal-a-great-hire.html

An article from Inc

https://www.inc.com/marcel-schwantes/19-interview-questions-that-cut-through-bs-to-reveal-a-great-hire.html


A twitter post with a nice infographic

https://twitter.com/helenbevan/status/1023241217205624832

 A recent article on mitigating UB. The author interviewed Anthony Greenwald (one of the developers of the IAT) who notes that much of what we do to mitigate bias may not be effective

https://www.knowablemagazine.org/article/mind/2020/how-to-curb-implicit-bias 

An article from HBR by Howard Ross on mitigating bias 

https://hbr.org/2015/04/3-ways-to-make-less-biased-decisions

A 2018 reference by Kasales et al. that lists many references supporting the use of structure/behavioral interviews:

https://www.academicradiology.org/article/S1076-6332(18)30488-4/pdf

A 2017 reference (thanks to my resident Florence Doo for finding this one)

https://www.atsjournals.org/doi/full/10.1513/AnnalsATS.201611-940OC

A 2017 article from the NYT on the Uselessness of the (Traditional) Interview. (The make a pitch for the structured interview)

https://www.nytimes.com/2017/04/08/opinion/sunday/the-utter-uselessness-of-job-interviews.html

A 2016 HBR article on "How to HIre Without Getting Fooled by First Impressions" describes an Ophthalmologist's problem with hiring a practice manager. Their solution involves using behavioral  interview type questions. (First impressions are important for sales positions)

https://hbr.org/2016/02/how-to-hire-without-getting-fooled-by-first-impressions

A 2013 article from the NYT discussing what Google has learned about hiring/interviews

https://www.nytimes.com/2013/06/20/business/in-head-hunting-big-data-may-not-be-such-a-big-deal.html

An article on the  Halo effect

https://www.people2people.com.au/blog/2015/06/the-halo-effect-interviewing-under-the-influence

Tip sheet for residents from U Wash which has sample behavioral questions

https://www.uwmedicine.org/education/Documents/md-program/Interviewing%20for%20Residency.pdf

Aritcles on how to interview for "grit" (essentially behavioral interview questions)

https://getlighthouse.com/blog/grit-skill-interview-for-how-to-look-for-it/

https://www.forbes.com/sites/raquelbaldelomar/2016/07/31/four-interview-questions-to-help-you-hire-people-with-grit/#3489366a50d3

A reference about the use of behavioral interviews for Obstetrics and Gynecology residents

https://www.ncbi.nlm.nih.gov/pubmed/27321984

A recent review from JGME:

http://www.jgme.org/doi/10.4300/JGME-D-14-00236.1

A reference on bias by Mark Manson

https://markmanson.net/cognitive-biases-that-make-us-terrible



Note in the references from Investigative Radiology

Curtis DJ, Riordan DD, Cruess DF, Brower AC. Selecting radiology resident candidates. Invest Radiol. 1989;24(4):324–330. 



A 2017 Article by McArthur, Flug, Restauri

https://www.ncbi.nlm.nih.gov/pubmed/28040296 



What can be done every day?

https://www.insidehighered.com/advice/2016/04/13/how-be-ally-someone-experiencing-microaggressions-essay

Many of the practices suggested to mitigate unconscious bias are out of the mindfulness "playbook", e.g. stopping to reflect on your thoughts, being nonjudgemental. There are studies that support mindfulness mitigating bias. Mindfulness has many other benefits as well. 

References:

These will improve your leadership skills and improve your ability to recognize and mitigate bias.

Thinking Fast and Slow- Daniel Kahneman

Difference Matters- Brenda Allen (Keynote Speaker at #AUR16

Altered Traits- Daniel Goleman and Richard Davidson


 As the 2004 IOM report states:

 “Knowing is not enough; we must apply.

Willing is not enough; we must do.”

—Goethe

Let me know if you have any questions/comments

Nolan Kagetsu, MD, FACR

Vice Chairman, Quality

Department of Radiology

Mount Sinai GME Diversity Committee

APDR Diversity Task Force, Co-Chair

ACR Committee for Diversity and Inclusion

ACGME Diversity Planning Group

Please e-mail me at nolan.kagetsu@mountsinai.org

follow me on twitter @nkagetsu

________________________________________________________________

Additional Resources:

NIH: The NIH has many resources devoted to diversity. They support "the science of diversity": https://diversity.nih.gov/


Mount Sinai: Message on diversity by Dean Charney: http://icahn.mssm.edu/about/diversity/dean


Disparities:

Article posted 715/16 on the Commonwealth Fund website: http://www.commonwealthfund.org/publications/blog/2016/jul/closing-the-equity-gap


Unconscious Bias:


This is a nice short video (recommended by our colleague Brenda Allen) introducing us to fast and slow thinking, this will help you to understand unconscious bias.

https://www.youtube.com/watch?v=JiTz2i4VHFw&sns=em

For those interested in more you can read Daniel Kahneman's "Thinking Fast and Slow"

An article in the Harvard Business Review by Howard Ross, expert on Unconscious Bias (thanks to Pam Abner, Mount Sinai Office of Diversity and Inclusion, who gave me this reference): https://hbr.org/2015/04/3-ways-to-make-less-biased-decisions

This article is a good intro to UB as well as discussing ways to mitigate bias.

This is a brief article by Richard Gunderman and Dr. Kagetsu that recently appeared in JACR. We focus on steps to mitigate bias in the interview process.

http://www.jacr.org/article/S1546-1440(17)30209-0/fulltext


From Fast Company: http://www.fastcompany.com/3037359/strong-female-lead/how-unconscious-bias-affects-everything-you-do

Facebook has made their resources available on line: https://managingbias.fb.com/

NIH has resources related to mitigating bias as well:  https://diversity.nih.gov/sociocultural-factors/unconscious-bias

UCLA resources: https://equity.ucla.edu/programs-resources/educational-materials/implicit-bias-resources/

The AAMC website has resources on unconscious bias:  https://www.aamc.org/initiatives/diversity/322996/lablearningonunconsciousbias.html

Interesting reference from some Boston colleagues: http://www.brighamandwomens.org/Medical_Professionals/career/CFDD/Images/UnconciousBiasTipSheet.pdf

Is Race Genetic or a "Socially Constructed:" http://bigthink.com/videos/philip-kitcher-is-race-genetic-or-socially-constructed


Holistic Review:

From the AAMC website:  https://www.aamc.org/initiatives/holisticreview/

From the New England Journal of Medicine:  ttp://www.nejm.org/doi/full/10.1056/NEJMp1300411


Mindfulness:

An organization that uses mindfulness techniques to reduce unconscious bias is Be More (They are based in New York City and made a presentation at a recent ACGME meeting): http://www.bemoreamerica.org/

Another article from the Harvard Business Review that supports the use of mindfulness techniques:  https://hbr.org/2014/12/mindfulness-mitigates-biases-you-may-not-know-you-have

Interesting article on Mindfulness-Based Color Insight Practice:  http://greatergood.berkeley.edu/article/item/how_mindfulness_can_defeat_racial_bias

Interesting article on the intersection of minfulness and unconscious bias training: https://hbr.org/2017/01/how-mindfulness-helped-a-workplace-diversity-exercise

A presentation addressing the issue of disparity in residencies created by UPMC: https://www.acgme.org/acgmeweb/Portals/0/PDFs/2015%20AEC/Presentations/SES086.pdf

"Why diversity matters in Radiology" a 2014 article from "Health Imaging": http://www.healthimaging.com/topics/practice-management/why-diversity-matters-radiology


AUR 2016

We also had a poster presentation at AUR 2016:

Diversity and Imaging 3.0: Is There an Alliance

Diversity.pdf

Author Bio

Nolan Kagetsu is a neuroradiologist at Mount Sinai West. He was a diagnostic radiology program director for 15 years and has used behavioral interviews during that time. He is also a Cook-Ross Unconscious bias trainer for the Mount Sinai Office of Diversity and Inclusion (ODI) He has written about mitigating unconscious bias.  (http://www.jacr.org/article/S1546-1440(17)30209-0/fulltext) He recently completed a Mindfulness Based Stress Reduction course at Mount Sinai. 

Dr. Kagetsu serves  the American College of Radiology Committee for Diversity and Inclusion and co-chairs the American Program Directors  in Radiology (APDR) Diversity Committee, as well as the New York State Radiological Society (NYSRS) DIversity and Inclusion  Committee.  He served on the  ACGME Diversity Planning Group, who then created the position of Chief Diversity and Inclusion Officer

https://www.acgme.org/Newsroom/Newsroom-Details/ArticleID/8038/ACGME-Names-First-Chief-Diversity-and-Inclusion-Officer 


I studied chemical engineering at MIT. I went to Albany Medical College (class of 1984) which makes me PGY 36. I went straight to radiology residency at what was then St.Luke's-Roosevelt (now Mount Sinai West and then completed a 2 year neuroradiology/interventional neuroradiology fellowship at NYU in 1990. I was a junior attending on Alex Berenstein's neuro IR team. In 1991, I joined the faculty at St.Luke's-Roosevelt. In 2015 we became part of the Mount Sinai system.  I currently serve on the  Mount Sinai GME committee, the GME diversity committee, the GME wellness committee,  as well as the GME CLER (quality and safety) committee. I co-chair the Hospital Based Specialty Subcommittee .

I wrote neuroradiology questions for the American Board of Radiology, Ongoing Longitudinal Assessment (OLA) team.

Action Items:

https://sites.google.com/site/neuroradiologyprimer/mitigating-unconscious-bias

PDF versions of our presentations


Kagetsu.Behavioral.Interviews.pdf
DeBenedectis.Resident.Recruitment.pdf
Strax.Faculty.Recruitment.pdf