The following are supplemental materials for our #AAR25 session on 3/11/25 on 1:15-2:45 PM PT.
(as well as out #RSNA19 workshop on Thursday 12/5/19 , and my talk at the APAMSA national meeting 10/5/19 )
Why shoud I use structured / behavioral interviews?
Can avoid employment lawsuits.
Can get better recruits/avoid having to coach (or even fire) the folks you hired.
Its recommended by AAMC
https://www.aamc.org/services/admission-interview-foundations/residency
It was promoted at the #AUR02
https://www.apdr.org/assets/docs/Resources/Semi-Structured_Conversation.pdf
Its widely accepted in the business world Google, etc.
https://www.cnbc.com/2017/11/30/google-uses-this-scientifically-proven-method-to-hire-employees.html
Radiologists seem to be the first who have used these methods for GME!
This 1992 paper from Iowa by Altamaier et al. discussed the utility of a structured accomplishment (behavioral) interview.
Altmaier EM, Smith WL, O'Halloran CM, Franken EA Jr. The predictive utility of behavior-based interviewing compared with traditional interviewing in the selection of radiology residents. Invest Radiol. 1992;27(5):385–389.
A recent review from JGME cites this reference as well:
https://www.jgme.org/doi/pdf/10.4300/JGME-D-14-00236.1
Theory:
Behavioral interviews are based on the assumption that past behavior predicts future behavior.
Method:
First one has to decide what are desirable qualities/behaviors for the company in general and the job in particular.
Then design questions to assess these qualities.
Historical notes:
If you have read "Thinking Fast and Slow and/or the "Undoing project", you will note that in 1955, Daniel Kahneman was tasked with designing a selection process for the Israeli Army.
He assessed 6 behaviors, including sociability, responsibility, and pride. He had interviewers grade applicants on a 5 point scale
1 never exhibits behavior to 5 always exhibits behavior (kind of like the ACGME milestones)
Thinking Fast and Slow p 229-233
https://socraticowl.com/post/hire-like-the-israeli-military/
Reference:
podcast with Lisa Stern Haynes explaining Google's approach to interviews
https://play.google.com/music/listen#/ps/Iuqrrs4psdoms4eyvof7nsekwjm
(June 19, 2017)
Lets review the case for diversity:
While there are many reasons to encourage diversity in the health care workforce, one of the most compelling is:
In 2002 the Institute of Medicine published "Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care". In 2004 the IOM published "In the Nation’s Compelling Interest: Ensuring Diversity in the Health-CareWorkforce". This report noted that one way to reduce racial and ethnic disparities in health care is to increase the diversity of the Health-Care workforce.
The Association of American Medical Colleges (AAMC) is doing its part to improve the diversity of medical students and has a section of its website devoted to the topic https://www.aamc.org/initiatives/diversity/.
The ACGME compiles statistics on diversity amongst the various residencies. Across all residencies between 2015-2016, only 9.6% of residents were underrepresented minorities (versus about 20% in the US population) and 44% were women. Within the field of Radiology, only 5.8 % of residents were underrepresented minorities and 27% were women. (These statistics can be found in the ACGME Data Resource Books)
What can PDs do to improve diversity in Radiology?
Make sure you complete race/ethnicity fields in ADS (nationwide 24% of data is missing !)
Consider discussing race/ethnicity with trainees.
Radiology should be promoted during third year medical clerkships as a field that embraces diversity, and be presented as a career option to all students, including underrepresented minorities and women.
Program directors should be mindful of how residents are chosen for leadership positions. e.g. if a review of the last 5 years shows that there have been no women chief residents, one could ask why??
Participate in pipeline programs of your medical school (e.g. Mount Sinai has VEPSUM https://erap.mssm.edu/Public/VEPSUM.aspx
Get involved in diversity activities at your hospital, e.g. Diversity Councils, Employee Resource Groups.
What can be done during the interview process?
Program directors should learn about unconscious bias and holistic review and then share this with their interview team.
PDs should use ERAS to become aware of all the URM applicants. See the slides I prepared for the ACR website.
PDs should know how many women and URM applicants have applied to their programs using ERAS. See slides I made which are posted on the ACR website:
PDs should track the percentage of underrepresented minority applicants and women that have applied, are invited to interview, interviewed, ranked, matched, and retained. (accountability as referred to in the Howard Ross article)
The interview team should have some UB training, e,g, AAMC website has this resource:
https://www.aamc.org/what-we-do/mission-areas/diversity-inclusion/unconscious-bias-training
(register for the free seminar)
and/or assign this JACR article
https://www.jacr.org/article/S1546-1440(17)30209-0/abstract
This is a short HBR article by Howard Ross
https://hbr.org/2015/04/3-ways-to-make-less-biased-decisions
The interview team should be diverse
Agree on ideal qualifications ahead of the interview. This avoids "reconstructing the merit" i.e., rationalizing why the candidates specific qualities make them suited for the position. These qualifications should be the basis of behavioral interview questions. (e.g. conflict resolution, teamwork)
Avoid traditional conversational interviews. (yes, one should put the applicant at ease, but not waste too much time talking about hobbies, etc.)
Be careful about assessing "fit". Ask what folks mean by that statement.
Consider an interview team with one interviewer that has not reviewed the folder (avoid confirmation bias) and one that has.
Use behavioral interview questions. (One can think of this as having a more relevant conversation.) Check to see if your hospital has internal training for behavioral interviews. (Mount Sinai does)
Many interviewers overestimate their skill in assessing applicants. When someone says I can usually tell in the first few minutes if they want a candidate, that means they are biased.
References:
An article on the history of behavioral interviews (created by Daniel Kahneman who would later win the Nobel Prize for creating the field of behavioral economics)
https://www.inc.com/marcel-schwantes/19-interview-questions-that-cut-through-bs-to-reveal-a-great-hire.html
An article from Inc
A recent article on mitigating UB. The author interviewed Anthony Greenwald (one of the developers of the IAT) who notes that much of what we do to mitigate bias may not be effective
https://www.knowablemagazine.org/article/mind/2020/how-to-curb-implicit-bias
An article from HBR by Howard Ross on mitigating bias
https://hbr.org/2015/04/3-ways-to-make-less-biased-decisions
A 2018 reference by Kasales et al. that lists many references supporting the use of structure/behavioral interviews:
https://www.academicradiology.org/article/S1076-6332(18)30488-4/pdf
A 2017 reference (thanks to MSW resident Florence Doo for finding this one)
https://www.atsjournals.org/doi/full/10.1513/AnnalsATS.201611-940OC
A 2017 article from the NYT on the Uselessness of the (Traditional) Interview. (The make a pitch for the structured interview)
https://www.nytimes.com/2017/04/08/opinion/sunday/the-utter-uselessness-of-job-interviews.html
A 2016 HBR article on "How to HIre Without Getting Fooled by First Impressions" describes an Ophthalmologist's problem with hiring a practice manager. Their solution involves using behavioral interview type questions. (First impressions are important for sales positions)
https://hbr.org/2016/02/how-to-hire-without-getting-fooled-by-first-impressions
A 2013 article from the NYT discussing what Google has learned about hiring/interviews
https://www.nytimes.com/2013/06/20/business/in-head-hunting-big-data-may-not-be-such-a-big-deal.html
An article on the Halo effect
https://www.people2people.com.au/blog/2015/06/the-halo-effect-interviewing-under-the-influence
Tip sheet for residents from U Wash which has sample behavioral questions
https://www.uwmedicine.org/education/Documents/md-program/Interviewing%20for%20Residency.pdf
Aritcles on how to interview for "grit" (essentially behavioral interview questions)
https://getlighthouse.com/blog/grit-skill-interview-for-how-to-look-for-it/
A reference about the use of behavioral interviews for Obstetrics and Gynecology residents
https://www.ncbi.nlm.nih.gov/pubmed/27321984
A recent review from JGME:
http://www.jgme.org/doi/10.4300/JGME-D-14-00236.1
A reference on bias by Mark Manson
https://markmanson.net/cognitive-biases-that-make-us-terrible
Note in the references from Investigative Radiology
Curtis DJ, Riordan DD, Cruess DF, Brower AC. Selecting radiology resident candidates. Invest Radiol. 1989;24(4):324–330.
A 2017 Article by McArthur, Flug, Restauri
https://www.ncbi.nlm.nih.gov/pubmed/28040296
What can be done every day?
Participate in social media to promote diversity.
Learn about microagressions and "microresistance"/being an ally
Practice mindfulness
Many of the practices suggested to mitigate unconscious bias are out of the mindfulness "playbook", e.g. stopping to reflect on your thoughts, being nonjudgemental. There are studies that support mindfulness mitigating bias. Mindfulness has many other benefits as well.
A NYC based group https://www.bemoreamerica.org/ has UB training based on mindfulness.
The NY Fire Department is using mindfulness to improve performance in stressful situations as well as reduce unconscious bias
References:
Learn about emotional intelligence, unconscious bias, mindfulness
These will improve your leadership skills and improve your ability to recognize and mitigate bias.
For those with more time here some some books I recommend:
Thinking Fast and Slow- Daniel Kahneman
Difference Matters- Brenda Allen (Keynote Speaker at #AUR16
Altered Traits- Daniel Goleman and Richard Davidson
As the 2004 IOM report states:
“Knowing is not enough; we must apply.
Willing is not enough; we must do.”
—Goethe
Let me know if you have any questions/comments
Nolan Kagetsu, MD, FACR
Department of Radiology
Mount Sinai GME Diversity Committee
ACR Committee for Diversity and Inclusion
Please e-mail me at nolan.kagetsu@mountsinai.org
follow me on twitter @nkagetsu
________________________________________________________________
Additional Resources:
NIH: The NIH has many resources devoted to diversity. They support "the science of diversity": https://diversity.nih.gov/
Mount Sinai: Message on diversity by Dean Charney: http://icahn.mssm.edu/about/diversity/dean
Disparities:
Article posted 715/16 on the Commonwealth Fund website: http://www.commonwealthfund.org/publications/blog/2016/jul/closing-the-equity-gap
Unconscious Bias:
This is a nice short video (recommended by our colleague Brenda Allen) introducing us to fast and slow thinking, this will help you to understand unconscious bias.
https://www.youtube.com/watch?v=JiTz2i4VHFw&sns=em
For those interested in more you can read Daniel Kahneman's "Thinking Fast and Slow"
An article in the Harvard Business Review by Howard Ross, expert on Unconscious Bias (thanks to Pam Abner, Mount Sinai Office of Diversity and Inclusion, who gave me this reference): https://hbr.org/2015/04/3-ways-to-make-less-biased-decisions
This article is a good intro to UB as well as discussing ways to mitigate bias.
This is a brief article by Richard Gunderman and Dr. Kagetsu that recently appeared in JACR. We focus on steps to mitigate bias in the interview process.
http://www.jacr.org/article/S1546-1440(17)30209-0/fulltext
From Fast Company: http://www.fastcompany.com/3037359/strong-female-lead/how-unconscious-bias-affects-everything-you-do
Facebook has made their resources available on line: https://managingbias.fb.com/
NIH has resources related to mitigating bias as well: https://diversity.nih.gov/sociocultural-factors/unconscious-bias
UCLA resources: https://equity.ucla.edu/programs-resources/educational-materials/implicit-bias-resources/
The AAMC website has resources on unconscious bias: https://www.aamc.org/initiatives/diversity/322996/lablearningonunconsciousbias.html
Interesting reference from some Boston colleagues: http://www.brighamandwomens.org/Medical_Professionals/career/CFDD/Images/UnconciousBiasTipSheet.pdf
Is Race Genetic or a "Socially Constructed:" http://bigthink.com/videos/philip-kitcher-is-race-genetic-or-socially-constructed
Holistic Review:
From the AAMC website: https://www.aamc.org/initiatives/holisticreview/
From the New England Journal of Medicine: ttp://www.nejm.org/doi/full/10.1056/NEJMp1300411
Mindfulness:
An organization that uses mindfulness techniques to reduce unconscious bias is Be More (They are based in New York City and made a presentation at a recent ACGME meeting): http://www.bemoreamerica.org/
Another article from the Harvard Business Review that supports the use of mindfulness techniques: https://hbr.org/2014/12/mindfulness-mitigates-biases-you-may-not-know-you-have
Interesting article on Mindfulness-Based Color Insight Practice: http://greatergood.berkeley.edu/article/item/how_mindfulness_can_defeat_racial_bias
Interesting article on the intersection of minfulness and unconscious bias training: https://hbr.org/2017/01/how-mindfulness-helped-a-workplace-diversity-exercise
A presentation addressing the issue of disparity in residencies created by UPMC: https://www.acgme.org/acgmeweb/Portals/0/PDFs/2015%20AEC/Presentations/SES086.pdf
"Why diversity matters in Radiology" a 2014 article from "Health Imaging": http://www.healthimaging.com/topics/practice-management/why-diversity-matters-radiology
AUR 2016
We also had a poster presentation at AUR 2016:
Diversity and Imaging 3.0: Is There an Alliance
Author Bio
Nolan Kagetsu is a neuroradiologist at Mount Sinai West. He was a diagnostic radiology program director for 15 years and has used behavioral interviews during that time. He is also a Cook-Ross Unconscious bias trainer for the Mount Sinai Office of Diversity and Inclusion (ODI) He has written about mitigating unconscious bias. (http://www.jacr.org/article/S1546-1440(17)30209-0/fulltext) He completed a Mindfulness Based Stress Reduction course at Mount Sinai.
Dr. Kagetsu served on the American College of Radiology Committee for Diversity and Inclusion and co-chaired the American Program Directors in Radiology (APDR) Diversity Committee. He cochairs the New York State Radiological Society (NYSRS) Outreach and Professional Opportunity Committee. He served on the ACGME Diversity Planning Group, which then created the position of Chief Diversity and Inclusion Officer
I studied chemical engineering at MIT. I went to Albany Medical College (class of 1984) which makes me PGY 36. I went straight to radiology residency at what was then St.Luke's-Roosevelt (now Mount Sinai West and then completed a 2 year neuroradiology/interventional neuroradiology fellowship at NYU in 1990. I was a junior attending on Alex Berenstein's neuro IR team. In 1991, I joined the faculty at St.Luke's-Roosevelt. In 2015 we became part of the Mount Sinai system. I served on the Mount Sinai GME committee, the GME diversity committee, the GME wellness committee, as well as the GME CLER (quality and safety) committee. I co-chaired the Hospital Based Specialty Subcommittee .
I wrote neuroradiology questions for the American Board of Radiology, Ongoing Longitudinal Assessment (OLA) team.
Action Items:
consider signing up for Mindfulness Based Stress Reduction (MBSR)
read/circulate this HBR Howard Ross reference:
3 Ways to Make Less Biased Decisions
consider structured/behavioral interviews