why-phone-screens-matter
Why Phone Screens Matter
I recently conducted an interview in which the candidate failed every single one of my interview questions. This candidate could have been weeded out of the recruiting pipeline in the first resume screen. Instead, we flew him out and put him up in a fancy hotel, and he proceeded to garner not-inclined to strong-no-hire votes from every interviewer on the loop.
That's not particularly interesting. What is interesting is that it's becoming a commonplace occurrence: this at least the 30th time I've been on such a loop in the past year. And the problem seems to be getting worse. In Q4 the majority of the loops I was on turned out exactly like this one.
I'll post my interview feedback here and discuss it a bit, since it has no personally identifiable information about the candidate or the interviewers. Take a look and see if you think this candidate would have passed your phone screen.
Loop Details
This was an SDE-2 candidate for a group that works primarily (though not exclusively) in C++. The candidate has 15+ years of experience on Unix and Windows, doing mostly C++ development. The first phone screener was inclined; the second strongly inclined. Everyone on the loop voted against hiring, so it was a perfect failure.
Would you have passed this person through your phone screen? Read my feedback and decide for yourself.
It's worth pointing out that the second phone screener is an absolutely outstanding engineer, possibly even world-class. He made the classic mistake of letting the candidate do most of the talking, and only asked about project work the candidate has done in the past. I don't know the first screener, but he also focused only on the candidate's limited areas of strength.
Perhaps the most common phone-screening mistake made at Amazon, though, is assuming that knowing C++ and basic OOP are all you need in order to pass your SDE interviews here. They're not. You need a great deal more than that, and hacking C++ for 15 years does not automatically make someone a good engineer. In fact, it rarely does.
I welcome comments and feedback on this interview. If you think I've made a tragic mistake, and asked totally inappropriate questions, please let me know by commenting below (anonymously if you like).
My MRT Feedback
(begin feedback)
Candidate: {name withheld}
Interview date: {date withheld}
Interviewer: Steve Yegge
Feedback submitted: Oct 23rd, 4004 B.C., 9:00am
Vote: Strong no hire
I had about a 20-minute interview with this candidate, since we had to get him out the door to catch his plane. But he succeeded in failing every question for the entire 20 minutes.
Summary:
he has no Linux experience
his general Unix experience is outdated (e.g. he couldn't remember the 'man' utility for looking up help on 'find')
he has no experience with scripting or regular expressions
he has no experience with XML
he has no knowledge of higher-level languages (Java, Perl, etc.)
he had never heard of a "graph" (in CS terms), even when I explained it to him
he had only a very weak grasp of big-O analysis
he thought the mathematical inverse of 2N is sqrt(n)
he didn't know how to print out the nodes of a tree: no idea whatsoever.
he didn't know how to find phone numbers in text files: no idea whatsoever.
he has only a weak knowledge of C++'s string-processing facilities
We spent 10 minutes talking through the problem of identifying all the HTML files in a dir tree that appear to have US phone numbers in them. He couldn't figure it out. He said he wasn't very comfortable with unix utilities or scripting or regexps, and that he'd prefer to write a C++ program to solve it. But he couldn't figure out how he'd write the program. I asked specifically how he'd keep track of "where he's at" in looking for parens, digits, hyphens and so on. I asked in different ways, and he still didn't have any ideas. With much prompting, he eventually decided that he'd tokenize each line, and in a loop, do something like this (in pseudo-code):
for i = 0 to tokens.length:
if tokens[i] == '(' and
isDigit(tokens[i+1]) and
isDigit(tokens[i+2])
...
then it's a match
else advance to next token and start over
And even that was a crappy solution that I basically handed to him.
He didn't think of making a state machine or doing anything else even remotely configurable or elegant (or correct). He didn't know the names of any C or C++ functions for tokenizing or manipulating strings, and he couldn't tell me how to traverse a filesystem programmatically.
We spent another few minutes talking about miscellaneous Unix-related topics, and it became clear that he hasn't done much with Unix in a very long time. I asked him, for example, where he'd go to find information about the 'find' utility. I'd have settled for any valid answer — heck, he could have said 'Google' and I'd have accepted it. But he couldn't remember man or info, and didn't think of looking on the web or in a book. He said he thought there was some sort of "help" utility on AIX that might have it. (Maybe there is, but it's a pretty weak answer, from a Unix perspective.)
I asked him how to print out a directed graph (possibly with cycles), and drew one on the board. He didn't understand the question, and I had to explain what a graph was, and give some real-life examples, before he got the concept. I asked how he'd avoid going into loops or printing the same node multiple times, and he couldn't think of a way. I gave him lots of hints:
how would you do it if you could modify the nodes?
how would you do it if you could *mark* the nodes?
could you keep track of nodes you've visited in another data structure?
He just couldn't do it. He had never once in his career encountered this alien concept.
I dialed it down to trees, and asked how he'd print the nodes of a well-formed binary tree, and drew one. We talked about tree properties a bit, and he didn't know any of the math behind them. Even when I drew a tree on the board to depth 5, he took a long time to figure out that the size at depth N is 2N-1, and he said he still wasn't sure. Then I asked what the mathematical inverse of 2N is, and even gave him some example numbers. He guessed it was some nth-root or square-root function. I asked if it was, in fact, log2(N), and he thought that sounded closer to the right answer.
I asked him to print the nodes of an n-ary tree, and that confused him (even when I drew it), so I had him do a binary tree, and he vaguely remembered that you "go down the left path until you can't any more, then do the right side". But he couldn't express the algorithm more clearly than that. I asked if he'd do it in a loop, and he didn't know for sure. I asked if he could do it recursively, and he brightened up — clearly he remembered that recursion was a "magic word" associated with his recollection of the traversal algorithm. He said he hadn't been thinking of recursion when he was trying to solve it. But then he couldn't tell me (much) about how it works, nor could he express the algorithm any more concretely.
I asked him if there are more or less standard, well-known tree-traversal algorithms that he could look up online. He just frowned and hemmed and hawed. He really didn't want to answer, so I didn't make him.
I asked if recursion and iteration are equivalent, phrasing the question in different ways. He wasn't entirely sure, but he seemed to remember hearing the answer was "yes" at one point.
(end feedback)
Summary
All of these questions could have been asked in the phone screen. I never made him to go to the whiteboard, so my interview didn't need to be conducted in person. If he had been asked the five essential phone screen questions, we wouldn't have needed to spend the money and time to fly him out here, put him in a fancy hotel, and have five people interview him.
We need to get better at phone-screening people.
Phone screens are hard, but doing them right can save us a lot of wasted time and money. It's important to remember to cover the basics when you screen someone. You can't assume they know the fundamentals just because they've worked on big, fancy-sounding projects.
(Published Jan 17th, 2005)
Comments
I'm amazed at how many questions he managed to bomb in only 20 minutes. My experience is that after a candidate bombs several questions in a row, I find it very hard to cut a line off to ask something new without sounding like I've made up my mind. Any key phrases to keep things moving along (and avoiding letting the candidate waste both our time digging a hole)?
Posted by: Ami F. at January 18, 2005 04:59 AM
Now I'm curious to see what the phone screen feedback was like.
But you've hit on a strong point here: SDE training needs to include training on how to identify and select quality SDEs.
We have a limited amount of recruiting training and process documentation. But it's all very high level, and none of it is geared towards technical interviewing.
Combined with our disposition for one-on-one interviews, and our systemic lack of pre-interview strategizing, and you end up with engineers being plopped down in rooms, or on phones, with no clue about how to run an interview, good kinds of questions to ask, or even what they're looking for.
We carp on people for not doing adequate phone screening, but I'd maintain that phone screens are the hardest part of the interview process, and that it's hard to run one well unless you've got extensive in-person interviewing experience. Phone screens flow differently, and you have to work harder to corral the candidate.
What we really need is a strong progression of interviewing skills training:
Lecture/training materials to introduce the interview process, guide engineers on technical skills assessment, and set in their minds what our bars are for the various SDE positions.
Pair in-person interviewing, with an experienced interviewer (bar raiser?).
Solo in-person interviewing (although my personal opinion is that we should abolish this entirely).
Pair/team phone screens (more of a leader/monitor type situation, though).
Solo phone screens.
I guess my point is that we shouldn't be having green SDEs be doing phone screens here.
-Jon
Posted by: Jon M at January 19, 2005 07:29 PM
My first reaction reading this is that the person on the phone screen isn't the same as the one who flew in. Any guesses on what the rate of this happening is?
Posted by: Albert W. at January 19, 2005 10:55 PM
Doesn't seem surprising. Don't most people just suck at most things?
I've seen cases here where a hiring manager will bring in a sub-par candidate just because the manager thought the team needed that body that badly.
Sigh.
I'm not sure I want to work at a company that does that kind of thing.
Posted by: Andrew W. at January 19, 2005 11:39 PM
I just recently brought in a candidate like this... (it was my 2nd phone screen so I think I deserve some slack :) )...
I was trying hard to do the right thing in the phone screen, I even had this particular blog open on my laptop at the time! I'd like like to echo that I think there needs to be some training associated with this process... The last two times I've phone screened I felt like I'm made major mistakes... The first one didn't go well, and I'm confident not bringing him in was the right move, it was stressful... And the second one bombed the loop.
I'd stress that, for anyone new to phone screens to really carefully prepare for it, and read all the material on the intranet and in this blog really carefully...
Posted by: Ben B. at January 25, 2005 09:12 PM