This is listed under “technical” interview preparation, but since my background is in labor economics, I’m going to focus on program evaluation technical interviews. The structure mostly applies for forecasting interviews, but the specific methods differ quite a bit. Hopefully I’ll write one of these articles on forecasting later!
I apologize if I what I say below seems obvious. All I can tell you is that I’ve seen candidates violate most of what I mention here at some point.
Some general guidelines about technical interviews before discussing the framework:
1. Stay in territory you know well. If you have to leave that territory, just be honest that you don’t know it that well. The interviewer can ask you to do your best or move on to another subject. Usually it’s the latter – we are not (or at least, shouldn't be) attached to one particular technique and are happy to test you on another technique that would solve the same problem.
There’s nothing we hate more than someone who’s a poser – because that person will eventually provide a bad recommendation to the business with high confidence, making our team look stupid and getting us fired.
2. Lead the interview - Here is a general feature of technical interviews – you, the interviewee, want to be guiding the interview! Don’t be playing defense. Yes, you have to solve the problem they give you – but you can lead them to that solution by asking them questions and having them make choices. Otherwise, you let them bombard you with questions until you get something wrong.
There’s another benefit: by asking questions of us, you often prevent yourself from going down the wrong rabbit hole – which interviewers should prevent but will often allow because interrupting is rude.
There’s also an endowment / reference point effect. This won’t impact all your interviewers, but if you present your interviewer with two choices – we can discuss identifying assumptions of specification details – you are essentially not encouraging them to talk about a third choice, e.g. power calculations.
With that in mind, here is a causal interview framework:
Establish that the interview is, in fact, a causal / program evaluation interview. The best way to do this is to ask about the business’s goals – what do they want to know and why?
You can often outright ask this question. Although it’s rare, sometimes you will be assigned to the wrong *type* of interview, e.g. you’re being given an EIO interview when your background is in experiment design. Verifying the match (or lack thereof) early on saves everyone time and pain. Usually, the interviewer is happy to go back to the recruiter and tell her you need a different interviewer.
You can often establish this more subtly, e.g. “We want to know the impact of adding a widget to the detail page – does adding the widget increase purchases?”
Agree on a specific variable of interest and a specific outcome metric the business cares about: do you care about click-through rate, units sold, revenue, or profit?
Generally, this will be your treatment variable and your LHS variable in your estimation equation.
In a few rare cases, it might not be (e.g. if you are doing a surrogates analysis [add link] or if you can apply a second layer, e.g. revenue to profit using margins). However, in these cases, it will still guide your analysis because you’ll be agreeing on steps to reach that metric.
If this is a forecasting interview, you need the outcome at a given horizon. Based their use case, they may also want the error distribution at point forecast (e.g. we order to the 75th percentile to avoid being out of stock).
Let me pause for a moment here. A good interviewer will probably not let you go down a full interview on the wrong track. But they will look very unfavorably on you if they have to guide you in step one of the interview (agree on the goal).
Think about the thought experiment. When you have a good idea, discuss it with the interviewer and *be clear that it’s a thought experiment*.
This is an exercise for your internal clarity. The reason to do is to think carefully about how you ‘could’ identify what you want under ideal conditions (an experiment).
This exercise is also, fundamentally, a prompt to think about what variation is needed to resolve selection / omitted variable bias issues.
You should also think about whether an experiment is feasible for the business, given time, cost, or customer trust constraints.
Presenting it to the interviewer acts as a further means of clarification. If they disagree that your thought experiment would identify what’s wanted, circle back right away to (1) and (2). Be sure to present it as, “Let me make sure I’m going down the right path. If I could run an experiment by randomly varying treatment T at level J among the units, I would identify the TE of X on Y. Is that what we want to know?” This lets the interviewer know you’re not assuming you can run an experiment. Don’t just assume you can run an experiment. This pisses off lots of interviewers. In many industries, experiments are costly and difficult to get approval. Moreover, they’re often easy to evaluate (though not always, see below), so we won’t have a chance to be super impressed by your skills if you just say, “I’d run an experiment.”
For forecasting, the thought experiment is probably much more related to what data you’d like to have to forecast the object of interest. How would you forecast this if you had good data? Would you do pure time series? Would it be cross-sectional? Or would it be some sort of hybrid model like ARIMAX or Recurrent Neural Network?
Assuming you’ve reached agreement on (3), ask your interviewer if you can run an experiment or if it’s not feasible. If you’re interviewer says it is, great! You can now start upon the design you’ve conceived of in (3) and flesh out the details. If they say no to an experiment, now the real work begins – you have to start fishing for variation in the data. There’s no set formula for this, but a good started is to think about the quasi-experimental methods that could help:
Diff-in-Diff: you can use this for so many interview questions. It’s insane. I’m amazed at how versatile it’s been, both in interview questions and also in actual industry program evaluation. It handles time invariant selection into the program (in most cases).
RD: ask about the program – are there any thresholds (in geography, X variables, or even time) that determine:
Who’s eligible for the program
Which of the applicants get treated
IV: This is dicey in most cases, since IVs are usually weak. But perhaps there’s an e-mail campaign; a classic inducement design.
Matching: sometimes, you can’t get rid of selection. You have to pretend you can see all the stuff that determines it.
Again, once you think you’ve got something, pose it as a question. Get their buy in before you proceed.
Now, once you’ve got the basic design, you can focus on the details, like power calculations, matching methods, other trends, controls for precision, and validating the identifying assumptions (to whatever extent possible). You should definitely know the identifying assumptions, because these are what we have to explain to the business and check in order to give them confidence to use our work.
Notice I don’t say much about which methods you use. There are lots of fancy ones. But once you’ve identified the outcome of interest, the treatment of interest, and the source of variation, you’re more than halfway there. My only advice after that would be that, on the margin, go with the simpler method. You can offer them more complex methods (but only if you know them well!); if they’re interested, this is a good way to earn bonus points. But in practice, we are time strapped on the job and a good linear regression does a lot of work for us. Also, showing clear thinking about linear regression is a good sign that you’ll be able to correctly apply other methods.