Current approaches to context-aware MT rely on a set of surface heuristics to translate pronouns, which break down when translations require real reasoning. We create a new template test set ContraCAT to assess the ability of Machine Translation to handle the specific steps necessary for successful pronoun translation.