Controllable Text Generation Beyond Auto-Regressive Models

Abstract: 

Recent advances in large auto-regressive language models have demonstrated strong results in generating natural languages and significantly improved the performances for applications such as machine translation and summarization. However, when the generation tasks are open-ended and the content is under-specified, existing techniques struggle to generalize to new scenarios and to generate long-term coherent and creative content. This happens because the generation models are trained to capture the surface patterns (i.e. sequences of words) following the left-to-right order, instead of capturing underlying semantics and discourse structures. It is hard to impose structural or content control/constraints on the model. In this talk, I will present our recent works on controllable text generation that go beyond the prevalent auto-regressive formulation. We propose novel insertion-based generation models and controllable decoding-time algorithms to steer models to better conform to constraints, with applications to creative story generation, poetry generation, and machine translation.