Young Jin Kim

Microsoft

Principal Researcher

Mixture of Experts (MoE) models are an emerging class of sparsely activated deep learning models that have sublinear compute costs with respect to their parameters. In contrast with dense models, the sparse architecture of MoE offers opportunities for drastically growing model size with significant accuracy gain while consuming a much lower compute budget. However, supporting large scale MoE training also has its own set of system and modeling challenges. In this talk, various training algorithms to improve MoE modeling in practice are introduced. Especially, MoE models suffer from overfitting problems with the large memory capacity by design in many cases. Several novel regularization techniques including Stochastic Experts, Gating Dropout and Random Token Selection are introduced together with a multi-task training paradigm. Finally, how those algorithms are improving the real world large scale models will be presented.