In many machine learning applications, one only has a limited number of training samples. To alleviate this problem, a successful approach is meta-learning, which tries to extract meta-knowledge from similar historical tasks. Obviously, the larger the number of tasks to learn from, the more meta-knowledge can be learned. However, popular meta-learning algorithms like MAML only learn a globally-shared meta-model. This can be problematic when the task environment is complex, and a single meta-model is not sufficient to capture diversity of the meta-knowledge. Moreover, with a large number of tasks, accessing all task gradients in each training iteration may not be feasible. The sampling of tasks in each iteration also increases variance in the stochastic gradient, resulting in slow convergence. In this talk, we propose to address these problems by structuring the task model parameters into multiple subspaces, so that each subspace represents one type of meta-knowledge. Moreover, we propose a scalable solver with theoretical optimality guarantees based on the improvement function. Variance reduction is also incorporated into meta-learning to achieve fast convergence. Experiments on various meta-learning tasks demonstrate its effectiveness over state-of-the-art algorithms.