Mode estimation on matrix manifolds