Hierarchical policy learning involves learning a hierarchy of policies where a low-level policy performs motor control actions and a high-level policy directs the low-level policy to solve a task. we extend a short-horizon latent skill based policy (Play-LMP), trained with imitation learning, to temporally extended horizons by combining it with offline reinforcement learning (CQL) as a high level policy. Unlike prior work, we learn completely from offline, diverse and unstructured data, no expert trajectories need to be collected and no environment resets need to be made, making our approach very scalable.