Mendelian Randomization (MR) has proved to be a powerful tool for inferring causal relationships among a wide range of traits using GWAS summary statistics. Great efforts have been made to relax MR assumptions to account for confounding due to pleiotropy. Here we show that sample structure is another major confounding factor, including population stratification, cryptic relatedness, and sample overlap. We propose a unified MR approach, MR-APSS, to account for pleiotropy and sample structure simultaneously by leveraging genome-wide information. By further correcting bias in selecting genetic instruments, MR-APSS allows to include more genetic instruments with moderate effects to improve statistical power without inflating type I errors. We first evaluated MR-APSS using comprehensive simulations and negative controls, and then applied MR-APSS to study the causal relationships among a collection of diverse complex traits. The results suggest that MR-APSS can better identify plausible causal relationships with high reliability, in particular for highly polygenic traits.