The continuous advancement of visual object tracking (VOT) algorithms has made significant contributions to computer vision and video processing. Correlation filter (CF)-based trackers have achieved promising tracking performance in VOT, but still face some problems under scale changes, occlusion, and boundary effects. To deal with these constraints, the learning spatial variance-key surrounding-aware correlation filter (LSVKSCF) via multi-expert deep feature fusion is proposed in this work. Initially, we extract the surrounding samples from the input based on the target size and shape, which integrates the context information and maintains the integrity of the target. Then, we introduce selective spatial information to the filter for penalizing the background and preserving the valid information of the target. Additionally, the spatial variance information is utilized to find the second-order difference of the filter which avoids over-fitting problems. Furthermore, we introduce the keyframes by periodically selecting the surrounding patches that handle the background distractions and improve the tracking speed. Finally, a multi-expert strategy is used in the filter learning stage which shows the reliability of the proposed method. Extensive experiments on OTB2013, OTB2015, TempleColor128, UAV123, UAVDT, and DTB70 datasets prove that our method performs well against several existing trackers.