With the emergence of camera technology, visual tracking has witnessed great attention in the field of computer vision. For instance, numerous discriminative correlation filter (DCF) methods are broadly used in tracking, nevertheless, most of them fail to efficiently find the target in challenging situations which leads to tracking failure throughout the sequences. In order to handle these issues, we propose contextual information-based spatial variation with a multi-feature fusion method (CSVMF) for robust object tracking. This work incorporates the contextual information of the target to determine the location of the target accurately, which utilizes the relationship between the target and its surroundings to increase the efficiency of the tracker. In addition, we integrate the spatial variation information which measures the second-order difference of the filter to avoid the over-fitting problem caused by the changes in the filter coefficient. Furthermore, we adopt a multi-feature fusion strategy to enhance the target appearance by using different metrics. The tracking results from different features are fused by employing a peak-to-sidelobe ratio (PSR) which measures the peak strength of the response. Finally, we conduct extensive experiments on TC128, DTB70, UAV123@10fps, and UAV123 datasets to demonstrate that the proposed method achieves a favorable performance over the existing ones.