Although great progress in visual tracking has been witnessed, recent visual trackers still adopt a relatively primitive and unitary inference strategy that only selects a location with the maximum response score as the final tracking result in each frame. However, locations with maximum scores may not be the optimal choice, especially when encountering challenging tracking scenarios like heavy occlusions and fast motion, in which errors will be accumulated continuously and thus scores are not reliable for tracking. In this paper, we instead propose a novel beam search-based multi-agent reinforcement learning strategy (named as BeamTracking) to improve visual tracking by maintaining and analyzing multiple tracking results simultaneously. In our method, we consider different tracking results as different tracking agents and formulate the tracking task as a sample selection problem based on multiple parallel decision-making processes, each of which aims at picking out one sample as their tracking result in each frame. Then, we introduce a multi-agent reinforcement learning (MARL) framework which mainly consists of a novel GRU-based context-aware observation network and multiple policy networks, to fulfill the tracking task. By applying our tracking strategy to the whole video sequence and maintaining multiple estimated trajectories for the target, we can achieve more robust tracking by selecting the trajectory with the maximum score. Extensive experiments on eight popular tracking benchmarks validated the effectiveness of the proposed algorithm.