Background subtraction is used as the first step in two main kinds of applications. Practically, it exist datasets to test algorithms in the different context:
1. Intelligent Visual Surveillance of Human Activities (31 datasets)
2. Intelligent Visual Observation of Animals and Insects (6 datasets)
For a survey, please see:
R. Kalsotra, S. Arora, "A Comprehensive Survey of Video Datasets for Background Subtraction", IEEE Access, 2019.