Abstract

Fight detection in multimedia sources and videos can be extremely useful in several applications such as video surveillance, parental control, etc. In this work, different approaches for feature extraction, training, and classification are used to detect violence, particularly physical fights, in videos. The performance of these approaches is then compared.

Results show that extracting Convolutional Neural Network (CNN) features from the training set and using them to train a Support Vector Machine (SVM) classifier gave 96.3% classification accuracy of the testing set. In addition, using those CNN features to train a K-Nearest Neighbor (KNN) classifier with 5 neighbors and a Decision Trees classifier gave 91.55% and 67.6% accuracy respectively. On the other hand, classifying the entire dataset using a pretrained CNN classifier gave 71.66% classification accuracy. Moreover, extracting Spatial Temporal Interest Points (STIP) features and using them to train an SVM classifier gave 55.94% classification accuracy. In addition, Augmented Affine Trajecton with SVM classifier gave 61.88% accuracy.

Introduction

Human action recognition works towards recognizing actions. It has been one of the prime areas of research in computer vision. However, most research has focused on simple actions like running, walking, jogging, etc., while detection of fights or aggressive behaviour has received less attention. This detection of fights from multimedia sources or video streams plays an important role in ensuring safety. It can be extremely useful in several applications such as video surveillance, parental control, etc.

Various methods have been proposed in the literature for both feature detection of space-time interest points and description of local video patches in the field of human action recognition such as Convolutional Neural Network (CNN) features, Spatio-Temporal Interest Points (STIP), Gradient Descriptor, HOG features, etc.

Space Time features give local events in a video and can be adapted to represent the frequency and velocity of moving patterns. Hence this representation is robust to dynamic transformations in the video. The Bag of words model is used and finally SVM is used for classification of fight.

In this method a trajectory based human action recognition method has been implemented which gives the adequate spatial or temporal relationships. In this approach, trajectories are extracted by extracting SURF features and tracking these features in consecutive frames and is combined with Bag of Words Model. A method for extracting the affine transformation is implemented which serves a representation of change in temporal features.Finally support vector machine is used to classify the fights.

In this work, CNN features and CNN classifiers are used. CNNs, which are Convolutional Neural Networks, are biologically-inspired variants of Multilayer Perceptron Models (MLP). The building blocks are neurons that have learnable weights and biases. Each neuron receives some inputs, performs a dot product and optionally follows it with a nonlinearity. The whole network still expresses a single differentiable score function from the raw image pixels on one end to class scores at the other [1]. CNN arranges its neurons in three dimensions width, height, and depth. Every layer of a CNN transforms the 3D input volume to a 3D output volume of neuron activations [2].

In this work, given a video describing an action, the action is classified as fight or non-fight and the performance of different feature extraction, training, and classification techniques is compared.

Experiments and results:

In this work, the Peliculas Fight Action dataset is used [6]. This dataset contains 100 videos of fight sequences and 100 videos of non-fight sequences. Several platforms were used for this work such as CAFFE, Theano, and MATLAB. The accuracy and confusion matrices are used to evaluate the performance of the proposed feature extraction and classification techniques. . The results obtained are as follows:

Random Decision Classifier:

Accuracy = 48.5%

The confusion matrix is given by

STIP Feature Extraction with SVM Augmented Affine Trajecton with SVM:

Accuracy = 55.94% Accuracy = 61.88%

The confusion matrix is given by The Confusion matrix is given by

CNN Classifier:

The classification result was the probability value for an image being a fight scene or non-fight scene. For final classification of the image as fight or non-fight, a threshold of 0.1 was empirically obtained. The model which was used for the classification is:

Accuracy =71.66%

The Confusion Matrix is given by

CNN feature Extraction with SVM, KNN, and Decision Trees Classifiers:

CNN with SVM CNN with KNN:

Accuracy = 96.3% Accuracy = 91.55%

The confusion matrix is given by The Confusion matrix is given by

CNN with Decision tree

Accuracy = 67.6%

The confusion Matrix is given by

Qualitative results:

STIP Feature Extraction with SVM Augmented Affine Trajecton with SVM:

KNN

Fight Non Fight

Accurately Classified Inaccurately Classified Accurately Classified Inaccurately Classified

Decision Trees

Fight Non Fight

Accurately Classified Inaccurately Classified Accurately Classified Inaccurately Classified

Conclusion and future work:

In this work, different techniques for feature extraction, training, and classification are used to detect fight actions in videos, and the performance of these different techniques is compared. Using various feature descriptors and extractors, the detection of fight actions is not very accurate because of the involvement of complex features and simple classifiers. Results show that using CNN features with SVM, KNN, and Decision Trees has 96.3%, 91.55%, and 67.6% accuracy respectively. On the other hand, using a pretrained CNN classifier has 71.66% accuracy. In addition, using STIP features with SVM has 55.94% accuracy, and using Augmented Affine Trajecton with SVM has 61.88% accuracy. The best results were obtained using feature extraction from CNN due to its self learning property of deciding the best features. Proper finetuning of CNN with longer iterations will give higher accuracy and less training loss. In addition, since all computations are done using CPU only, challenges arise when fight actions are detected in real time due to the computational power and capacity limitations. For future work, the CNN network can be trained from scratch using a significantly large dataset. In addition, sounds from videos can be used to further enhance the accuracy results.