Galaxy-X

Citation:

W. Dhifli, A. B. Diallo. Toward an Efficient Multi-class Classification in an Open Universe. 12th International Conference on Machine Learning and Data Mining (MLDM), New York City, NY, USA 2016 (Best paper award).

W. Dhifli, A. B. Diallo. Face Recognition in the Wild. 20th International Conference on Knowledge-Based and Intelligent Information & Engineering Systems (KES), York, UK 2016.

Existing classification methods are designed to classify unknown instances within a set of previously known classes that are seen in training. Such classification takes the form of prediction within a closed-set. However, a more realistic scenario that fits the ground truth of real world applications is to consider the possibility of encountering instances that do not belong to any of the classes that are seen in training, i.e., an open-set classification. In such situation, existing closed-set classification methods will assign a training label to these instances resulting in a misclassification.

We introduce Galaxy-X, a novel multi-class classification method for open-set problem. Our method is able to distinguish instances resembling previously seen classes from those that are of unseen classes. Galaxy-X presents high flexibility through a softening parameter that allows extending or shrinking class boundaries adding more generalization or specialization to the classification models.

Example

In this Figure, we show an illustrative example of the transformation of a closed-set classification into a challenging multi-class open-set recognition problem. The figure shows an evaluation in terms of classification accuracy (Figure (a)) and weighted f-measure (Figure (b)) on the classification of handwriting digits (A description of the dataset is available here). The classification performance is shown for a standard closed-set classifier (SVM) versus an open-set classifier (our approach: Galaxy-SVM) (for both approaches, SVM is used with a linear kernel). In the experiment, we only used five handwriting digits in training while instances of the rest of the digits are held out. Each classification scenario was performed for five runs where in each run, the five training classes are divided using a stratified shuffle split with 75% of the training data used in training and 25% of them used in testing. In each classification scenario, we progressively added other testing instances representing one class of the held out digits. Instances of labels that were unseen in training should be labeled as unknown in prediction. As the number of unknown evaluation classes grows, the accuracy and f-measure of SVM drops significantly in contrast to that of our approach which showed higher performance especially in open-set classification scenarios.

Experimental Evaluation

How Open is an Open-set Classification?

We propose Openness as a measure to quantify the openness of a classification scenario (S):

openness(S) = | UnseenLabels| / |labels of the dataset D|

Experimental Settings

- We compare our approach with the multi-class classification strategy One-vs.-Rest (OvR-SVM), the open-set multi-class classifier One-vs.-Set SVM (OvS-SVM) and OCSVM+OvR-SVM. OCSVM+OvR-SVM is a two-step open-set multi-class classifier that we built using a one-class SVM (OCSVM) with an RBF kernel is trained on the entire training instances considered as a single super-class, and OvR-SVM. Instances that are rejected by OCSVM are classified as of "unknown" classes, otherwise they are classified using OvR-SVM.

- For our approach, we use SVM as the closed-set classifier. We show the results of Galaxy-SVM using a fixed softening value δ=-0.3 and for H(yper)-Galaxy-SVM using the optimal δ value for each openness

Evaluation on Face Recognition (URL)