Case Studies
1 Active learning
Active learning is any learning activity in which the student participates or interacts with the learning process, as opposed to passively taking in the information. In practice, learners have a pool of unlabelled data and use different strategies to choose whether to query their labels or not. Different strategies would result in different performances. If a strategy can select more informative points, its testing accuracy would increase much quicker. But whether a strategy could successfully choose informative points or not is hard to tell. Here we propose to use our DVI tool to visualize the positions of selected points as supporting information.
1.1 Details of 4 active learning query strategies
Random: randomly select points from the unlabeled pool.
Uncertainty-based method:
Least Confident: choose points from the unlabeled pool with the least confidence scores.
LL4AL (Learning Loss for Active Learning): choose points with the highest loss from the unlabeled pool.
Diversity-based method:
Coreset: choose points that can satisfy ε-cover condition from the unlabeled pool.
Based on the intuition of 4 strategies, our visualization should show:
random strategy should select points with a distribution similar to unlabeled data pool
Least Confident and LL4AL strategies should select points near decision boundaries
Coreset strategies should select points that are diverse and can cover the unlabeled pool.
*new selection points are shown as highlight red points whereas unlabeled pool are shown in black*
Random
As shown in screenshots, random strategy do choose points similar to the distribution of unlabeled pool.
data pool
new selction
new selections are highlighted in red, while true distribution(unlabeled pool) are in black. This shows that new selections are similar to true distribution.
Least Confidence(Uncertainty-based)
As shown in the screenshots, new selections are lying close to decision boundaries.
data pool
New selections are closer to decision boundaries(white regions).
New selctions are clearly closer to decision boudaries comparing to unlabeled data.
LL4AL(Unceertainty-based)
As shown in the screenshots, new selections are lying close to decision boundaries.
data pool
New selections are closer to decision boudaries(white regions).
New selctions are clearly closer to decision boudaries comparing to unlabeled data.
coreset(Diversity-based)
As shown in the screenshots, the new selections are much diverse. Different from random strategy, it has more points around decision boundaries. And similarly, it covers more data from main clusters than uncertainty based methods.
data pool
New selections are more diverse and cover both white regions and unlabeled data clusters,
Different from random strategy, it has more points around decision boundaries(white regions).
Comparing to uncertainty based method, it covers more data from main clusters.
1.2 Examples in Paper
random
coreset
LL4AL
2 noisy data
2.1 Tool demo
Training dataset is important for machine learning. However, the presence of mislabeled data could affect the deep neural network. We are interested in how those mislabeled data or noisy data would affect subject models. Again, we propose to use DVI tool to visualize how noisy data would behave in the training process.
Here we show that, by searching and pinpointing the interesting samples and tracking their movements, DVI can further allow users to zoom in to a local region and check the sample details including labels and appearances, which serve as a potential model debugging facility.
2.1 Details of noisy dataset setting
We generate symmetric noise by flipping the label of 10% of the CIFAR-10 samples to train a classifier. The figures below show the process of how clean/noisy sample embeddings are learned during training.
Comparing to the clean samples smoothly pulled into their color-aligned territory in the first few epochs, noisy samples show "reluctance'" to be pulled (i.e., learned). Those "hard'' samples continue to stay in their "original'' territory in early-mid epochs, but some are forcefully pulled into their "expected'' territory in the late training process.
clean in epoch 10
clean in epoch 100
clean in epoch 150
clean in epoch 200
noisy in epoch 10
noisy in epoch 100
noisy in epoch 150
noisy in epoch 200
E10
E50
E100
E150
E200