Oxford - Automatic sign learning

The goal of this work was to automatically learn a large number of British Sign Language (BSL) signs from TV footage. We achieve this by using the supervisory information available from subtitles broadcast simultaneously with the signing. From this material, our objective is to learn the translation of English words to British Sign Language signs.

The problem is cast as one of Multiple Instance Learning: Given an English word we select a set of subtitles which contain the word (positive sequences) and a much larger set of subtitles that do not contain the word (negative sequence). We then find the window (7-13 frames long) which appears often within the positive sequences and seldom in the negative sequence.

We evaluate our approach on a dataset of 210 English words. Our method atomically finds the correct sign with a precision of 68%. For the best 65 returns, this precision increases to 92%.

Example videos: results_65_signs_auto_learned.avi

(xVid codec) auto_sign_learning_example.avi