Text Sonification
Towards Sound-to-Sight Sensory Substitution
Towards Sound-to-Sight Sensory Substitution
The inspiration behind our project lies in Sensory Substitution Devices (SSDs), which aim to use one form of sensory stimulation to replicate neurological responses typical of another form. Through use of SSDs, individuals with sensory disabilities are able to gain more information about the world they live in. To cite one example in particular, a visual-to-auditory SSD is one that attempts to substitute an individual's vision with auditory signals.
Beyond sensory-substitution, text sonification has a whole a field dedicated to it called literacoustics that has applications in multimodal data analysis, musical experimentation, and even collaborative storytelling.
Our project in particular aims to visually sonify text and images, and to provide another method for individuals who lose their sight during their lives to interpret visual "inputs". In order to do this, we attempt to synthesize musical, auditory representations of the visual appearance of individual letters and words.
However, we do not attempt to claim that we have replicated the neurological responses created by the sense being substituted, and given that true SSDs aim to successfully replicate the neural signals generated by the sense that was lost, our project thus can not be classified as a true SSD. We hope that our text sonification device could be a feasible stepping stone towards a true visually-based SSD for written text or shapes.
This ssd is called the vOICe and was created in 2007. It sonifies black and white images into "soundscapes", so one would be able to "see" with sound. After viewing the video, one is able to see how it once an object enters the field of view, sounds start to play indcating the location of the object. It also demonstrates the steep learning curve that comes with all SSDs.