Background

Introduction

On average, visually impaired people require either a seeing-eye dog or sighted guide to travel safely at their fastest, but walk significantly slower than the average person when navigating independently using a white cane, which constantly occupies one of their hands and requires active concentration. Handicapped people need more efficient ways to navigate themselves indoors and outdoors. More specifically, a lot of visually impaired people have a need to navigate independently. There are an estimated 253 million people worldwide who suffer from visual impairment [1] and nearly 23,000 seeing eye dogs serving their visually impaired owners worldwide [2]. As such, the visually impaired gaining independence to mitigate the workload for caretakers without compromising their walking speed would be more optimal. 

Currently, the use of a seeing-eye dog or a sighted guide is the most efficient means of navigation for a visually impaired person, but it comes with its required maintenance and daily costs. For a seeing-eye dog, the cost of training is estimated to be around $20,000 to $30,000, and it takes about $200 a month to take care of it [3]. If a visually impaired person is not able to handle tasks to take care of the dog, an additional cost for a caretaker may also be required. This heavily detracts from the appeal of a seeing-eye dog for the visually impaired, which is also shown by its low usage rate of 2% of all visually impaired people in the United States [4].  Traditional methods of navigation can potentially pose to be inaccurate or overloading for a visually impaired person. For example, the white cane is simple and commonly used by the visually impaired community, because it’s cheap and it has little to no maintenance. However, the user must swing it side to side in order to “scan” for obstacles to avoid. This unreliability could lead to unexpected collisions or even a cognitive overload to its user [5]. 

With our multi-channel sound engine working in tandem with a continuous sound output, we aim to have the visually impaired will be able to walk much more confidently without bumping into approaching obstacles. As a result, they will be able to get from point A to point B faster than they can with traditional methods such as walking canes and guide dogs.

The people who most immediately need to use the sound engine employed by the simulation are those who suffer from visual impairment, ranging from partial to complete blindness. Secondarily, the visually impaired would benefit most from being independent. At the third level, their caretakers would be relieved of at least some of their duties. The approximate number of caretakers of the visually impaired is unknown, partially due to the fact that many aren’t paid to do so (i.e family/relatives). The potential market size might be lower considering economic and technological factors. 

State of the Art

Device Inputs 

The two most predominant inputs for consideration are either cameras or some form of distance sensor (namely ultrasonic or LiDAR). The projects we saw involving cameras then relied on image processing to identify obstacles to notify the user of. 

Camera Inputs

One intriguing patent we looked into described a system using cameras as an input device where the video input was then processed through image processing and a convolutional neural network, wherein discrete objects are identified as disparity images and formed as a layer discrete from the rest of the environment, like the floor [9]. These objects are then treated as obstacles and tracked with respect to the user using the disparity image layers. One of the limitations of this input is that it would have the same troubleshooting and limitations of normal cameras, with issues and considerations such as white-balancing, focus, resolution, frame rate, etc. 

Another one of the systems we examined employed a Time of Flight (TOF) sensor on the user’s belt, which is usually run on standard cameras capable of capturing image colors [10]. Time of Flight sensors are extremely similar to LiDARs conceptually, and operate by measuring the time between frames in which an object moves to determine its relative distance. This type of technology is also used for cars to track obstacles in their surroundings as well as for tracking human motion. 

Distance Sensors

Almost every other device we saw involved either an ultrasonic sensor or a LiDAR as an input to its system. They both function on similar principles of that they emit a signal with a known constant velocity (an ultrasonic sound or UV/IR/visible light) and measure how long it takes for that signal to be reflected back. From there, the distance of wherever the sensor is pointing can be calculated from the velocity and time of travel. These sensors are usually 1-dimensional and read a continuous stream of distance data. The most common areas on the user we saw these sensors being intended for were as headgear, belts, or shoes [11]. A more conducive approach was to integrate the device into their existing lifestyle by augmenting the walking cane with an input, controller, power source, and output [12]. 

One of the main distinctions between ultrasonic sensors and LiDARs besides the signal emitted is that ultrasonic sensors are substantially cheaper, but also theoretically less responsive since the velocity of any light emitted by a LiDAR will always be higher than that of the ultrasonic sensor, so there is less latency in LiDARs. An additional consideration is that LiDARs are more expensive, even while considering that 1-Dimensional LiDARs are cheaper than 3-Dimensional LiDARs. 

Audio Outputs

Many new devices that are designed for those who are visually impaired function under the assumption that the user’s hearing is functional. The user would have to listen to audio cues that the device emits when their respective sensors detect something that the user should attempt to avoid.

One such device is proposed to be a wearable smart system that is worn on the wrist [1]. When an object is detected in front of the wearer, it will emit an alarm sound that scales in intensity with how close the object is. Furthermore, this device also has the capability to alert nearby people if the wearer happens to fall over. Simultaneously, the device will also send a text to a previously set list of contacts, notifying them of the incident. Additionally, the user can also say “Help” to trigger the same text alert to their contacts. These are some great features that would greatly benefit a visually impaired person, but the author failed to touch upon some possible limitations for these features. For one, it would be inconvenient for the wearer and the surrounding people if the device were to wrongly report a falling incident. Since a text message is also sent to a list of contacts, those people will also have their daily lives disrupted by a false emergency alert. Another limitation is that the device can only detect 4 meters in front of it, which might not be far enough for the user to react and avoid an obstacle. This problem is exacerbated if the object happens to be moving towards the wearer. 

Another device that uses audio outputs to communicate with its user is the system that proposes using voice recognition and a GPS to navigate the user [5]. It would listen to its wearer using speech recognition software in order to determine where they want to go, and then it uses the GPS system in tandem with a text-to-speech software to communicate directly with the user. It also retains the ability to detect obstacles, for which it will read out “obstacle detected” to the wearer. Overall, the features of this device serve the practical purpose of getting its visually impaired user from one location to another safely. However, its limitations mainly lie in the limitations of its software, not its hardware. For instance, the system uses a text-to-speech software called Espeak, which is an open source software that converts text data into speech data, and vice versa. A conceivable problem is that the software misinterprets what the user says, which leads to the GPS telling them to go to the wrong location. Another example of a limitation is if there’s construction over an entire road, which will force people to navigate around them. However, the GPS may not have updated information about inaccessible roads, which will pose a serious navigational problem for the user. 

Python Libraries

As our project is entirely simulated in Python, which includes a variety of libraries to choose from. For the simulation aspect of the project, we used Pygame solely to build it. Pygame is a library that is designed for writing video games and designing virtual environments, but we used the applicable modules to develop a rudimentary simulation that is capable of turning user input into simulated movement and screen-turning. The simulation is built in two dimensions, but we used Pygame’s raycasting technique to make it appear in three dimensions. We later refer to this as “pseudo-three-dimensional”. 

PySinewave is the audio component of the project that the sound engine is built on. The library’s main purpose is to play a continuous sound in the form of a sound wave that can change pitch and volume based on input parameters. Two sound sources are created as objects to represent the Left and Right audio channels. The distance data from the PyGame simulation is sorted into logarithmic bins that assign appropriate pitches to the given distance bin. The Left and Right sound sources change pitch based on what distance is outputted by the Left and Right rays, and the parameter for the speed of pitch changes is set to a high value so that the information for the user is updated responsively. 

Patent/Intellectual Property Status

The most relevant publication we found was the “Wearable navigation system for the visually impaired” filed 12/03/2018 and published 06/04/2020 under the classifications: G01C21/20 Instruments for performing navigational calculations, G01S15/08 - Systems for measuring distance only, G06T7/70 - Determining position or orientation of objects or cameras, G06K9/00 - Methods or arrangements for recognising patterns. Another relevant publication we found was the “Eyesight Aid System” under the classifications: G09B21/00; Teaching, or communicating with, the blind, deaf or mute. 

Similarly, the most relevant classifications for our project were: G01C21/20 Instruments for performing navigational calculations, G06T7/70 - Determining position or orientation of objects or cameras.