Development of an AI Assistant for Individuals with Low Vision and Blindness

The National Eye Institute estimates that in the United States there are presently over 1 million legally blind people and this number will increase to about 4 million by 2050. According to the VA Office of Blind Rehabilitation Services, there are approximately 130,000 US Veterans who are legally blind and more than one

million who have lost the ability to perform daily tasks because of low vision. Some of these people can be helped by relatively simple and inexpensive devices such as magnifiers and canes. However, with more severe vision loss, people are likely to experience loss of independence and reduced quality of life. Blindness has a

major impact on Veterans and others, as evidenced by the fact that over 70% of working-age people with significant visual impairment are unemployed. While there are causes of blindness that can be readily corrected, in many cases there is no effective treatment. One solution is to restore vision by surgically placing electrodes in the retina or visual cortex and

electrically stimulating the brain based on the light level recorded by a head-mounted camera. Systems of this type are being developed and they have great potential. However, there are downsides to the implanted systems that include the risks of surgery, high cost, and a form of vision that is severely limited compared to

normal perception. There is also a wide array of devices and smartphone apps that perform specialized functions (text readers, obstacle detectors, color identifiers). The goal of the proposed research is to develop and test a new type of visual assistive device, named AEyes, for individuals with blindness and low vision. It takes advantage of recent advances in artificial

intelligence including computer vision, machine learning, optical character recognition, speech recognition, and 3D sound rendering to give the user the ability to recognize, localize, and interact with objects and people in their vicinity. In other words, the AI technology that can benefit people with low vision already exists and it will

be implemented in a device tailored to the needs of people with visual disorders. The system will be intuitive and easy to learn and interact with as it understands spoken instructions and it speaks to the user. AEyes has been developed based on a needs-analysis and feedback from focus groups, one-on-one interviews, and

prototype tests with Veterans conducted at the VA Providence Healthcare System. Aim 1 concerns device development. The system architecture and algorithms are largely implemented and functional, but a range of refinements to the software and the integration of separate functional modules into the overall system are needed. These improvements include upgrading to a higher resolution camera,

retraining the recognition neural network, and integrating the face recognition, hand tracking, and voice recognition modules into the central architecture. Aim 2 consists of device tests that will be conducted with subjects having a range of low vision conditions that include no light sense, retinitis pigmentosa, glaucoma, macular degeneration, and diabetic

retinopathy. The tests to be conducted include the ability of a user to recognize and localize people, reach for and grasp objects, read text on handheld paper, computer displays, and signs, and train the system to recognize new people. Comparing performance with residual vision alone and with the addition of AEyes will

establish the device’s effectiveness as a visual assistant and indicate where improvements are needed.