The scientists from St. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences (SPIIRAS) are working on a robot that will be able to lip-read and understand sign language. It will help disabled customers to do the shopping in supermarkets.
Alexey Karpov, Head of the Speech and Multimodal Interfaces Laboratory, says the following about a new project: “Our Czech colleagues and we are implementing a bilateral project supported by the Ministry of Science and Higher Education of Russia. The project is
aimed to make a mobile information robot with multi-modal user interfaces, a Russian-language speech interface and also a sign interface, so that the robot can interact with deaf people. ”
According to Alexey Karpov, a prototype of a robot based on these technologies will become an assistant for supermarket customers. The machine will be able to understand the voice requests of a customer and the elements of sign language, to get a person to the place with necessary goods and to transport the purchases in a trolley. In addition, the developed system can be used in banks and other organizations to work with clients.
The second related project aims to recognize speech by the movement of lips. This will be useful not only for communication with deaf people but also for speech recognition in noisy places.
“This is a combination of voice recognition and lip reading. In order to perform the research we used a high-speed video camera that produces 200 frames per second, which is several times larger than that of ordinary cameras. We get a video, process sound and then merge the data. This makes possible to improve the accuracy of speech recognition in noisy rooms, outside and so on. Currently, the system can already recognize hundreds of words in Russian in this way”, — says Alexey Karpov.
Source: tass.ru