Abstract:

This paper reports work-in-progress of the development of a sign language translation system that enhances communication between the hearing and hearing-impaired communities. The objectives are to design an AI-powered tool capable of translating real-time sign language gestures into English text, and to ensure the system operates in real time with high accuracy and minimal latency. Over 5% of the world’s population, or 430 million people, suffer from disabling hearing loss. This inhibits the communication between non-deaf and deaf people, therefore widening the social gap between them. As such, we choose to tackle this issue by creating an AI-powered tool capable of translating real-time sign language gestures into English text or speech. We have chosen to focus on the Singapore Sign Language (SgSL). As there are an estimated 500,000 deaf people in Singapore, we believe that our AI-powered tool can bridge the segregation between the non-deaf and the deaf in Singapore. Sign language detection and translation using AI have been widely popularised and implemented. Research in this area typically focuses on widely used sign languages such as American Sign Language (ASL). However, little work can be found about SgSL, a unique language that combines elements of ASL and locally developed signs that include popular Hokkien terms. To implement the AI-powered tool, we first detect sign-language poses. After obtaining the poses, we analyse the poses by using deep learning models to train our model The first step in recognising sign languages is pose and gesture estimation. We use Google’s MediaPipe library to detect hands or a full-body pose from a webcam. OpenCV is usually the library of choice when it comes to computer vision and working with webcams. Then, we use different variations of neural networks to recognise them. One of the main challenges for developing an AI system of this calibre is the availability of datasets. While languages such as ASL have readily available and comprehensive datasets online, this is not the case for SgSL. To address this issue, many researchers who worked on niche sign languages rely on data augmentation techniques to artificially expand the limited dataset. Common approaches include randomly rotating, flipping and scaling image and video frames. These techniques help improve the model’s accuracy as it reduces overfitting, allowing such a system to recognise signs more reliably. We have developed the first prototype of our project. This current version focuses on alphabet detection. Its features include a data collection mode, where users can manually input the sign for each letter of the alphabet using a webcam. After that, the program will utilize the collected data to train a model to recognise such signs in real time. This marks the first step toward building a more complete sign language interpreter. Our system is still in the early stages of implementation, with the main challenge being the insufficient annotated SgSL datasets. We have identified strategies such as data augmentation to address this issue, but realistically these methods can only take us so far. A more comprehensive dataset will always be necessary to achieve reliable and accurate performance. As we develop our model further, we hope to successfully develop a system that will make meaningful contributions to the deaf community in Singapore.

Keywords:

Sign Language, Computer Vision, Deep Learning, AI, Localisation.