A WEBCAM-BASED RECOGNITION SYSTEM FOR THE UZBEK FINGERSPELLING ALPHABET
DOI:
https://doi.org/10.51699/x67ey438Keywords:
Uzbek Sign Language, Fingerspelling Alphabet, Mediapipe, Random Forest, LSTM, WebcamAbstract
For individuals with hearing or speech impairments, sign language serves as the primary channel of natural connection with society. To date, work on the automated recognition of Uzbek sign language remains scarce, and an openly available dataset for this purpose is essentially nonexistent. This paper presents a system that recognizes the Uzbek fingerspelling alphabet using an ordinary webcam. Hand movements are tracked with the MediaPipe Hands tool; letters represented by static hand poses are classified using a Random Forest classifier, while letters expressed through motion are recognized using an LSTM neural network. Trained on a manually collected dataset of 1,523 static frames and 468 motion sequences, the system recognized static letters with 99.11% accuracy and dynamic letters with 65.96% accuracy. The results indicate that the recognition of stable hand poses can be achieved with high reliability, whereas the recognition of temporally evolving gestures remains a considerably more complex scientific and technical challenge.
References
[1] World Health Organization, “Deafness and hearing loss: Fact sheet,” WHO, 2021.
[2] President of the Republic of Uzbekistan, “Resolution No. PQ-2909 of April 21, 2017: On the rights of persons with disabilities,” 2017.
[3] F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, S. Wei, and C. L. Chang, “MediaPipe Hands: On-device real-time hand tracking,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), 2020.
[4] Z. Cao, G. Hidalgo, T. Simon, S. E. Wei, and Y. Sheikh, “OpenPose: Realtime multi-person 2D pose estimation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 1, pp. 172–186, 2021.
[5] L. Breiman, “Random forests,” Mach. Learn., vol. 45, no. 1, pp. 5–32, 2001.
[6] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, 1997.
[7] O. Koller, S. Zargaran, and H. Ney, “Deep Sign: Hybrid CNN-HMM for continuous sign language recognition,” in Proc. Brit. Mach. Vis. Conf. (BMVC), 2016.
[8] N. C. Camgoz et al., “Neural sign language translation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2018.
[9] D. B. Jurayev, “Technology for constructing a dataset for translating into Uzbek sign language based on MediaPipe,” Int. J.: Theory Pract., vol. 8, no. 3, 2025.
[10] “A machine learning algorithm to assist people with speaking disabilities for sign language detection of people speaking Uzbek language,” in Proc. ICTACS, 2024.
[11] O. A. Kayumov, N. R. Kayumova, and D. F. Xodjabekova, “Uzbek sign language classifier based on machine learning,” Eur. Int. J. Multidisciplinary Res. Manage. Stud., vol. 4, no. 5, pp. 269–280, 2024, doi: 10.55640/eijmrms-04-05-43.
[12] O. Koller, N. C. Camgoz, H. Ney, and R. Bowden, “Weakly supervised learning of sign language,” Found. Trends Comput. Graph. Vis., vol. 13, no. 3, pp. 1–118, 2020.
[13] N. C. Camgoz, S. Hadfield, O. Koller, and R. Bowden, “Sign language transformers: Joint end-to-end sign language recognition and translation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020, pp. 10023–10033.
[14] O. Koller, O. Zargaran, H. Ney, and R. Bowden, “Deep sign: Hybrid CNN-HMM for continuous sign language recognition,” in Proc. Brit. Mach. Vis. Conf. (BMVC), 2016.
[15] A. M. Sincan and H. Y. Keles, “AUTSL: A large scale multi-modal Turkish sign language dataset and baseline methods,” IEEE Access, vol. 8, pp. 181340–181355, 2020.
