STATIC SIGN LANGUAGE TRANSLATOR USING HAND GESTURE AND SPEECH RECOGNITION
DOI:
https://doi.org/10.46754/jmsi.2024.10.002Keywords:
Sign language, Hand gesture, Image recognition, Speech recognition, MediaPipeAbstract
Communication between ordinary and deaf people often has issues because ordinary people lack knowledge of sign language. This research aims to help ordinary people communicate with hearing impaired (deaf) people using sign language. This research aims to produce an Android-based mobile app that can translate static sign language using hand movements into text and also convert the spoken voice into sign language using speech recognition. The framework in this research for hand gesture detection uses the MediaPipe software program. This framework allows the creation of apps that translate hand movements into text that help ordinary people understand sign language. Speech recognition in this research uses the Android speech library. This research succeeded in detecting static letters of the alphabet from the a-z and numbers 0-9. Tests of 540 hand gestures carried out in the morning, afternoon, evening, and night had an average detection time of 4.37 seconds. The fastest object detection times were in the morning at a distance of 30 cm with an average detection time of 2.5 seconds. Based on acceptance testing, 83.13% of the features in this static sign language translator have met the users ’users’ needs when communicating with the deaf using sign language.
References
Hall, J. A., Horgan, T. G., & Murphy, N. A. (2019). Nonver-bal communication. Annual Review of Psychology, 70, 271-294. https://doi.org/10.1146/annurev-psych-010418-103145
Burgoon, L. K. G., Judee, K., Valerie, M. (2021). Nonverbal communication. Routledge.
Parera, C. H., Harlili, & Satya, D. P. (2019). Design of translator android application for sign language and Indonesian using user-centered design approach. In 2019 International Conference of Advanced Informatics: Concepts, Theory and Applications (ICAICTA), Yogyakarta, Indonesia, 2019 (pp. 1-6). https://doi.org/10.1109/ICAICTA.2019.8904098
Aditama, P. W., Putra, P. S. U., Yusa, I. M. M. & Putra, I. N. T. A. (2021). Designing augmented reality sibi sign language as a learning media. Journal of Physics: Conference Series, 1810(1), 012038. https://doi.org/10.1088/1742-6596/1810/1/012038
Rakun, E., & Darmana, I. S. A. (2020). Generating of SIBI animated gestures from Indonesian text. In ICEMT 2020: 2020 The 4th International Conference on Education and Multimedia Technology, pp. 256-264. https://doi.org/10.1145/3416797.3416822
Indra, D., Madenda, S., & Wibowo, E. P. (2017). Recognition of Bisindo alphabets based on chain code contour and similarity of Euclidean distance. International Journal on Advanced Science, Engineering and Information Technology, 7(5), 1644-1652. https://doi.
org/10.18517/ijaseit.7.5.2746
Aljabar, A., & Suharjito (2020). BISINDO (Bahasa isyarat indonesia) sign language recognition using CNN and LSTM. Advances in Science, Technology and Engineering Systems Journal (ASTESJ), 5(5), 282-287. https://doi.org/10.25046/AJ050535
Yusnita, L., Rosalina, R., Roestam, R., & Wahyu, R. B. (2017). Implementation of real-time static hand gesture recognition using artificial neural network. CommIT (Com-munication Information Technology Journal), 11(2), 85-91. https://doi.org/10.21512/commit.v11i2.2282
Suchithra, M., Snehaa, S., Nambiar, G., Arya, C., & Diviya Bharrathi, L. B. (2015). Indian sign language recognition system for asymmetrical hand gestures. International Journal of Applied Engineering Research (IJAER), 10(55), 2628-2632.
Raheja, J., Singhal, A., Sadab, N., & Chaudhary, A. (2015). Android based portable hand sign recognition system. In Gate to Computer Science and Research (pp. 1-18). https://doi.org/10.15579/gcsr.vol3.ch1
Handhika, T., Zen, R. I. M., Murni, N., Lestari, D. P., & Sari, I. (2018). Gesture recognition for Indonesian Sign Language (BISINDO). Journal of Physics: Conference Series, 1028(1), 012173. https://doi.org/10.1088/1742-6596/1028/1/012173
Tsai, T., Huang, C., & Zhang, K. (2019). Design of hand gesture recognition system for human-computer interaction. Multimedia Tools and Applications, 79(9-10), 5989-6007. https://doi.org/10.1007/s11042-019-08274-w
Thao, N. D., Anh, N. V., Ha, L. T., & Duyen, N. T. (2021). Robustify hand tracking by fusing generative and discriminative methods. VNU Journal of Science Computer Science and Communication Engineering, 37(1), 16-25. https://doi.org/10.25073/2588-1086/vnucsce.261
Li, S., & Deng, W. (2020). Deep Facial Expression Recognition: A survey. IEEE Transactions on Affective Computing, 13(3), 1195-1215. https://doi.org/10.1109/taffc.2020.2981446
Halder, A., & Tayade, A. (2021). Realtime vernacular sign language recognition using MediaPipe and machine learning. International Journal of Research Publication and Reviews, 2(5), 9-17. https://ijrpr.com/uploads/V2ISSUE5/IJRPR462.pdf
Bazarevsky, V., & Zhang, F. (2019). On-Device, RealTime hand tracking with MediaPipe. Google AI Blog. https://research.google/blog/on-device-real-time-hand-tracking-withmediapipe/
Subhalaxmi Chakraborty, N., Bandyopadhyay, P., Chakraverty, S. B., & Zinnia Sarkar, S. G. (2021). Indian Sign Language Classification (ISL) using Machine Learning. American Journal of Electronics & Communication, 1(3), 17-21. https://doi.org/10.15864/ajec.1304
Kumar, A., Raushan, R., Aditya, S., Jaiswal, V. K., & Divyashree, M. (2017). An innovative communication system for deaf, dumb and blind people. International Journal Research Applied Science & Engineering Technology, 5(VI), 1933-1939. https://www.ijraset.com/
fileserve.php?FID=8560
Abdallah, K. (2014). Gesture-based human computer interactions: Signs language recognition. Open Science Repository. https://doi.org/10.7392/openaccess.23050480
Oudah, M., Al-Naji, A., & Chahl, J. (2020). Hand gesture recognition based on computer vision: A review of techniques. Journal of Imaging, 6(8), 73. https://doi.org/10.3390/jimaging6080073
Cheok, M. J., Omar, Z., & Jaward, M. H. (2017). A review of hand gesture and sign language recognition techniques. International Journal of Machine Learning and Cybernetics, 10(1), 131-153. https://doi.org/10.1007/s13042-017-0705-5
Anilkumar, A., KT, A., Sajan, S., & KA, S. (2021). Pose estimated yoga monitoring system. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3882498
Mi, W., Zhou, N., & Jin, Y. (2017). A real-time Cyber-Physical system for indoor environment monitoring. Computer Science and Engineering. https://doi.org/10.12783/dtcse/icmsie2016/6356
Andersen, C. W., Armiento, R., Blokhin, E., Conduit, G. J., Dwaraknath, S., Evans, M. L., Fekete, Á., Gopakumar, A., Gražulis, S., Merkys, A., Mohamed, F., Oses, C., Pizzi, G., Rignanese, G., Scheidgen, M., Talirz, L., Toher, C., Winston, D., Aversa, R., . . . Yang, X. (2021). OPTIMADE, an API for exchanging materials data. Scientific Data, 8(1), 217. https://doi.org/10.1038/s41597-021-00974-z
Norton, B. (2021). APIs: A common interface for the Global Biodiversity Informatics community. Biodiversity Information Science and Standards, 5, e75267. https://doi.org/10.3897/biss.5.75267
Black, D., Rapos, E. J., & Stephan, M. (2019). Voice-driven modeling: Software modeling using automated speech recognition. In 2019 ACM/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C), Munich, Germany, 2019, (pp. 252-258). https://doi.org/10.1109/models-c.2019.00040B.
Jolad, B., & Khanai, R. (2019). An art of speech recognition: A review. In 2019 2nd International Conference on Signal Processing and Communication (ICSPC), Coimbatore, India, 2019, (pp. 31-35). https://doi.org/10.1109/icspc46172.2019.8976733
Africa, A. D. M., Tabalan, A. R. V., & Tan, M. A. A. (2020). Speech emotion recognition using support vector machines. International Journal of Emerging Trends in Engineering Research, 8(4), 1212-1216. https://doi.org/10.30534/ijeter/2020/43842020
Kamble, B. C. (2016). Speech recognition using artificial neural network – A review. International Journal of Computing, Communication and Instrumentation Engineering, 3(1), 1-4. https://doi.org/10.15242/ijccie.u0116002
Bhakre, S. K., & Bang, A. (2016). Emotion recognition on the basis of audio signal using Naive Bayes classifier. In 2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI), Jaipur, India, 2016, (pp. 2363-2367). https://doi.org/10.1109/ICACCI.2016.7732408
Kose, M. R., Ahirwal, M. K., & Kumar, A. (2021). A new approach for emotions recognition through EOG and EMG signals. Signal Image and Video Processing, 15(8), 1863-1871. https://doi.org/10.1007/s11760-021-01942-1
Hidayat, R., Bejo, A., Sumaryono, S., & Winursito, A. (2018). Denoising speech for MFCC feature extraction using wavelet transformation in speech recognition system. In 2018 10th International Conference on Information Technology and Electrical Engineering (ICITEE), Bali, Indonesia, 2018, (pp. 280-284). https://doi.org/10.1109/iciteed.2018.8534807
Inoue, K., Milhorat, P., Lala, D., Zhao, T., & Kawahara, T. (2016). Talking with ERICA, an autonomous android. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 212-215. https://doi.org/10.18653/v1/w16-3625
Agustin, E. I., Yunardi, R. T., & Firdaus, A. A. (2019). Voice recognition system for controlling electrical appliances in smart hospital room. Telecommunication Computing Electronics and Control, 17(2), 965. https://doi.org/10.12928/TELKOMNIKA.V17I2.11781
Meliones, A., & Duta, C. (2019). SeeSpeech: An android application for the hearing impaired. In PETRA ’19: Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments. https://doi.org/10.1145/3316782.3324013
Sagale, U., Bhutkar, G., Karad, M., & Jathar, N. (2018). An eye-free android application for visually impaired users. In Ray, G., Iqbal, R., Ganguli, A., Khanzode, V. (Eds.), Ergonomics in caring for people. Singapore: Springer. https://doi.org/10.1007/978-981-10-4980-4_36
Barata, M., Salman, A. G., Faahakhododo, I., & Kanigoro, B. (2018). Android based voice assistant for blind people. Library Hi Tech News, 35(6), 9-11. https://doi.org/10.1108/lhtn-11-2017-0083
Landicho, J. A. (2016). Voisee Communicator: An android mobile application for hearing impaired and blind communications. International Journal of Interactive Mobile Technologies (iJIM), 10(4), 26. https://doi.org/10.3991/ijim.v10i4.5859
Downloads
Published
Issue
Section
License
Copyright (c) 2024 Journal of Mathematical Sciences and Informatics
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
