An AI-Based Framework for Translating American Sign Language to English and Vice Versa

Vijayendra D. Avina, Md Amiruzzaman, Stefanie Amiruzzaman, Linh B. Ngo, M. Ali Akber Dewan

Research output: Contribution to journalJournal Articlepeer-review

Abstract

In this paper, we propose a framework to convert American Sign Language (ASL) to English and English to ASL. Within this framework, we use a deep learning model along with the rolling average prediction that captures image frames from videos and classifies the signs from the image frames. The classified frames are then used to construct ASL words and sentences to support people with hearing impairments. We also use the same deep learning model to capture signs from the people with deaf symptoms and convert them into ASL words and English sentences. Based on this framework, we developed a web-based tool to use in real-life application and we also present the tool as a proof of concept. With the evaluation, we found that the deep learning model converts the image signs into ASL words and sentences with high accuracy. The tool was also found to be very useful for people with hearing impairment and deaf symptoms. The main contribution of this work is the design of a system to convert ASL to English and vice versa.

Original languageEnglish
Article number569
JournalInformation (Switzerland)
Volume14
Issue number10
DOIs
Publication statusPublished - Oct. 2023

Keywords

  • ASL
  • deep learning
  • image
  • translation
  • video

Fingerprint

Dive into the research topics of 'An AI-Based Framework for Translating American Sign Language to English and Vice Versa'. Together they form a unique fingerprint.

Cite this