JCSE, vol. 10, no. 3, pp.95-101, 2016

DOI: http://dx.doi.org/10.5626/JCSE.2016.10.3.95

Enhanced Sign Language Transcription System via Hand Tracking and Pose Estimation

Jung-Ho Kim, Najoung Kim, Hancheol Park, and Jong C. Park
School of Computing, Korea Advanced Institute of Science and Technology, Daejeon, Korea

Abstract: In this study, we propose a new system for constructing parallel corpora for sign languages, which are generally underresourced in comparison to spoken languages. In order to achieve scalability and accessibility regarding data collection and corpus construction, our system utilizes deep learning-based techniques and predicts depth information to perform pose estimation on hand information obtainable from video recordings by a single RGB camera. These estimated poses are then transcribed into expressions in SignWriting. We evaluate the accuracy of hand tracking and hand pose estimation modules of our system quantitatively, using the American Sign Language Image Dataset and the American Sign Language Lexicon Video Dataset. The evaluation results show that our transcription system has a high potential to be successfully employed in constructing a sizable sign language corpus using various types of video resources.

Keyword: Sign language recognition; Hand tracking; Depth prediction; Pose estimation; Transcription

Full Paper:   470 Downloads, 1367 View