Towards subject independent continuous sign language recognition: A segment and merge approach

• Published in: Pattern recognition Vol. 47; no. 3; pp. 1294 - 1308
• Main Authors: Kong, W.W., Ranganath, Surendra
• Format: Journal Article
• Language: English
• Published: Kidlington Elsevier Ltd 01.03.2014; Elsevier
• Subjects: Algorithms; Applied sciences; Bayesian network; Boron nitride; Channels; Classifiers; Coding, codes; Conditional random field (CRF); Exact sciences and technology; Gesture recognition; Hidden Markov model (HMM); Information, signal and communications theory; Pattern recognition; Recall; Recognition; Semi-Markov CRF; Sentences; Sign language recognition; Signal and communications theory; Signal processing; Signal representation. Spectral analysis; Signal, noise; Signer independence; Speech processing; Support vector machine (SVM); Support vector machines; Telecommunications and information theory; Sign language recognition; Support vector machine (SVM); Hidden Markov model (HMM); Conditional random field (CRF); Semi-Markov CRF; Gesture recognition; Signer independence; Bayesian network; Automatic classification; Semimarkovian process; Probabilistic approach; Decoding; Support vector machine; Conditional probability; Algorithm; Signal classification; Random field; Sign language; Hidden Markov models; Bayes network; Piezoelectric resonator; Sentence; Language recognition; Coupled resonator
• ISSN: 0031-3203; 1873-5142
• DOI: 10.1016/j.patcog.2013.09.014

This paper presents a segment-based probabilistic approach to robustly recognize continuous sign language sentences. The recognition strategy is based on a two-layer conditional random field (CRF) model, where the lower layer processes the component channels and provides outputs to the upper layer for sign recognition. The continuously signed sentences are first segmented, and the sub-segments are labeled SIGN or ME (movement epenthesis) by a Bayesian network (BN) which fuses the outputs of independent CRF and support vector machine (SVM) classifiers. The sub-segments labeled as ME are discarded and the remaining SIGN sub-segments are merged and recognized by the two-layer CRF classifier; for this we have proposed a new algorithm based on the semi-Markov CRF decoding scheme. With eight signers, we obtained a recall rate of 95.7% and a precision of 96.6% for unseen samples from seen signers, and a recall rate of 86.6% and a precision of 89.9% for unseen signers. •Variations in sign language are examined to develop a signer independent system.•A 4-channel phoneme-based approach is used.•Continuous sentence is segmented into sign or movement epenthesis sub-segments.•Sign sub-segments are merged and recognized with a two-layer CRF.•Novel decoding scheme is proposed for the semi-Markov CRF used in the 2-layer CRF.