Towards subject independent continuous sign language recognition: A segment and merge approach - Kong, W.W.; Ranganath, Surendra (Pattern recognition, 2014)
This paper presents a segment-based probabilistic approach to robustly recognize continuous sign language sentences. The recognition strategy is based on a two-layer conditional random field (CRF) model, where the lower layer processes the component channels and provides outputs to the upper layer for sign recognition. The continuously signed sentences are first segmented, and the sub-segments are labeled SIGN or ME (movement epenthesis) by a Bayesian network (BN) which fuses the outputs of independent CRF and support vector machine (SVM) classifiers. The sub-segments labeled as ME are discarded and the remaining SIGN sub-segments are merged and recognized by the two-layer CRF classifier; for this we have proposed a new algorithm based on the semi-Markov CRF decoding scheme. With eight signers, we obtained a recall rate of 95.7% and a precision of 96.6% for unseen samples from seen signers, and a recall rate of 86.6% and a precision of 89.9% for unseen signers. •Variations in sign language are examined to develop a signer independent system.•A 4-channel phoneme-based approach is used.•Continuous sentence is segmented into sign or movement epenthesis sub-segments.•Sign sub-segments are merged and recognized with a two-layer CRF.•Novel decoding scheme is proposed for the semi-Markov CRF used in the 2-layer CRF.