The visual image of a talker provides information complementary to the acoustic speech waveform, and enables improved reconition accuracy, especially in the environments corrupted by high acoustic noise or multiple talkers. Because most of the phonologically relevant visual information can be obtained from the mouth and lips, it is important to measure their dynamics in an accurate and robust manner, Moreover it is desirable to extract information on the mouth and lips without use of artificial invasive markers or patterned illumination. In this thesis, a new method is proposed to extract features of the lips from a color image. The color image is transformed into one that can sharply represent the inner and outer lip contour in spite of the existence of the tongue, teeth and shadow in the image. Thus, we can obtain more exact length information such as heights and width of the lips in the talker's image.
Visual speech features, consisting of shape and color information of the lips, are extracted from the lip-tracking results of many words spoken by many people. Then, hidden Markov models(HMMs) are trained using these features. Audio-visual speech recognition as well as visual recognition is performeed for isolated digit recognition using late integration of audio-visual features. All speech recognition experiments are performed in a speaker-dependent way, using HMMs with 20 states and 10 mixture components and the same speakers for training and testing. The visual only recognition system yields the performance up to 91.5% recognition rate. Addition-ally white gaussian noise is added to the audio signal resulting in signal-to-noise ratio(SNR) from 20dB to -20dB. The experimental result shows that, even though the audio recognition system shows bad performance in the presence of noise, the performance of the audio-visual recognition system is close to that of the visual system. The developed method can be applied for the deformable template with better results than the conventional ones.
