The industrial robots are isolated from human in the workspace, but service or rehabilitation robots coming in future, require cooperation with human, and so the man-machine interface needs more human-friendly capability. In recent years, the advances of man-machine interface, which includes computer graphics, gesture, voice or facial expression, have been made in robot and automation, communications, virtual reality or multimedia. In particular, the recognizing system of Korean Sign Language(KSL) using dynamic hand gesture to assist communication between the hearing-impaired persons, who use nonverbal communications, and the normal persons, was evaluated as the good application in which the recent high-technology meets human welfare. During the developing process, it was observed that the deaf-mute using nonverbal communications use facial expressions as more basic communication means than hand gestures. They use facial expressions more stronger than the normal do, so far as they hardly understand the persons who perform only hand gesture of KSL without facial expressions.
According to psychology, for even the normal in face-to-face communication, facial expressions are more important than paralanguage such as voice tone or body language. Recognition of facial expressions is thought very important and much necessary in man-machine interface of future robots. Facial expressions in psychology have been studied for a long time, but there are still many questions involved about human understanding facial expression of emotion. Despite of the fact that facial expressions are understood differently according to situations and persons, experience and familiarity with the person, according to the study by Ekman, psychology professor, the facial appearance of six emotions, surprise, fear, disgust, happiness, anger and sadness, are universal, not learned differently in each culture.
Since the characteristics of facial expressions make it very difficult to computerize, use of visual observations by experts can be essential in establishing the recognition system. In this work, a systematic framework using soft computing techiques for recognizing facial expressions of emotion from color facial images is presented. The fuzzy rule based classifier will effectively make use of explicit human knowledge based on description of visual analysis by experts, Ekman and Friesen. Analysis is performed for the universal visual cues of the six principle facial expressions and the seven fuzzy systems (14x1) are built through selection process of linguistic variables and their values, and determination of the relationship with each emotion. To handle a large rule base problem, which stems from the use of many linguistic variables, decomposition using specific visual characterisitics and the rough set approach are used. The rough set approach is shown to be automatic and effective method for implementing the fuzzy system of recognizing facial expressions.
And a fuzzy observer with learning capability is proposed indirectly to measure linguistic variables used in the fuzzy system. While the linguistic variables used in the fuzzy system necessitate to measure wrinkle features, for example, nasolabial fold, crow's feet wrinkles, nose wrinkles, or shapes of human feeling such as stretched/squarish mouth, it is not an easy task to extract features directly from camera image of human face since appropriate mathematical models for features are not easily determined. If any measurement technique is available, it should be capable of tolerating vagueness or uncertainty that are inherent in human linguistic values. In this work, it is proposed to transform classical image features and numerical information into linguistic variables which are measured by a fuzzy observer and expressed as fuzzy numbers. A feedforward multilayered artificial neural network(ANN) is employed to develop parameter adjustment of the fuzzy observer based on available crisp input--fuzzy ouput sample sets. An experiment is described for indirectly measuring facial wrinkles by the proposed fuzzy observer.
The Discrete Fourier Transform(DFT) of a scanned data, called slice DFT, obtained from a 2D image is presented for the facial wrinkle features to be extracted from images. The slice DFT uses a representation of 2D sequences as 1D sequences which is formed by scanning x-directional or y-directional image data. The local frequency analysis around 1Hz in the normalized frequency domain shows characteristics for facial wrinkles independent of an image size. The spectral features using the slice DFT are proposed and are used as an automatically extracted region-based gray-level image features for facial wrinkles in the nasolabial fold.
To automatically extract regions for face, eyes, eyebrows, mouth, and nostrils from facial images, a thresholding technique in a color space and the integral projection profile analysis are used. The fuzzy system is integrated with the proposed fuzzy observer, the proposed image features in the nasolabial fold region and a simple fuzzy rule set for happiness. And an experiment was performed for automatically recognizing happiness on a real color facial image sequence. It takes about 5 sec in total to recognize a single face image in Indigo2 workstation. The recognition rate for the degree of happiness is 100% for 15 images. Since most of time is spent on low-level face processing such as color space transform and connected component analysis which can be improved with ease in high-speed DSP system, the proposed approach is shown to be an effective and efficient approach.
