Automatic Facial Expression Recognition

Transcription

1 Automatic Facial Expression Recognition Huchuan Lu, Pei Wu, Hui Lin, Deli Yang School of Electronic and Information Engineering, Dalian University of Technology Dalian, Liaoning Province, China Abstract. We present a fully automatic real time system for face detection and basic facial expression recognition from video and images. The system automatically detects frontal faces in the video stream or images and classifies each of them into 7 expressions: neutral, happiness, anger, fear, disgust, sadness and surprise. Each video frame is first scanned in real time to detect upright-frontal faces. The faces found are scaled into image patches of equal size and sent downstream for further processing. Gabor energy filters are applied at the scaled image patches followed by a recognition engine. Best results are obtained by selecting a subset of Gabor features using AdaBoost and then training Support Vector Machines on the outputs of the features selected by AdaBoost. 1 Introduction In this paper we present a fully automatic real time system for face detection and basic facial expression recognition from video and images. The system automatically detects frontal faces in the video stream or images and classifies each of them into 7 expressions: neutral, happiness, anger, fear, disgust, sadness and surprise. The method we present here is a combination of several typical methods. First, the images are converted into a Gabor magnitude representation, using a bank of Gabor filters. Then, we use AdaBoost to select a subset of Gabor features and train Support Vector Machine (SVM) on the outputs of the features selected by AdaBoost. This method was first researched by Bartlett [2]. Our report is based on a series of experiments comparing spatial frequency ranges, feature selection and recognition techniques. The combination of AdaBoost and SVM enhances both speed and accuracy of the system. It can reach a high level of accuracy above 90% in real time. 2 Facial Expression Analysis System 2.1 Preprocessing and Feature Extraction The facial expression system is trained on Cohn Kanade s database [3]. It includes a series of facial expressions video sequences and each display is begun and ended with

2 a neutral face. Two peak frames of the video sequences are selected per subject to form experiments database from which the training images are random selected, and the rest images are used for testing. We use a real time face detection system based on [4]. The system consists of a cascade of classifiers trained by boosting techniques, thus has a high detection rate and a low false positive rate. In order to gain precise results, our experiments labeled face area manually for training images. The located faces are rescaled to 48*48 pixels. The images are converted into a Gabor magnitude representation, using a bank of Gabor filters at 8 orientations and 5 spatial frequencies. After that, an image is converted into 40 images with different scales and orientations and the features are the individual Gabor filters. So, there are 48*48*40=92160 possible features for a 48*48 pixels image. 2.2 AdaBoost AdaBoost which is based on selecting a series of simple classifiers to form a strong classifier is a method to boost classification. AdaBoost is an effective procedure for searching out a small number of good features which nevertheless have significant variety. After a round of training, samples are re-weighted and the weight of the error classified samples is emphasized. The different weighted weak learners are combined to form a strong classifier. 2.3 SVM We propose a new SMO-type decomposition method to training SVM, According to [7], [8], which represent libsvm method. Our system uses cross validate method in the operation, that is, random dividing the training set into five parts averagely, any four of which are used in turn for training and the left one part for testing, then we can get five different correct rates. Their average can predict the testing accuracy of the system when it is used in practice. It is more credible and extensible because of adding the unknown test samples during training. This system can also automatically gain the best parameters for SVM by the method which is called grid search. We use two kernel functions in our SVM: linear kernel function and RBF kernel function and the latter one is used more. The main parameters of SVM are penalty C and Sigma (parameter of RBF kernel). For improving speed, we choose both of them between 2 5 ~2 20.After training with cross validate method, we will gain the correct rates according to each parameter pairs. The system can automatically select the best parameter pair with the highest correct rate by the means of distributing the correct rates into two-dimensional space (see Fig. 1). We use k*k (k=3, 5, ) neighborhood mode to calculate the means of correct rates around the highest dot, and compare them to get the highest. With the cross validate method and neighborhood mode method we can believe the results is precise and credible.

3 Fig. 1. SVM parameters recognition rate curve for happy VS neutral. For happy VS neutral, we choose a large range at 2-5 ~2 40 to find a more proper range of parameters. The figure has two parts to show the results, the left part is a 3D graph where log 2 C and log 2 Sigma performs as coordinate plane, recognition rate as vertical axis. The height represents the recognition rate which is filled by different colors. On the right, there is a 2D graph using contour line to represent the recognition rate. Using the grid search method, the system will get the best parameter pair. 2.4 Combination of SVM with AdaBoost In this system, we use AdaBoost to select features instead of being a classifier. A subset of these features is selected by AdaBoost from Gabor features. At each step, the chosen feature can be shown to be uncorrelated with the output of the previous features. AdaBoost training continues until the classifier output distributions for the positive and negative samples are completely separated by a gap proportional to the widths of the two distributions. We selected 100 features for each of the binary expression classifiers in our experiment and we proved that it gives better performance than others. Because the selected features are much less than the total features, the speed is improved. We explore training SVM classifiers on the features selected by AdaBoost. The combined classifiers are informally called AdaSVM in abbreviation of Adaptive Boosting Selected Feature representations in Support Vector Machines. We examined the effect of feature selection by AdaBoost on the number of support vectors. Smaller numbers of support vectors proffer two advantages: (1) the classification procedure is faster, and (2) the expected generalization error decreases as the number of support vectors decreases. The number of support vectors for the linear SVM ranged from 10 to 33 percent of the total number of training vectors. Nonlinear SVM s employed 14 to 43 percent, despite better generalization performance. Feature selection by AdaBoost reduced the number of support vectors employed by the nonlinear SVM in the AdaSVM system, to 12 to 26 percent [2]. We compared the number of support vectors for happy VS neutral using these different methods (see table 1). The number of training samples is 180. The combination of the two methods has the advantage both of speed and accuracy. AdaSVM performs better than AdaBoost and SVM, which is shown in the part of experiments.

4 Table 1. The number of support vectors of different methods. For linear SVM, the number of support vectors is 32.8% of the total number of training vectors. Nonlinear SVM, here we used RBF kernel, employed 43.3% and the nonlinear SVM in the AdaSVM system 18.3 Number of support vectors Linear SVM Nonlinear SVM(RBF) Nonlinear SVM(RBF) in AdaSVM Experiments 3.1 Experiments for expressions recognition In order to solve multiclass facial expression recognition, we first train binary classifiers including one-versus-one (1 VS 1), one-versus-all (1 VS all). The simplest strategy is to train 1 VS all. The classifier is trained taking the examples from one kind of facial expression as positive and the examples from all other expressions as negative, i.e. happy VS nonhappy. 1 VS 1 method is pairwise partitioning that SVMs are trained to discriminate all pairs of emotions. The database for experiments has been introduced as before, that the faces are labeled manually for training and scaled into 48*48 pixels. To reduce the unbalance of the expressions, we choose identify-matched pairs as the training set in 1 VS 1 method, where for example, the happy VS surprise SVM is trained on only those subjects who gave samples of both happiness and surprise. But in practice, we broaden the restriction as the images of some expression are too few. While in 1 VS all the strategy employed a larger number of training examples which diluted identity effects. To bring the effect of identify-matched pairs more clear in 1 VS 1, we first use images random picked from Cohn Kanade s database to form training set. The results of this situation are shown in table 2. The table shows recognition rate for every pair of emotions. You can see some of the correct rates of the expressions are very low, e.g. anger VS sad only reaches 60.53%. Table 2. 1 VS 1 recognition results (Training images are not identify-matched pairs) (%) happy fear surprise sad disgust anger neutral happy fear surprise sad disgust anger neutral

5 Table 3 shows the results of recognition that uses identify-matched pairs as training set. Comparing with table 2, the recognition rates, which were low before, are improved remarkably. The recognition rate of anger VS sad reaches 90% much higher than before. So, for some facial expressions which are difficult to classify, it s very important to keep the balance of the images. Identify-matched pairs will help a lot to achieve higher recognition rate. According to the expression pairs presented here, we can get that some of them are difficult to classify directly by eyes too. That is the season why there still are several expression pairs with lower recognition rate than others in this table. With the same as people do, this system reaches different recognition rates for different facial expressions. Table 3. 1 VS 1 recogniton rates (%) happy fear surprise sad disgust anger neutral happy fear surprise sad disgust anger neutral VS all method trains 7 SVMs with one kind of expression as positive and others as negative. The size of the training set or testing set is different for different expression pairs. The results (see table 4) show that all of them could reach a satisfying correct rate except neutral VS nonneutral and fear VS nonfear. Table 4. 1 VS all recognition rates (%). Here the non is related to every expression, for example the first column is hap. VS non which means happy VS nonhappy happy VS fear VS surprise sad VS disgust anger VS neutral non non VS non non VS non non VS non In the system presented here, the SVM outputs are combined to make a 7 alternative forced choice. The binary classifiers used here are 21 SVMs that are got from 1 VS 1 experiment and they are combined by voting to get the output. The procedure counts the number of all classifiers aligned with each emotion. For example, if one SVM indicates happiness and not surprise, happiness gets +1 and surprise gets -1. These votes are summed over all of the SVMs. The expression getting the most votes is the recognized one. Here the testing set is 626 images. One experiment is to recognize the images, manually labeled face images, and the recognition rate is %. The other experiment is to recognize original images with automatically face detection and the correct rate is %. The recognition rate decreases using the automatic face detection because the face location is not as precise as that labeled manually.

6 3.2 Comparison We also trained SVM classifiers and AdaBoost classifiers directly on the output of the Gabor features. They performed no better than AdaSVM. The experiments use the same training set and testing set to compare the performances. We show 7 groups of expression pair here (see table 5). Here SVM uses RBF kernel function. Table 5. Experiments results of comparing with SVM and AdaBoost (%). Here shows 7 groups of recognition rates for using AdaSVM, AdaBoost and SVM on a bank of Gabor features AdaSVM AdaBoost SVM happy vs fear happy vs sad happy vs neutral happy vs anger happy vs disgust happy vs surprise happy vs nonhappy We also compare the time used by the three different methods for training and recognition on the expression pair happy VS neutral. Here the training set has 180 images and 116 images are in the testing set. The system presently operates on a computer of 2.0GHZ CPU and 512M memory. Table 6 shows the time used for every process. Table 6. Comparison of the experimental speed. Here shows the time of every method used in training and testing process. The Gabor column shows the time for Gabor wavelet transforming of the input images and the following columns show the time of different methods working on the Gabor features Gabor AdaSVM AdaBoost SVM (RBF) Training Testing It is shown that, according to the recognition rates, AdaBoost+Gabor method performs better than SVM+Gabor method, but AdaSVM+Gabor method performs the best. Table VII shows that SVM+Gabor method costs more time to calculate the results than any others. So, the method we explored AdaSVM+Gabor method has the obvious superiority and worth doing. 3.3 Real time expression recognition from video We combine the face detection and expression recognition into a system that operates on live digital video in real time. When the video is played, the system first detects face area automatically and then recognizes the expression and shows the result on the

7 screen. We have done video tests on our own database and it performs very well. Our database consists of 7 expressions of 6 persons (3 are male and 3 are female), and every person has a video stream for each of the expression. The video begins with neutral expression and ends with the peak of target expression. The frame is 768*576 with uniform lighting. We did experiments on real time facial expression recognition using our database. The training images are all selected from Cohn Kanade s database and our database are only used for test. Fig.2 shows the output of the expression recognizer for a test video in which the subject poses a series of facial expressions from neutral to happy. Fig. 2. Expression recognition from video. Dot A refers to the first image and dot B refers to the last image. We selected 4 images from the video to show the changes of the expression. Here the horizontal axis represents video frames and the vertical axis stands for the margin. The expression started with neutral and the curve is below the horizontal axis, when the expression changes the curve goes over the axis and finally reaches the peak when the expression reaches the target. We labeled two dots on the curve, dot A refers to the first image where the expression is neutral and dot B refers to the last image which is a happy one. During the test, only several images are recognized incorrectly which may have the relation to automatic face detection 4 Conclusions We present a system applied to the problem of fully automatic recognition of facial expressions, including Gabor wavelet transform, AdaBoost and support vector machines. We report results on a series of experiments comparing methods for multiclass decisions, spatial frequency ranges, feature selection methods, and recognition engines. Best results are obtained by selecting a subset of Gabor features using AdaBoost and then training Support Vector Machines on the outputs of the features selected by AdaBoost. The combination of AdaBoost and SVMs enhances both speed and accuracy of the system. Its performance in Cohn Kanade s database is averagely 93.4% for any expression pair recognition and 88.6% for 7-way forced choice.

8 Our results suggest that user independent fully automatic real time coding of basic expressions is an achievable goal with present computer power, at least for applications in which frontal views can be assumed. The following problem we will study on is how to reach high accuracy for 7 basic expressions recognition and how to solve the problem by a simple system, after the images are preprocessed by a bank of Gabor filters. Acknowledgment The Cohn Kanade database of facial images is used in portions of the research in this paper. So the authors would like to thank the Cohn Kanade Technical Agent. We also sincerely thank suggestions of Senior Research Scientist Wataru Ito and Dr. Yuanzhong Li at Imaging Software Technology Center, Research & Development Management Headquarters of FUJI PHOTO FILM CO., LTD. References 1. Lyons, M., Budynek, J., Plante, A., Akamatsu, S.: Classifying Facial Attributes using a 2-d Gabor Wavelet Representation and Discriminant Analysis, Proceedings of the 4th International conference on Automatic Face and Gesture Recognition (2000), Littlewort, G., Bartlett, M. S., Fasel, I., Chenu, J., Movellan, J. R.: Analysis of Machine Learning Methods for Real-Time Recognition of Facial Expressions from Video, MPLab TR (2003) 3. Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis, Proceedings of the 4th IEEE International Conference on Automatic Face and Gesture Recognition (2000), Viola, P., Jones, M.: Robust Real-time Object Detection, Technical Report CRL 2001/01, Cambridge Research-Laboratory (2001) 5. Osuna, E., Freund, R.: An Improved Training Algorithm for Support Vector Machine, Proceedings of 1997 IEEE Workshop on Neural Networks for Signal Processing (1997), Platt, J.: Fast Training of Support Vector Machines using Sequential Minimal Optimization, MIT Press (1998) 7. Fan, R. E., Chen, P. H., Lin, C.J.: Working Set Selection using the Second Order Information for Training SVM, (2005) 8. Chen, P. H., Fan, R. E., Lin, C.J.: A Study on SMO-type Decomposition Methods for Support Vector Machines, Technical report, Department of Computer Science, National Taiwan University (2005),