Development of Adaptive Mask for Regions-based Facial Micro-Expressions Recognition Using Objective Classes

Abstract

Facial micro-expressions are very brief spontaneous expressions that appear on the face of humans when a person either deliberately or unconsciously conceals a feeling or emotion. Unlike regular facial expressions, it is difficult to fake a micro-expression due to its subtlety and very short duration. Facial micro-expression recognition is still a challenging area with some gaps and limitations which need to fill such as the low accuracy achieved so far for the recognition, there is no investigation on the effect of frame rate and resolution changes for facial micro-expression recognition on the existing datasets, the emotion classes within the datasets are based on Action Units (AUs) and selfreports, which creating conflicts when training the machine learning method. This research focuses on exploring the best features of descriptors and representation of facial micro-expressions for recognition. Objective classes labeling based on AUs has been introduced. In addition, the effect of resolution and frame rate for facial micro-expressions recognition have been experimented and identified. To provide new insights into the roles of temporal and spatial settings, an investigation has been conducted into the use of different frame rates and resolutions on current benchmark datasets (SMIC and CASME II). By using Temporal Interpolation Model, SMIC has been sub-sampled (original frame rate is 100 fps) to 50 fps, and CASME II (original frame rate is 200 fps) into 100 fps and 50 fps. In addition, the resolution settings have been adjusted to three scaling factors: (100% original resolution), 75%, and 50%. Three feature types have been used to test the performance of these settings. Emotion classes within the current dataset are based on self-reports. Instead of that, restructuring for the classes has been done around the AUs to removes the potential bias of human reporting. A list of AUs and combinations are proposed for a fair categorization of the SAMM and CASME II datasets. Categorizing in this way make datasets classes more unified. Finally, a new method for facial micro-expression recognition has been proposed. The proposed method is a region-based with an adaptive mask for facial micro-expression recognition. Based on the most frequent Action Units on the two publicly available datasets, i.e. CASME II and SAMM, 14 ROIs are defined. The adaptive mask has been created by calculating the oriented magnitude of optical flow after Gaussian smoothing. also the problem of light condition which considers as micromovement has been solved using a proposed method called remove random displacements which remove the random pixels caused by brightness changes or head-movements. features have been extracted from each region using Local Binary Patterns on Three orthogonal Planes (LBP-TOP). The proposed method evaluated on two benchmark datasets: CASME II and SAMM.It performing better than state-of-the-art, achieved results up to 69.6 and 0.59 in terms of accuracy and F1-score respectively on CASMEII, and 59.7 and 0.51 on SAMM. The proposed method has tested using objective classes and achieved a higher result reach to 77.9 accuracy and 0.72 F1-score on CASME II.