Document Type : Research Articles
Authors
Faculty of Electrical and Computer Engineering, University of Tabriz, Tabriz, Iran
Abstract
Objective: The Micro-Expression (ME), which automatically reveals genuine human emotions, has gained significant attention. Recognizing the ME is crucial for many real-time applications. However, there are significant challenges to overcome. For instance, the number of ME frames are limited due to their short duration, and the subtle facial movements can be hard to detect due to their low intensity. These challenges need to be addressed to improve ME recognition. Materials and Methods: We propose a novel method for the ME recognition in real-time. In this method, first, the apex frame is spotted using the rotated local binary pattern from six planes (RLBPS) and correlation coefficient (CC). Next, three hand-crafted methods such as the multi-color rotated local binary pattern from six planes (MRLBPS), the histograms of directed gradients from six planes (HDGS), and the histogram of image gradient direction from six planes (HIGDS) extract the features from the apex frame and its surrounding frames. Finally, the stacks of features as matrixes are fed into a three-dimensional convolutional neural network (3D-CNN), and the output is the maximum recognition rate by voting three results. Results: The proposed method has shown promising results when compared to most state-of-the-art methods. According to the results, an average precision of 99% has been obtained using our proposed method. Conclusion: The combination of the RLBPS and the CC creates a strong method for spotting the apex frame. Also, feeding the stacks of spatiotemporal features into the 3D-ResNet increases the ME recognition rate in real-time.
Keywords
- Micro-Expression Recognition
- Spatiotemporal hand-crafted methods
- Spatiotemporal deep-learning method
Main Subjects
[1] V. Esmaeili, M. Mohassel Feghhi, and S. O. Shahdi, "A comprehensive survey on facial micro-expression: approaches and databases," Multimedia Tools and Applications, Vol. 81, No. 28, pp. 40089-40134, 2022.
[2] X. Li et al., "Towards reading hidden emotions: A comparative study of spontaneous micro-expression spotting and recognition methods," IEEE transactions on affective computing, Vol. 9, No. 4, pp. 563-577, 2017.
[3] G. Zhou et al., "Micro-expression action unit recognition based on dynamic image and spatial pyramid," The Journal of Supercomputing, pp. 1-24, 2023.
[4] Y. Li, J. Wei, Y. Liu, J. Kauttonen, and G. Zhao, "Deep learning for micro-expression recognition: A survey," IEEE Transactions on Affective Computing, 2022.
[5] G. Zhao, X. Li, Y. Li, and M. Pietikäinen, "Facial Micro-Expressions: An Overview," Proceedings of the IEEE, 2023.
[6] P. Ekman and W. V. Friesen, "Nonverbal leakage and clues to deception," Psychiatry, Vol. 32, No. 1, pp. 88-106, 1969.
[7] P. Ekman, Micro expressions training tool Emotionsrevealed. com, 2003.
[8] V. Esmaeili, M. Mohassel Feghhi, and S. O. Shahdi, "Automatic Micro-Expression Recognition Using LBP-SIPl and FR-CNN," AUT Journal of Modeling and Simulation, Vol. 54, No. 1, pp. 59-72, 2022.
[9] V. Esmaeili, M. Mohassel Feghhi, and S. O. Shahdi, "Autonomous apex detection and Micro-expression recognition using proposed diagonal Planes," International Journal of Nonlinear Analysis and Applications, Vol. 11, pp. 483-497, 2020.
[10] V. Esmaeili, M. M. Feghhi, and S. O. Shahdi, "Micro-expression recognition using histogram of image gradient orientation on diagonal planes," in 2021 5th International Conference on Pattern Recognition and Image Analysis (IPRIA), 2021: IEEE, pp. 1-5.
[11] V. Esmaeili and S. O. Shahdi, "Automatic micro-expression apex spotting using Cubic-LBP," Multimedia Tools and Applications, Vol. 79, pp. 20221-20239, 2020.
[12] V. Esmaeili, M. M. Feghhi, and S. O. Shahdi, "Automatic micro-expression apex frame spotting using local binary pattern from six intersection planes," arXiv preprint arXiv:2104.02149, 2021.
[13] V. Esmaeili, M. Mohassel Feghhi, and S. O. Shahdi, "Micro-Expression Recognition based on the Multi-Color ULBP and Histogram of Gradient Direction from Six Intersection Planes," Journal of Iranian Association of Electrical and Electronics Engineers, Vol. 19, No. 3, pp. 123-130, 2022.
[14] V. Esmaeili, M. M. Feghhi, and S. O. Shahdi, "Early COVID-19 Diagnosis from Lung Ultrasound Images Combining RIULBP-TP and 3D-DenseNet," in 2022 9th Iranian Joint Congress on Fuzzy and Intelligent Systems (CFIS), 2022: IEEE, pp. 1-5.
[15] V. Esmaeili and M. Mohassel Feghhi, "Real-time Authentication for Electronic Service Applicants using a Method Based on Two-Stream 3D Deep Learning," Soft Computing Journal, 2023.
[16] V. Esmaeili and M. Mohassel Feghhi, "Diagnosis of Covid-19 Disease by Combining Hand-crafted and Deep-learning Methods on Ultrasound Data," Journal of Machine Vision and Image Processing, Vol. 9, No. 4, pp. 31-41, 2022.
[17] V. Esmaeili, M. Mohassel Feghhi, and S. O. Shahdi, "Spotting micro‐movements in image sequence by introducing intelligent cubic‐LBP," IET Image Processing, Vol. 16, No. 14, pp. 3814-3830, 2022.
[18] V. Esmaeili and M. M. Feghhi, "COVID-19 Diagnosis:ULBPFP-Net Approach on Lung Ultrasound Data," Iranian Journal of Electrical & Electronic Engineering, Vol. 19, No. 3, 2023.
[19] V. Esmaeili, M. Mohassel Feghhi, and S. Shahdi, "Applying Partial Differential Equations on Cubic Uniform Local Binary Pattern to Reveal Micro-Changes," Journal of Electrical and Computer Engineering Innovations (JECEI), pp. 259-270, 2023.
[20] A. Ebrahimi, S. Luo, and R. Chiong, "Introducing transfer learning to 3D ResNet-18 for Alzheimer’s disease detection on MRI images," in 2020 35th international conference on image and vision computing New Zealand (IVCNZ), 2020: IEEE, pp. 1-6.
[21] Z. Raisi and J. Zelek, "Investigation of Deep Learning Optimization Algorithms in Scene Text Detection," International Journal of Industrial Electronics Control and Optimization, Vol. 6, No. 3, pp. 171-182, 2023. and Optimization, Vol. 6, No. 3, pp. 171-182, 2023.
[22] R. Mehta and K. Egiazarian, "Rotated local binary pattern (RLBP): rotation invariant texture descriptor," in 2nd International Conference on Pattern Recognition Applications and Methods, ICPRAM 2013, Barcelona, Spain, 15.-18.2. 2013, 2013, pp. 497-502.
[23] T. Ojala, M. Pietikäinen, and T. Mäenpää, "A generalized local binary pattern operator for multiresolution gray scale and rotation invariant texture classification," in Advances in Pattern Recognition—ICAPR 2001: Second International Conference Rio de Janeiro, Brazil, March 11–14, 2001 Proceedings 2, 2001: Springer, pp. 399-408.
[24] T. Pfister, X. Li, G. Zhao, and M. Pietikäinen, "Recognising spontaneous facial micro-expressions," in 2011 international conference on computer vision, 2011: IEEE, pp. 1449-1456.
[25] H.-Y. Wu, M. Rubinstein, E. Shih, J. Guttag, F. Durand,and W. Freeman, "Eulerian video magnification for revealing subtle changes in the world," ACM transactions on graphics (TOG), Vol. 31, No. 4, pp. 1-8, 2012.
[26] A. Asthana, S. Zafeiriou, S. Cheng, and M. Pantic, "Robust discriminative response map fitting with constrained local models," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2013, pp. 3444-3451.
[27] W.-J. Yan, Q. Wu, Y.-J. Liu, S.-J. Wang, and X. Fu, "CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces," in 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG), 2013: IEEE, pp. 1-7.
[28] W.-J. Yan et al., "CASME II: An improved spontaneous micro-expression database and the baseline evaluation," PloS one, Vol. 9, No. 1, p. e86041, 2014.
[29] J. Li et al., "CAS (ME) 3: A third generation facial spontaneous micro-expression database with depth information and high ecological validity," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 45, No. 3, pp. 2782-2800, 2022.
[30] J. Tang, L. Li, M. Tang, and J. Xie, "A novel micro-expression recognition algorithm using dual-stream combining optical flow and dynamic image convolutional neural networks," Signal, Image and Video Processing Vol. 17, No. 3, pp. 769-776, 2023.
[31] X.-B. Nguyen, C. N. Duong, X. Li, S. Gauch, H.-S. Seo, and K. Luu, "Micron-bert: Bert-based facial micro-expression recognition," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1482-1492.
[32] H. Yang, S. Sun, and J. Chen, "Deep Learning-Based Micro-Expression Recognition Algorithm Research," International Journal of Computer Science and Information Technology, Vol. 2, No. 1, pp. 59-70, 2024.
[33] C. Nicholson, "Evaluation metrics for machine learning accuracy, precision, recall, and F1 defined,"Pathmind.http://pathmind.com/wiki/accuracy-precision-recall-f1, 2019.
[34] S.-T. Liong, J. See, K. Wong, and R. C.-W. Phan, "Less is more: Micro-expression recognition from video using apex frame," Signal Processing: Image Communication, Vol. 62, pp. 82-92, 2018.
[35] Y. S. Gan, S.-T. Liong, W.-C. Yau, Y.-C. Huang, and L.-K. Tan, "OFF-ApexNet on micro-expression recognition system," Signal Processing: I mage Communication, Vol. 74, pp. 129-139, 2019.
[36] S.-T. Liong, Y. S. Gan, J. See, H.-Q. Khor, and Y.-C. Huang, "Shallow triple stream three-dimensional cnn (ststnet) for micro-expression recognition," in 2019 14th IEEE international conference on automatic face & gesture recognition (FG 2019), 2019: IEEE, pp. 1-5.
[37] Y. Gan, S.-E. Lien, Y.-C. Chiang, and S.-T. Liong, "LAENet for micro-expression recognition," The Visual Computer, Vol. 40, No. 2, pp. 585-599, 2024.
[38] Z. Wang, K. Zhang, W. Luo, and R. Sankaranarayana, "HTNet for micro-expression recognition," arXiv preprint arXiv:2307.14637, 2023.
[39] Y. Su, J. Zhang, J. Liu, and G. Zhai, "Key facial components guided micro-expression recognition based on first & second-order motion," in 2021 IEEE International Conference on Multimedia and Expo(ICME), 2021: IEEE, pp. 1-6.
[40] M. Wei, W. Zheng, Y. Zong, X. Jiang, C. Lu, and J. Liu, "A novel micro-expression recognition approach using attention-based magnification-adaptive networks," in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022: IEEE, pp. 2420-2424.
[41] H. Jin, N. He, Z. Li, and P. Yang, "Micro-expression recognition based on multi-scale 3D residual convolutional neural network," Mathematical Biosciences and Engineering, Vol. 21, No. 4, pp. 5007-5031, 2024.
[42] B. Song et al., "Recognizing spontaneous micro-expression using a three-stream convolutional neural network," Ieee Access, vol. 7, pp. 184537-184551, 2019.
[43] B. Sun, S. Cao, D. Li, J. He, and L. Yu, "Dynamic micro-expression recognition using knowledge distillation," IEEE Transactions on Affective Computing, Vol. 13, No. 2, pp. 1037-1043, 2020.
[44] Y. Wang et al., "Micro expression recognition via dual-stream spatiotemporal attention network," Journal of Healthcare Engineering, Vol. 2021, No. 1, 2021.
[45] G. Zhu et al., "SKD-TSTSAN: Three-Stream Temporal-Shift Attention Network Based on Self-Knowledge Distillation for Micro-Expression Recognition," arXiv preprint arXiv:2406.17538, 2024.
)