Abstract

Facial expression analysis is an important tool for human-computer interaction. In this paper, we introduce LibreFace, an open-source toolkit for facial expression analysis. This open-source toolbox offers real-time and offline analysis of facial behavior through deep learning models, including facial action unit (AU) detection, AU intensity estimation, and facial expression recognition. To accomplish this, we employ several techniques, including the utilization of a large-scale pre-trained network, feature-wise knowledge distillation, and task-specific fine-tuning. These approaches are designed to effectively and accurately analyze facial expressions by leveraging visual information, thereby facilitating the implementation of real-time interactive applications. In terms of Action Unit (AU) intensity estimation, we achieve a Pearson Correlation Coefficient (PCC) of 0.63 on DISFA, which is 7% higher than the performance of OpenFace 2.0, while maintaining highly-efficient inference that runs two times faster than OpenFace 2.0. Despite being compact, our model also demonstrates competitive performance to state-of-the-art facial expression analysis methods on AffecNet, FFHQ, and RAF-DB. Our code will be released soon at LibreFace.

Proposed Method

We first pre-process the input images or video, which involves face mesh and landmark detection and image alignment. Next, we feed the pre-processed images into a pre-trained MAE encoder, followed by a linear regression or classification layer that predicts the AU intensity values or facial expression labels. Once the MAE is fine-tuned, we employ feature-wise distillation to transfer the teacher model's (MAE) knowledge to a lightweight student model (ResNet-18). Finally, we use the distilled ResNet-18 for efficient Action Unit Intensity Estimation, Action Unit Detection and Facial Expression Recognition. All outputs from the pipeline can be saved, and training/inference code is open-sourced.

Contributions

We present LibreFace, an open-source and comprehensive toolkit for accurate and real-time facial expression analysis with both CPU-only and GPU-acceleration versions. LibreFace eliminates the gap between cutting-edge research and an easy and free-to-use non-commercial toolbox. We propose to adaptively pre-train the vision encoders with various face datasets and then distillate them to a lightweight ResNet-18 model in a feature-wise matching manner. We conduct extensive experiments of pre-training and distillation to demonstrate that our proposed pipeline achieves comparable results to state-of-the-art works while maintaining real-time efficiency. LibreFace system supports cross-platform running, and the code is open-sourced in C\# (model inference and checkpoints) and Python (model training, inference, and checkpoints).

Software Demo

Citation

Acknowledgements

This work is sponsored by the U.S. Army Research Laboratory (ARL) under contract number W911NF-14-D-0005. The content of the information does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.