Type: Bachelor Master UROP
Status: Assigned February 2024
Student: Eleonora Bardelli

Affective computing is an interdisciplinary field that aims at the development of computer science techniques that enable machines to recognize, understand and simulate human affective states [1, 2]. A fundamental assumption is that different mental states (e.g., emotions and stress), and different intensities of those states, manifest through physiological and behavioral changes. A variety of sensing modalities can capture these changes [3, 4].

Video-based sensing is one promising approach for affect recognition, however, it is also privacy intrusive. Thus, this project will develop a method for privacy-aware personal-video sensing for affect recognition. The method will utilize personal camera (e.g., smartphone or laptop camera) and will include privacy-aware features such as: to record only when the users grant permission, to record only when a specific user is in front of the camera (user identification). Once the video is collected in a privacy-aware manner, existing video-based affect recognition software will be sued to extract affect related information.

Project tasks:
1. Review existing software (e.g., GitHub and Google scholar) for user identification, software for counting faces in a video, and software for Facial Action Coding System (FACS) [5].
2. Implement privacy-aware user identification;
3. Implement privacy-aware method for extracting Facial Action Units (based on FACS);
4. Test the overall processing pipeline is a small user-study (e.g., 5 to 10 participants).

This project is available as a MSc thesis (all four tasks), as a BSc thesis (only task two and task three), or as a UROP internship project (all four tasks).

LiteratureL

[1] R. Picard. “Affective Computing,” Cambridge, MA: MIT Press., 1997.
[2] J. Tao, J., and T. Tan, “Affective computing: A review. In International Conference on Affective computing and intelligent interaction,” Springer, Berlin, Heidelberg, 2005.
[3] Y. Wang, W. Song, W. Tao, A. Liotta, D. Yang, X. Li, S. Gao, Y. Sun, W. Ge, W. Zhang, and W. Zhang, “A systematic review on affective computing: Emotion models, databases, and recent advances.” Information Fusion, 2022.
[4] R. Arya, S. Jaiteg Singh, and A. Kumar. “A survey of multidisciplinary domains contributing to affective computing.” Computer Science Review 40 (2021): 100399.
[5] https://en.wikipedia.org/wiki/Facial_Action_Coding_System

For more information contact: Martin Gjoreski