Student Project |

A Novel Approach Using Bilateral Data Fusion for EDA Data Classification

Mon, 01 Jan 0001 00:00:00 +0000

Type: Bachelor Master UROP Status: Available

This thesis addresses the impact of lateralization on Electrodermal Activity (EDA) sensors in wearable devices. Lateralization, influenced by brain hemisphere activation, affects the accuracy of EDA readings based on the device’s placement on a specific body side. Despite recent studies highlighting this issue, there is limited exploration of the potential benefits of using EDA devices on both sides simultaneously. The research aims to fill this gap by investigating how leveraging data from both sides concurrently can enhance classifier accuracy through machine learning. The focus is on datasets in the lab, with implications for medical-grade applications affected by lateralization. Success in demonstrating improved accuracy may revolutionize the field, particularly in sensitive medical tasks, offering more reliable predictions for tasks impacted by lateralization.

For more information contact: ,

Embodied Large Language Models for Personalized MeetingSummarization

Mon, 01 Jan 0001 00:00:00 +0000

Type: Bachelor Master UROP Status: Available

The rapid advancement of pervasive systems and wearables has paved the way for personal informatics systems, e.g., personal assistants and chatbots. Such systems gear towards enhancing users’ productivity in both work setting and personal life. In work settings, users leverage personal informatics systems in managing their tasks, tracking their progress and scheduling their meetings.

Meetings are crucial for communication, decision-making, and brainstorming. The effectiveness of meetings relies on participants’ ability to remember key topics, often using meeting minutes or notes as memory cues. However, this approach is time-consuming, demands high attention levels, and may lead to varied perceptions among participants, causing a lack of synchronization.

In this project, we aim at developing a personalized meeting summarization tool that aims at optimizing the meeting experience. The proposed tool integrates users’ affection from wearables with large language models, e.g., ChatGPT.

This project is available as a MSc thesis, as a BSc thesis, or as a UROP internship project.

For more information contact: ,

Empathetic Virtual Agents using Large Language Models

Mon, 01 Jan 0001 00:00:00 +0000

Type: Bachelor Master UROP Status: Available

The rapid advancement of pervasive systems and wearables has paved the way for personal informatics systems, e.g., personal assistants and chatbots. These systems can be deployed in different personal and professional settings. Such personal assistants aim at personalizing the user ‘s experience by taking into consideration the user’s affection status and respond based on the user’s emotions and mental state status.

In this project, we aim at developing an empathetic personal assistant tool that provides an affection-aware user experience. The proposed tool integrates users’ affection from wearables with large language models, e.g., ChatGPT.

This project is available as a MSc thesis, as a BSc thesis, or as a UROP internship project.

For more information contact: ,

Explainable AI for Federated Models in Wearable Sensing

Mon, 01 Jan 0001 00:00:00 +0000

Type: Master UROP Status: Available

Federated learning and its combination with differential privacy is the latest technique for building privacy-aware machine-learning models [1]. Its primary assumption – no data leaves the local data storage, has enabled its application in a variety of privacy-sensitive domains: mobile keyboard prediction [2], human mobility modeling based on GPS data [3], modeling from electronic health records [4] , etc.

Artificial Intelligence (AI) methods can bring significant and sustainable improvements to our lives. However, end-users must be able to understand those systems. Unfortunately, today’s groundbreaking AI methods are black-boxed (i.e., the decision model and the process are not understandable). The increased complexity of AI algorithms has made previous eXplainable AI (XAI) tools unsuitable, including the fact that most of the XAI solutions are not designed to operate under privacy constraints.

This project will investigate XAI techniques compatible with privacy-aware approaches (e.g., federated learning). The focus will be on counterfactual explainers [55] for wearable sensing data. Specific project tasks are:

Analyze XAI tools that can operate under privacy constraints, focusing on counterfactuals.
Pre-process one dataset from wearable sensing systems. Example datasets include emotion recognition, activity recognition and energy expenditure estimation [6, 7, 8].
Develop machine learning models for one of the datasets in step 2, and apply existing XAI tools on the models developed, including the method for generating counterfactual explanations, BayCon [10, 11].
Develop XAI tool for counterfactual explanations that can operate under privacy constraints.

This project is available as a MSc thesis and as a UROP internship project.

Literature:

[1] J. Konečný, H. B. McMahan, F. X. Yu, P. Richtárik, A. Theertha Suresh, and D. Bacon. “Federated learning: Strategies for improving communication efficiency.” arXiv preprint arXiv:1610.05492 (2016). [2] Andrew Hard, Kanishka Rao, Rajiv Mathews, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. CoRR, abs/1811.03604, 2018. URL 1811.03604. [3] Ezequiel, C. E. J., Gjoreski, M., & Langheinrich, M. (2022). Federated Learning for Privacy-Aware Human Mobility Modeling. Frontiers in Artificial Intelligence, 5, 867046. [4] Brisimi, T. S., Chen, R., Mela, T., Olshevsky, A., Paschalidis, I. C., & Shi, W. (2018). Federated learning of predictive models from federated electronic health records. International journal of medical informatics, 112, 59-67. [5] Jiang, J. C., Kantarci, B., Oktug, S., & Soyata, T. (2020). Federated learning in smart city sensing: Challenges and opportunities. Sensors, 20(21), 6230. [6] T. Miller, “Explanation in artificial intelligence: Insights from the social sciences,” Artificial intelligence 267: 1-38, 2019. [7] M. Laporte, D. Gasparini, M. Gjoreski, and M. Langheinrich. “Exploring LAUREATE-the Longitudinal multimodAl stUdent expeRience datasEt for AffecT and mEmory research.” in the UbiComp/ISWC’22 Adjunct Proceedings, 2022. [8] Gashi, S., Min, C., Montanari, A., Santini, S., & Kawsar, F. (2022). A multidevice and multimodal dataset for human energy expenditure estimation using wearable devices. Scientific Data, 9(1), 1-14. [9] Gjoreski, M., Kiprijanovska, I., Stankoski, S., Mavridou, I., Broulidakis, M. J., Gjoreski, H., & Nduka, C. (2022). Facial EMG sensing for monitoring affect using a wearable device. Scientific reports. [10] M. Gjoreski M, V. Kuzmanovski V, and M. Bohanec “BAG-DSM: A Method for Generating Alternatives for Hierarchical Multi-Attribute Decision Models Using Bayesian Optimization,” Algorithms, 2022. [11] P. Romashov, M. Gjoreski, K. Sokol, M. V. Martinez, and M. Langheinrich. “BayCon: Model-agnostic Bayesian Counterfactual Generator,”. In IJCAI 2022.

For more information contact:

Fed-CogLoad: Federated Cognitive Load Estimation

Mon, 01 Jan 0001 00:00:00 +0000

Type: Bachelor Status: Assigned March 2023 Student: Mattias Formo

Federated learning (FL) is a state-of-the-art machine-learning technique developed by Google, where the users’ privacy is guaranteed by implementing one simple rule: “No personal data leaves the user-device”. This project will investigate FL techniques for cognitive load estimate. Cognitive load can be estimated through the analysis of from pupillometry data, brain activation data (EEG), breathing rate, heart rate, heart rate variability and other related physiological responses.

Specifically, this thesis has four main tasks:

overview excising datasets for cognitive load estimation;

develop a centralized machine learning pipeline for cognitive load estimation;

develop a FL pipeline for cognitive load estimation;

The main challenge of the work will be to develop enhanced privacy-aware models for cognitive-load modeling using federated learning.

For more information contact:

Multimodal Federated Learning for Sensor Data

Mon, 01 Jan 0001 00:00:00 +0000

Type: Bachelor Status: Assigned February 2023 Student: Alessandro Gobbetti

This project will investigate single modality vs. multi-modality federated models. This is an important issue for wearable sensing systems that utilize multiple sensing devices, e.g., smartphone and smartwatch. Each device, and each sensor in the devices, may have a different availability –– data coming from the smartwatch may be unavailable at certain periods (e.g., while charging). To enable the collaborative learning of joint models between users with a variable data/modality availability, we will investigate several multi-modal schemes.

Project tasks:

Pre-process one dataset from wearable sensing systems. Example datasets include emotion recognition, activity recognition and energy expenditure estimation [6, 7, 8].
Build centralized multimodal and single-modal models using the dataset from step 1.
Build federated multimodal and single-modal models using the dataset from step 1 and compare them with the centralized models from 2.
Develop novel multi-modal federated learning method considering device/sensor availability, computational cost, model accuracy.

This project is available as a MSc thesis (all four tasks), as a BSs thesis (only task 2 and task 3), or as a UROP internship project (all 4 tasks).

Literature:

[1] Konečný, Jakub, H. Brendan McMahan, Felix X. Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. “Federated learning: Strategies for improving communication efficiency.” arXiv preprint arXiv:1610.05492 (2016). [2] Andrew Hard, Kanishka Rao, Rajiv Mathews, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. CoRR, abs/1811.03604, 2018. URL 1811.03604. [3] Ezequiel, C. E. J., Gjoreski, M., & Langheinrich, M. (2022). Federated Learning for Privacy-Aware Human Mobility Modeling. Frontiers in Artificial Intelligence, 5, 867046. [4] Brisimi, T. S., Chen, R., Mela, T., Olshevsky, A., Paschalidis, I. C., & Shi, W. (2018). Federated learning of predictive models from federated electronic health records. International journal of medical informatics, 112, 59-67. [5] Jiang, J. C., Kantarci, B., Oktug, S., & Soyata, T. (2020). Federated learning in smart city sensing: Challenges and opportunities. Sensors, 20(21), 6230. [6] M. Laporte, D. Gasparini, M. Gjoreski, and M. Langheinrich. “Exploring LAUREATE-the Longitudinal multimodAl stUdent expeRience datasEt for AffecT and mEmory research.” in the UbiComp/ISWC’22 Adjunct Proceedings of the 2022 ACM International Joint Conference on Pervasive and Ubiquitous Computing, 2022. [7] Gashi, S., Min, C., Montanari, A., Santini, S., & Kawsar, F. (2022). A multidevice and multimodal dataset for human energy expenditure estimation using wearable devices. Scientific Data, 9(1), 1-14. [8] Gjoreski, M., Kiprijanovska, I., Stankoski, S., Mavridou, I., Broulidakis, M. J., Gjoreski, H., & Nduka, C. (2022). Facial EMG sensing for monitoring affect using a wearable device. Scientific reports, 12(1), 1-12.

For more information contact:

Protect privacy in wearable devices using data anonymization

Mon, 01 Jan 0001 00:00:00 +0000

Type: Master Status: Available

Wearables facilitate continuous data collection to monitor diverse human behaviors, covering activity, health, and stress. This personal data, including electrocardiogram, movements, and heart rate, is often accessible online for research. Despite the common practice of masking names with random identifiers, studies show that this is insufficient for user identity protection due to the subject-dependent nature of physiological data. This research utilizes existing data and models to implement and assess anonymization techniques such as noise addition and synthetic data generation. The objective is to determine the extent of effective user identity protection while minimizing disruption to human behavior prediction.

This project is available as a MSc thesis or as a BSc thesis (with reduced tasks).

For more information contact: ,

Self-supervised Domain Adaptation for Sensor Data

Mon, 01 Jan 0001 00:00:00 +0000

Type: Master UROP Status: Available

Wearable devices, combined with Artificial Intelligence (AI) methods, can bring significant and sustainable improvements to our lives – from improved patient monitoring and decreased healthcare costs to enhanced sports performance and improved quality of life. Standard approaches involve Machine Learning (ML) techniques applied to the data captured from body-worn sensing devices. The ML techniques can be based on classical (feature-based) ML, or Deep Learning (DL) applied on the raw sensor data (end-to-end learning). A typical weakness that all ML-based HAR systems have, regardless of whether they are classical or DL-based, is the domain shift that can be caused by different sensor placements [1, 2].

This project will explore personalization and domain-adaptation techniques to address important challenges in wearable computing: noisy data, limited data, and domain shifts in the labels and the sensor data due to subjectivity. ML processing pipelines (including deep learning techniques) will be augmented with the latest unsupervised and self-supervised learning techniques, including contrastive learning [3]. These advanced techniques should produce more robust and data-efficient models (i.e., requiring fewer person-specific labels). Diffusion-based approaches [4, 5] could also be considered.

Project tasks:

Overview of existing self-supervised learning approaches [6, 7, 8, 9, 10].
Pre-process one dataset from wearable sensing systems. Example datasets include emotion recognition, activity recognition and energy expenditure estimation [11, 12, 13]
Build baseline ML models using the dataset from step 2.
Develop self-supervised ML approach and compare self-supervised models with the baseline ML models from step 3.

This project is available as a MSc thesis and as a UROP internship project.

Literature:

[1] Gjoreski, M.; Gjoreski, H.; Luštrek, M.; Gams, M. How Accurately Can Your Wrist Device Recognize Daily Activities and Detect Falls? Sensors 2016, 16, 800. . [2] Kalabakov, S.; Stankoski, S.; Kiprijanovska, I.; Andova, A.; Rešˇciˇc, N.; Janko, V.; Gjoreski, M.; Gams, M.; Luštrek, M. What Actually Works for Activity Recognition in Scenarios with Significant Domain Shift: Lessons Learned from the 2019 and 2020 Sussex-Huawei Challenges. Sensors 2022, 22, 3613. . [3] Haresamudram, H., Essa, I., & Plötz, T. (2021). Contrastive predictive coding for human activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 5(2), 1-26. [4] K. Rasul, C. Seward, I. Schuster, and R. Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning, 2021. [5] L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, Y. Shao, W. Zhang, B. Cui, and M.-H. Yang. “Diffusion models: A comprehensive survey of methods and applications.” arXiv preprint arXiv:2209.00796 (2022). [6] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch et al. “Bootstrap your own latent-a new approach to self-supervised learning.” Advances in neural information processing systems 33 (2020): 21271-21284. [7] D. Wei, J. J. Lim, A. Zisserman, and W. T. Freeman. “Learning and using the arrow of time.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8052-8060. 2018. [8] C. Tong, J. Ge, and N.D. Lane. “Zero-Shot Learning for IMU-Based Activity Recognition Using Video Embeddings.” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, no. 4 (2021): 1-23. [9] H. Kwon, C. Tong, H, Haresamudram, Y. Gao, G. D. Abowd, N.D. Lane, and T. Ploetz. “IMUTube: Automatic extraction of virtual on-body accelerometry from video for human activity recognition.” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, no. 3, 2020. [10] Y. Jain, C. I. Tang, C. Min, F. Kawsar, and A. Mathur. “ColloSSL: Collaborative Self-Supervised Learning for Human Activity Recognition.” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, no. 1 (2022): 1-28. [11] M. Laporte, D. Gasparini, M. Gjoreski, and M. Langheinrich. “Exploring LAUREATE-the Longitudinal multimodAl stUdent expeRience datasEt for AffecT and mEmory research.” in the UbiComp/ISWC’22 Adjunct Proceedings of the 2022 ACM International Joint Conference on Pervasive and Ubiquitous Computing, 2022. [12] Gashi, S., Min, C., Montanari, A., Santini, S., & Kawsar, F. (2022). A multidevice and multimodal dataset for human energy expenditure estimation using wearable devices. Scientific Data, 9(1), 1-14. [13] Gjoreski, M., Kiprijanovska, I., Stankoski, S., Mavridou, I., Broulidakis, M. J., Gjoreski, H., & Nduka, C. (2022). Facial EMG sensing for monitoring affect using a wearable device. Scientific reports, 12(1), 1-12.

For more information contact:

Uncertainty-aware Deep Learning in digital healthcare

Mon, 01 Jan 0001 00:00:00 +0000

Type: Master UROP Status: Available

In this thesis, the objective is to investigate the use of Monte Carlo Dropout, a Bayesian Deep Learning technique, to make uncertainty prediction on Neural Network outputs when dealing with physiological data. In particular, the student will investigate the creation of a system using Early Exit to adjust the computational power based on the uncertainty prediction. The student will leverage state-of-the-art Deep Learning models to make predictions on publicly available datasets for health applications, e.g., ECG data from healthy and unhealthy individuals.

This project is available as a MSc thesis or as a EUROP project.

For more information contact: ,

Student Project |

A Novel Approach Using Bilateral Data Fusion for EDA Data Classification

Embodied Large Language Models for Personalized Meeting Summarization

Empathetic Virtual Agents using Large Language Models

Explainable AI for Federated Models in Wearable Sensing

Literature:

Fed-CogLoad: Federated Cognitive Load Estimation

Multimodal Federated Learning for Sensor Data

Literature:

Protect privacy in wearable devices using data anonymization

Self-supervised Domain Adaptation for Sensor Data

Literature:

Uncertainty-aware Deep Learning in digital healthcare

Embodied Large Language Models for Personalized MeetingSummarization