Selected Research Projects

Multimodal Sentiment Analysis Using Multiple Labels from Different Modalities

With the extensive amount of social media data in different modalities, particularly text and images, unimodal sentiment analysis that uses only text modality is not sufficient to determine sentiment polarity. Although multiple modalities are unitedly considered in multimodal sentiment analysis by combining features from such modalities, a unified multimodal label is merely adopted. Moreover, a unimodal label for each modality has not been fully explored in the multimodal sentiment analysis.

In this work, we propose a method for multimodal sentiment analysis that employs multiple labels, including multimodal labels and individual unimodal labels, from different modalities. Concretely, we construct multimodal modules to manipulate multimodal features jointly, and unimodal modules to handle unimodal labels independently.

We conduct experiments on two publicly available multimodal datasets, MVSA-Single and MVSA-Multiple, as well as compare with existing methods to verify the effectiveness of our method on a fair evaluation scheme.

Further reading:

• Click here to see the PDF version of this work (Abstract and TOC).