Texts, images and other information are posted everyday on the social network and provides a large amount of multimodal data. The aim of this work is to investigate if combining and integrating both visual and textual data permits to identify emotions elicited by an image. We focus on image emotion classification within eight emotion categories: amusement, awe, contentment, excitement, anger, disgust, fear and sadness. Within this classification task we here propose to adopt ensemble learning approaches based on the Bayesian model averaging method, that combine five state-of-the-art classifiers. The proposed ensemble approaches consider predictions given by several classification models, based on visual and textual data, through respectively a late and an early fusion schemes. Our investigations show that an ensemble method based on a late fusion of unimodal classifiers permits to achieve high classification performance within all of the eight emotion classes. The improvement is higher when deep image representations are adopted as visual features, compared with hand-crafted ones.

Corchs, S., Fersini, E., Gasparini, F. (2019). Ensemble learning on visual and textual data for social image emotion classification. INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS, 10(8), 2057-2070 [10.1007/s13042-017-0734-0].

Ensemble learning on visual and textual data for social image emotion classification

Corchs, S
;
Fersini, E
;
Gasparini, F
2019

Abstract

Texts, images and other information are posted everyday on the social network and provides a large amount of multimodal data. The aim of this work is to investigate if combining and integrating both visual and textual data permits to identify emotions elicited by an image. We focus on image emotion classification within eight emotion categories: amusement, awe, contentment, excitement, anger, disgust, fear and sadness. Within this classification task we here propose to adopt ensemble learning approaches based on the Bayesian model averaging method, that combine five state-of-the-art classifiers. The proposed ensemble approaches consider predictions given by several classification models, based on visual and textual data, through respectively a late and an early fusion schemes. Our investigations show that an ensemble method based on a late fusion of unimodal classifiers permits to achieve high classification performance within all of the eight emotion classes. The improvement is higher when deep image representations are adopted as visual features, compared with hand-crafted ones.
Articolo in rivista - Articolo scientifico
Bayesian model averaging; Image emotion; Multimodal ensemble learning; Visual and textual social data;
Image emotion,· Multimodal ensemble learning,· Bayesian model averaging, Visual and textual social data
English
7-ott-2017
2019
10
8
2057
2070
reserved
Corchs, S., Fersini, E., Gasparini, F. (2019). Ensemble learning on visual and textual data for social image emotion classification. INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS, 10(8), 2057-2070 [10.1007/s13042-017-0734-0].
File in questo prodotto:
File Dimensione Formato  
Ensemble learning on visual and textual data for social image emotion classification .pdf

Solo gestori archivio

Tipologia di allegato: Publisher’s Version (Version of Record, VoR)
Dimensione 6.19 MB
Formato Adobe PDF
6.19 MB Adobe PDF   Visualizza/Apri   Richiedi una copia

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/10281/174105
Citazioni
  • Scopus 48
  • ???jsp.display-item.citation.isi??? 27
Social impact