Personalization in Information Retrieval has been a hot topic in both academia and industry for the past two decades. However, there is still a lack of high-quality standard benchmark datasets for conducting offline comparative evaluations in this context. To mitigate this problem, in the past few years, approaches to derive synthetic datasets suited for evaluating Personalized Search models have been proposed. In this paper, we put forward a novel evaluation benchmark for Personalized Search with more than 18 million documents and 1.9 million queries across four domains. We present a detailed description of the benchmark construction procedure, highlighting its characteristics and challenges. We provide baseline performance including pre-trained neural models, opening room for the evaluation of personalized approaches, as well as domain adaptation and transfer learning scenarios. We make both datasets and models available for future research.

Bassani, E., Kasela, P., Raganato, A., Pasi, G. (2022). A Multi-Domain Benchmark for Personalized Search Evaluation. In International Conference on Information and Knowledge Management, Proceedings (pp.3822-3827). Association for Computing Machinery [10.1145/3511808.3557536].

A Multi-Domain Benchmark for Personalized Search Evaluation

Bassani, Elias
Primo
;
Kasela, Pranav
Secondo
;
Raganato, Alessandro
Penultimo
;
Pasi, Gabriella
Ultimo
2022

Abstract

Personalization in Information Retrieval has been a hot topic in both academia and industry for the past two decades. However, there is still a lack of high-quality standard benchmark datasets for conducting offline comparative evaluations in this context. To mitigate this problem, in the past few years, approaches to derive synthetic datasets suited for evaluating Personalized Search models have been proposed. In this paper, we put forward a novel evaluation benchmark for Personalized Search with more than 18 million documents and 1.9 million queries across four domains. We present a detailed description of the benchmark construction procedure, highlighting its characteristics and challenges. We provide baseline performance including pre-trained neural models, opening room for the evaluation of personalized approaches, as well as domain adaptation and transfer learning scenarios. We make both datasets and models available for future research.
poster + paper
dataset; information retrieval; personalization; personalized search;
English
CIKM 2022 - 31st ACM International Conference on Information and Knowledge Management - October 17 - 21, 2022
2022
International Conference on Information and Knowledge Management, Proceedings
9781450392365
2022
3822
3827
none
Bassani, E., Kasela, P., Raganato, A., Pasi, G. (2022). A Multi-Domain Benchmark for Personalized Search Evaluation. In International Conference on Information and Knowledge Management, Proceedings (pp.3822-3827). Association for Computing Machinery [10.1145/3511808.3557536].
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/10281/395822
Citazioni
  • Scopus 6
  • ???jsp.display-item.citation.isi??? 5
Social impact