This paper details materials, methods, results, and analyses of the Consumer Health Search Task of the CLEF eHealth 2021 Evaluation Lab. This task investigates the effectiveness of information retrieval (IR) approaches in providing access to medical information to laypeople. For this a TREC-style evaluation methodology was applied: a shared collection of documents and queries is distributed, participants' runs received, relevance assessments generated, and participants' submissions evaluated. The task generated a new representative web corpus including web pages acquired from a 2021 CommonCrawl and social media content from Twitter and Reddit, along with a new collection of 55 manually generated layperson medical queries and their respective credibility, understandability, and topicality assessments for returned documents. This year's task focused on three subtask: (i) ad-hoc IR, (ii) weakly supervised IR, and (iii) document credibility prediction. In total, 15 runs were submitted to the three subtasks: eight addressed the ad-hoc IR task, three the weakly supervised IR challenge, and 4 the document credibility prediction challenge. As in previous years, the organizers have made data and tools associated with the task available for future research and development.

Goeuriot, L., Suominen, H., Pasi, G., Bassani, E., Brew-Sam, N., Gonzalez-Saez, G., et al. (2021). Consumer Health Search at CLEF eHealth 2021. In 2021 Working Notes of CLEF - Conference and Labs of the Evaluation Forum, CLEF-WN 2021 (pp.751-769). CEUR-WS.

Consumer Health Search at CLEF eHealth 2021

Pasi G.;Bassani E.;Upadhyay R.;Viviani M.;
2021

Abstract

This paper details materials, methods, results, and analyses of the Consumer Health Search Task of the CLEF eHealth 2021 Evaluation Lab. This task investigates the effectiveness of information retrieval (IR) approaches in providing access to medical information to laypeople. For this a TREC-style evaluation methodology was applied: a shared collection of documents and queries is distributed, participants' runs received, relevance assessments generated, and participants' submissions evaluated. The task generated a new representative web corpus including web pages acquired from a 2021 CommonCrawl and social media content from Twitter and Reddit, along with a new collection of 55 manually generated layperson medical queries and their respective credibility, understandability, and topicality assessments for returned documents. This year's task focused on three subtask: (i) ad-hoc IR, (ii) weakly supervised IR, and (iii) document credibility prediction. In total, 15 runs were submitted to the three subtasks: eight addressed the ad-hoc IR task, three the weakly supervised IR challenge, and 4 the document credibility prediction challenge. As in previous years, the organizers have made data and tools associated with the task available for future research and development.
paper
Dimensions of relevance; EHealth; Evaluation; Health records; Information storage and retrieval; Medical informatics; Self-diagnosis; Test-set generation;
English
2021 Working Notes of CLEF - Conference and Labs of the Evaluation Forum, CLEF-WN 2021 - 21 September 2021 through 24 September 2021
2021
Faggioli G; Ferro N; Joly A; Maistro M; Piroi F;
2021 Working Notes of CLEF - Conference and Labs of the Evaluation Forum, CLEF-WN 2021
2021
2936
751
769
none
Goeuriot, L., Suominen, H., Pasi, G., Bassani, E., Brew-Sam, N., Gonzalez-Saez, G., et al. (2021). Consumer Health Search at CLEF eHealth 2021. In 2021 Working Notes of CLEF - Conference and Labs of the Evaluation Forum, CLEF-WN 2021 (pp.751-769). CEUR-WS.
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/10281/325119
Citazioni
  • Scopus 5
  • ???jsp.display-item.citation.isi??? ND
Social impact