A Decision-Theoretic Framework to Multidimensional Relevance Estimation

Relevance in Information Retrieval systems is a multidimensional and dynamic notion, significantly influenced by user, task and domain factors. Existing multidimensional relevance models often only account for factors that positively affect a document’s overall relevance (utility), neglecting the potential impact of factors that negatively influence utility. This paper addresses this gap by introducing the Decision-theoretic Multidimensional Relevance Framework (DtMRF), a formal framework for multidimensional relevance estimation capable of accounting for positive and negative relevance factors. These factors are first assessed and subsequently aggregated by a decision-theoretic method to provide an overall relevance estimate of a document to a considered query. DtMRF overcomes the computational complexity limitations of data-driven approaches while offering interpretable document rankings. When evaluated on three benchmark collections for clinical trial retrieval, DtMRF improves P@10 by 5–28% over standard ad-hoc retrieval methods, achieving statistically significant gains. Further analysis indicates that DtMRF effectively balances efficiency and effectiveness while being far from its upper effectiveness bound, leaving ample room for future improvements.