In this article we discuss a novel utility metrics for the evaluation of AI-based decision support systems, which is based on the users’ perceptions of the relevance of, and risks associated with, the validation cases. We discuss the relationship between the proposed metric and other previous proposals in the specialist literature; in particular, we show that our metric generalizes the well-known Net Benefit. More in general, we make the point for having utility as the prime dimension to optimize machine learning models in critical domains, like the medical one, and to evaluate their potential impact on real-world practices.

Campagner, A., Conte, E., Cabitza, F. (2021). Weighted Utility: A Utility Metric Based on the Case-Wise Raters’ Perceptions. In Machine Learning and Knowledge Extraction - 5th IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross-Domain Conference, CD-MAKE 2021, Virtual Event, August 17–20, 2021, Proceedings (pp.203-210). Cham : Springer Science and Business Media Deutschland GmbH [10.1007/978-3-030-84060-0_13].

Weighted Utility: A Utility Metric Based on the Case-Wise Raters’ Perceptions

Campagner A.;Cabitza F.
2021

Abstract

In this article we discuss a novel utility metrics for the evaluation of AI-based decision support systems, which is based on the users’ perceptions of the relevance of, and risks associated with, the validation cases. We discuss the relationship between the proposed metric and other previous proposals in the specialist literature; in particular, we show that our metric generalizes the well-known Net Benefit. More in general, we make the point for having utility as the prime dimension to optimize machine learning models in critical domains, like the medical one, and to evaluate their potential impact on real-world practices.
paper
Decision support; Medical machine learning; Utility; Validation
English
5th IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross-Domain Conference on Machine Learning and Knowledge Extraction, CD-MAKE 2021
2021
Machine Learning and Knowledge Extraction - 5th IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross-Domain Conference, CD-MAKE 2021, Virtual Event, August 17–20, 2021, Proceedings
978-3-030-84059-4
2021
12844
203
210
reserved
Campagner, A., Conte, E., Cabitza, F. (2021). Weighted Utility: A Utility Metric Based on the Case-Wise Raters’ Perceptions. In Machine Learning and Knowledge Extraction - 5th IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross-Domain Conference, CD-MAKE 2021, Virtual Event, August 17–20, 2021, Proceedings (pp.203-210). Cham : Springer Science and Business Media Deutschland GmbH [10.1007/978-3-030-84060-0_13].
File in questo prodotto:
File Dimensione Formato  
978-3-030-84060-0_13.pdf

Solo gestori archivio

Descrizione: Conference Proceedings
Tipologia di allegato: Publisher’s Version (Version of Record, VoR)
Dimensione 741.79 kB
Formato Adobe PDF
741.79 kB Adobe PDF   Visualizza/Apri   Richiedi una copia

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/10281/388637
Citazioni
  • Scopus 1
  • ???jsp.display-item.citation.isi??? 1
Social impact