Weighted Utility: A Utility Metric Based on the Case-Wise Raters’ Perceptions

In this article we discuss a novel utility metrics for the evaluation of AI-based decision support systems, which is based on the users’ perceptions of the relevance of, and risks associated with, the validation cases. We discuss the relationship between the proposed metric and other previous proposals in the specialist literature; in particular, we show that our metric generalizes the well-known Net Benefit. More in general, we make the point for having utility as the prime dimension to optimize machine learning models in critical domains, like the medical one, and to evaluate their potential impact on real-world practices.