Beyond distortions: a benchmark for subjective evaluation of image rendering quality

Traditional Image Quality Assessment (IQA) has primarily aimed to quantify perceptual quality in terms of technical degradations such as noise, blur, or compression artifacts. However, in image rendering, the key factor influencing perceived quality is not the presence of degradations but the manner in which color processing algorithms are applied, as they directly shape the overall aesthetic appearance of the image. To date, the quantitative evaluation of how rendering methods affect image quality has been insufficiently addressed. In this work, we introduce Image Rendering Quality Assessment (IRQA) as a new problem setting within IQA and present REPID, a benchmark designed for its study. REPID contains 30,000 edited images and preference annotations collected from 13,648 voters, resulting in an over 2.5 million unique votes. Based on REPID, we investigate content-dependent render preferences and the influence of rendering parameters, and further explore applications such as aesthetic preference prediction (including personalization), render ranking, and benchmarking of aesthetic evaluation methods. We further perform an extensive experimental comparison of traditional IQA metrics, handcrafted features, deep learning approaches, and foundation-model embeddings. On the REPID benchmark, IRQA-specific models achieve up to 40% better precision than conventional distortion-oriented IQA methods.