When ‘Scarce’ Becomes ‘Too Scarce’: The Rare Diseases Drama

Rare diseases pose unique methodological challenges for causal questions, where data scarcity is not incidental but inherent. We present a case study on the external validation of a causal Bayesian network (BN) developed to investigate cardiovascular diseases (CVDs) in adolescent and young adult female breast cancer survivors. Despite strong internal performance, external validation across multiple Italian regions revealed scenarios where the model’s predictions collapsed into complete uncertainty. These included: (i) circumstances in which shifts in missingness and selection mechanisms made the original model parameters inappropriate (ii) unobserved infrequent treatments in training data; and (iii) novel treatment combinations emerging from evolving clinical protocols. We discuss the implications of these findings, emphasizing the need for including informative priors, making timely model updates, and explicit uncertainty quantification. Our experience underscores that in rare disease research, robust causal modeling demands critical scrutiny, adaptive learning, and the indispensable role of human revision. These lessons could be broadly applicable to other domains where data are scarce, emphasizing that methodological rigor and adaptability are as crucial as the models themselves.