Zoom In Disparities in Healthcare LLM Q&A

This paper systematically examines cross-lingual disparities in pre-training source and factuality alignment in Large Language Model (LLM) answers for multilingual healthcare Q&A across English, German, Turkish, Chinese (Mandarin), and Italian. To support this analysis, we (i) constructed MultiWikiHealthCare, a multilingual dataset derived from Wikipedia; (ii) used it to examine cross-lingual differences in healthcare-related coverage; (iii) evaluated the alignment between LLM-generated responses and these reference sources; and (iv) conducted a case study on factual alignment through the use of contextual information and Retrieval-Augmented Generation (RAG). Our findings reveal substantial cross-lingual disparities in both Wikipedia coverage and LLM factual alignment. Based on our Wikipedia analysis, the lowest alignment is observed between English and Chinese Wikipedia pages. Across models, responses align more with English Wikipedia, even when the prompts are non-English. We further show that providing contextual excerpts from non-English Wikipedia at inference time effectively shifts factual alignment toward target knowledge.