Evaluating the Use of Generative Artificial Intelligence to Support Genetic Counseling for Rare Diseases

Background/Objectives: Rare diseases often present challenges in obtaining reliable and accurate information than common diseases owing to their low prevalence. Patients and families often rely on self-directed learning, but understanding complex medical information can be difficult, increasing the risk of misinformation. This study aimed to evaluate whether generative artificial intelligence (AI) provides accurate and non-harmful answers to rare disease-related questions and assesses its utility in supporting patients and families requiring genetic counseling. Methods: We evaluated four generative AI models available between 22 September and 4 October 2024: ChatGPT o1-Preview, Gemini advanced, Claude 3.5 sonnet, and Perplexity sonar huge. A total of 102 questions targeting four rare diseases, covering general information, diagnosis, treatment, prognosis, and counseling, were prepared. Four evaluators scored the responses for professionalism and accuracy using the Likert scale (1: poor, 5: excellent). Results: The average scores ranked the AI models as: ChatGPT (4.24 ± 0.73), Gemini (4.15 ± 0.74), Claude (4.13 ± 0.82), and Perplexity (3.35 ± 0.80; p < 0.001). Perplexity had the highest proportion of scores of 1 (very poor) and 2 (poor) (7.6%, 31/408), followed by Gemini (2.0%, 8/408), Claude (1.5%, 6/408), and ChatGPT (1.5%, 6/408). The accuracy of responses in the counseling part across all four diseases was significantly different (p < 0.001). Conclusions: The four generative AI models generally provided reliable information. However, occasional inaccuracies and ambiguous references may lead to confusion and anxiety among patients and their families. To ensure its effective use, recognizing the limitations of generative AI and providing guidance from experts regarding its proper utilization is essential.