中文大语言模型在耳念珠菌医院感染防控决策支持中的性能评价

Performances of Chinese large language models in decision support for prevention and control of health care-associated Candida auris infection

  • 摘要: 目的 评价国内5款主流大语言模型(LLMs)在耳念珠菌医院感染防控辅助决策中的应用效能。方法 基于《耳念珠菌医院感染预防与控制专家共识》构建6个维度、24道测试问题,并按事实型、流程型和判断型任务分层,对通义千问、DeepSeek、智谱清言、豆包和文心一言进行横断面测试。由3名医院感染管理专业人员依据评分细则评价各个模型回答的整体质量、准确性和全面性,并采用组内相关系数(ICC)评价评分者一致性。结果 3名评价者总体ICC为0.927,一致性极佳。通义千问(360分)和DeepSeek(358分)总分位于前两名。各模型在基础知识与管理维度均获满分,在手卫生与防护、患者隔离管理等标准化维度表现较稳定; 失分主要在于皮肤消毒准备、筛查与监测及患者隔离管理等涉及专科细节、适用条件和关键数值的维度。5款模型整体性能总分差异具有统计学意义(χ2=14.400,P=0.006),事实型任务准确性差异亦具有统计学意义(P=0.045)。结论 LLMs在耳念珠菌感控中具有一定辅助决策潜力,整体表现出较好的准确性、信息整合能力和文本生成能力,可作为信息检索、流程提示和初步建议生成工具,但不能替代临床专业判断。

     

    Abstract: OBJECTIVE To evaluate the performance of five mainstream Chinese large language models (LLMs) in supporting decision-making for the prevention and control of Candida auris healthcare-associated infection (HAI). METHODS Based on the Expert Consensus on the Prevention and Control of Candida auris Health Care-associated Infection, a test set comprising 24 questions across six dimensions was developed and stratified into factual, procedural, and judgment-based tasks. A cross-sectional evaluation was conducted on Qwen, DeepSeek, GLM, Doubao, and ERNIE Bot. Three infection control professionals assessed the overall quality, accuracy, and comprehensiveness of model-generated responses by using an operationalized scoring rubric. The inter-rater reliability was evaluated by the intraclass correlation coefficient (ICC). RESULTS The overall ICC of the three raters was 0.927, indicating excellent agreement. Qwen achieved the highest total score (360 points), followed by DeepSeek (358 points). All of the models achieved full scores in the basic knowledge and management dimension and showed relatively stable performance in standardized dimensions such as hand hygiene and personal protection, and patient isolation management. The performance losses were mainly observed in screening and monitoring, skin disinfection preparation and patient isolation management, which involved specialized details, application condition and key numerical dimensions. The Friedman test showed a significant difference in the overall performance scores among the five models (χ2=14.400, P= 0.006), as well as a significant difference in the accuracy for factual tasks (P= 0.045). CONCLUSIONS LLMs exhibit certain potential for decision support in control of the C. auris infection and show generally high accuracy, powerful information integration, and text generation capabilities, and they may serve as tools for information retrieval, process reminders, and preliminary recommendation generation, but should not replace the professional clinical judgment.

     

/

返回文章
返回