How consistent are artificial intelligence responses with cochlear implant guidelines?

Background: Intraoperative testing is a critical component of cochlear implant surgery. As AI tools like ChatGPT intersect with medical communication, it is important to assess their ability to handle highly specialized clinical content. Objective: To evaluate ChatGPT-4’s capacity to generate accurate and expert-level responses in a specialized surgical domain by comparing its answers on intraoperative testing in cochlear implant (CI) surgery with statements from an international expert consensus. Methods: Key questions and statements from the International Consensus on Intraoperative CI Testing were presented to GPT-4 twice. Two independent reviewers rated response similarity as high, medium, or low. Discrepancies were resolved by a third reviewer. GPT-4’s self-assessments were also collected. Results: Of 24 questions, 54.2% of responses were rated highly similar, 33.3% moderately similar, and 12.5% low similarity. GPT-4 self-rated 33.3% as highly similar and 66.7% as moderate. Response reproducibility was 79.2%. Inter-reviewer agreement was almost perfect (κ = 0.86), while agreement between reviewers and GPT-4 was moderate (κ = 0.44). Conclusion: GPT-4 demonstrates moderate alignment with expert consensus on cochlear implant testing but lacks the clinical depth required for autonomous decision-making. While it shows promise for supporting clinical communication, further refinement is needed in high-stakes surgical contexts.