Introduction

Artificial intelligence (AI) can transform every aspect of daily living, including healthcare [1]. The field of oncology was one of the most studied specialties in which AI emerged as a feasible way to improve patient care. In this regard, AI has already been used in cancer screening, molecular pathology, and big data analysis [2,3]. Furthermore, AI was promising as a decision-making tool after the earlier success of several models like IBM Watson [4]. However, the interest in this area was diminished after the limited clinical benefit of these models in complex clinical scenarios. Although not primarily developed for clinical decision and healthcare, the newly created large language models (LLMs) could potentially counter the limitations of previous clinical decision support systems due to their training with larger datasets [5,6]. However, the potential of these LLMs in resolving complex clinical scenarios and generating true medical information has not been thoroughly investigated.

The most important prominent members of the recently developed LLMs are ChatGPT 3.5, ChatGPT 4, and Google Bard. In earlier studies, these LLMs were able to give accurate medical information in over the clinical scenarios [7,8]. Additionally, to evaluate their performance against human intelligence, these LLMs were tested in medical board exams. While the earlier results were very promising and demonstrated proficiency in board exams like United States Medical Licensing Examination (USMLE) [9] and radiology board exams [10], the LLMs’ success was lower in specialties like ophthalmology [11] and neurology [12], in which more complex clinical scenarios and higher-order questions were more frequent. Additionally, the performance of individual LLMs, as well as the versions of the individual LLMs varied [11,13-15]. Despite this body of evidence in cardiology [16], radiology [17], and surgery boards [18], the performance of LLMs was not investigated in oncology board exams. Considering the huge burden of cancer, the generation of accurate information and the proficiency of LLMs in oncology is paramount. Therefore, we evaluated the performance of LLMs in an oncology board exam (Turkish Society of Medical Oncology Board Exam) and compared the performance of individual LLMs.

Methods

Sample questions

A question bank comprising the questions from the last three years (2021-2023) and including a total of 290 questions was used (Supplement). The questions for the board certification of Medical Oncologists from Turkey were created by a Turkish Society of Medical Oncology Proficiency Board Members, and the passing grades for each year were previously calculated according to the difficulty of this year’s exam. The passing score was 53, 49, and 58 for 2021, 2022 and 2023, respectively. The full question set used in this study is openly available

The questions were multiple choice questions with five options and one correct answer with four distractors. The questions were retrieved from the website of the Turkish Society of Medical Oncology (members area). They were used with the approval of the Turkish Society of Medical Oncology Executive Board. The proficiency board members previously calculated the individual question difficulties, and these difficulty levels were used to compare the difficulty level annotated by the LLMs. Question topics and formats were categorized as case-based or general knowledge questions.

Data collection

The ChatGPT 3.5, ChatGPT 4, and Google Bard LLMs were used via the individual website interfaces. While the previous ChatGPT models were trained up to September 2021, this restriction was removed on September 27th, 2023, and both ChatGPT versions have access to live data via internet browsing. Similarly, the Google Bard could have live data from the internet. Therefore, all three LLMs were expected to provide answers in light of the most up-to-date data.

The researchers did not additionally pre-trained the LLMs before replying to the questions. The questions were asked in Turkish, and the LLMs were asked to answer in Turkish whenever possible. In another turn, the LLMs were also asked to translate the questions into English and give answers in English with explanations.

The following command was given to individual LLMs to gather data and all answer choices to this command and explanations were recorded.

“You are a medical oncologist and you are taking the oncology board exam. The board exam consists of multiple choice questions.

• Please write your answer so that there is only one correct answer among 5 options.
• Give an explanation
• Rate your confidence in your answer according to Likert performance with the following scales: 1 = do not trust [indicates that he/she does not know]; 2 = little confidence [i.e., maybe]; 3 = some interference; 4 = confidence [i.e., likely]; 5 = high confidence [stating the answer and explanation without doubt])
• Grading the question’s difficulty level according to Likert performance with the following grades: 1=Very easy question, 2=Easy question, 3=Medium question, 4=Difficult question, 5=Very difficult question.

Make sure you have these 4 in your output.

Answer each question with this format:

Answer:

Explanation:

Confidence level

Difficulty level:”

Statistical analyses and ethical considerations

The baseline question characteristics, and the accuracy of the LLMs were expressed with frequencies and percentages. The comparison of the accuracy of the individual LLMs and the comparison of the accuracy of the LLMs in original and translated versions of the questions were conducted with Chi-square tests. The comparison of median Likert scores for case questions vs general knowledge was conducted with Mann-Whitney U test. All statistical analyses were conducted with SPSS, version 25.0 (IBM Inc., Armonk, NY, USA), and a type 1 error level of 5% (p<0.05) was considered as the threshold limit for statistical significance.

Due to the use of a previously available question bank and no involvement of human subjects, the study is exempt from ethical approval. The study was conducted and reported according to STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) guidelines [19].

Results

A total of 290 questions were evaluated via three LLMs. Breast (n=31), lung (n=24), and colorectal cancers (n=29) were the most frequently assessed tumor types in the question bank. The distribution of case questions and general knowledge-based questions were even (49.7% vs. 50.3%). The treatment was the most frequently evaluated area (49.3%). The topic distribution for the questions is summarized in Table 1.

The overall accuracy of LLMs was 59.3%, 42.8%, 36.2% for ChatGPT4, ChatGPT3.5, and Google Bard, respectively. The accuracy of ChatGPT4 was significantly higher than that of ChatGPT3.5 (p<0.001) and Google Bard (p<0.001), while the accuracy of ChatGPT3.5 was higher than that of Google Bard (p<0.001). Only the ChatGPT4 was proficient in all three examination years (2021-2023). The three LLMs had moderate correlation in their accuracy for individual responses (r=0.319, p<0.001 for ChatGPT 3.5 vs. ChatGPT 4, r=0.263, p<0.001 for ChatGPT 3.5 vs. Google Bard and r=0.288, p<0.001 for ChatGPT 4 vs. Google Bard). The accuracy of the LLMs was higher for the translated questions compared to original language in all three LLMs (62.8 vs. 59.3%, p<0.001 for ChatGPT 4, 48.3 vs. 42.8%, p<0.001 for ChatGPT3.5and 43.1 vs. 36.2%, p<0.001 for Google Bard). While the accuracy was improved with the translated questions, the LLMS were not able to replicate the correct answers given in the original language for over 10% of the questions (Table 2).
The LLMs were more accurate in general knowledge than case questions (Table 3). The ChatGPT 3.5 (p=0.301) and Google Bard (p=0.378) had similar accuracy across variable knowledge domains (treatment, diagnosis, prognosis, toxicity, general information), while the performance of ChatGPT 4 varied across knowledge domains (Table 3). The accuracy of the LLMs across tumor types demonstrated similar accuracy across most tumor types.

The median Likert score was 4 for ChatGPT4, ChatGPT3.5, and Google Bard (Figure 1). The LLMs’ answers’ certainty was higher in general knowledge than in case questions for Google Bard (p=0.001), while the certainty for general knowledge questions and case questions were similar in ChatGPT 3.5 (p=0.135) and ChatGPT 4 (p=0.111). The median difficulty was 3/5 for all three LLMs (Figure 2). The question difficulty was regarded as higher case questions than general knowledge questions for ChatGPT 3.5 (p<0.001), ChatGPT 4 (p<0.001) and Google Bard (p<0.001).

Discussion

In the present study, we observed that LLMs had moderate success in medical oncology proficiency. The performance of the individual LLMs significantly varied, with ChatGPT4 outperforming the two other LLMs. The LLMs underperformed in clinical cases, mirroring complex clinical scenarios in daily practice. Additionally, the performance of LLMs was lower in the native language compared to questions translated to English. To best our knowledge, our study is the first study evaluating the proficiency of LLMs in oncology proficiency.

The use of LLMs as an assistant to clinical practice garnered a lot interest in the last year, especially after the eye opening performance of the ChatGPT 3 in USMLE examination [20]. In the pivotal study which was broadcasted via even television news, the ChatGPT achieved around 60% accuracy and an acceptable reasoning for the responses (20). In a later work, Brin et al. compared the performance of ChatGPT 4 and ChatGPT 3.5. The ChatGPT4 outperformed ChatGPT 3.5 (correct response rate 90 vs. 62.5%) [21]. Additionally, the ChatGPT4 was consistent with repeated evaluations, while the ChatGPT 3.5 revised its’ responses in 82.5% of the cases [21], further supporting the use of ChatGPT 4 in medical knowledge. However, later studies challenged these findings.

Regarding the accuracy of LLMs in cancer care, similar unequivocal results exist. The ChatGPT 4 was accurate and comprehensive, with over 85% of questions related to head and neck cancer knowledge [22]. The questions in the study were generated from the frequently asked questions of professional societies, support groups, and social media, reflecting the potential of the LLMs in population-level healthcare education [22]. In contrast, ChatGPT provided accurate and comprehensive responses to only 53.1% of the questions regarding cervical cancer care in another study [23]. It should be noted that this study used ChatGPT 3.5 [23], which underperformed in our analysis compared to ChatGPT 4. Whether the newer versions of the LLMs could improve healthcare information should be investigated.

There are several caveats regarding the use of LLMs in cancer care. First, the LLMs gather information from several sources with various publishing dates. In the earlier works with the ChatGPT in healthcare, the training time limitation up to September 2021 was an important problem. While this issue was resolved in September 2023, some of the recommendations by LLMs are still outdated. Although the LLMs gather information from several sources, the reasoning for selecting a particular reference is not fully delineated by artificial intelligence, as previously noted in IBM Watson studies [24]. This issue is particularly problematic in complex scenarios with more than one therapeutic option. Additionally, the LLMs could not understand and solve the complex clinical scenarios that are very common in clinical practice [25]. Moreover, LLMs may generate errors in treatment sequencing and timing in such cases and may assign high confidence to incorrect or potentially unsafe recommendations. Furthermore, in line with the general principles of the LLMs, the LLMs used very confident language even in the obviously wrong simple questions [13,26]. The reasoning for the problems requiring the best option. Lastly, less is known about the comparative efficacy of different LLMs. In several studies, the performance of newer versions of the LLMs was better compared to previous versions [11,13], and the ChatGPT outperformed Google Bard in a very recent study on radiology exam questions [17]. Further studies should separately evaluate the performance of different LLMs to delineate the best model for individual scenarios.

In conclusion, the LLMs had below acceptable performance in a national oncology board exam with only the ChatGPT 4.0 had proficiency in an oncology board examination. The LLMs’ success in questions mirroring clinical practice was lower. Further research is needed to improve the proficiency of LLMs in cancer care-related information.

Author contributions

Conception and design: T.K.S., M.D., N.K., D.C.G.; Data acquisition: T.K.S., D.C.G.; Data analysis: T.K.S., D.C.G.; Data interpretation: T.K.S., D.C.G.; Drafting of the manuscript: T.K.S., M.D., N.K., D.C.G.; Critical revision of the manuscript: T.K.S., D.C.G.. All authors reviewed the results, approved the final version of the manuscript, and agreed to be accountable for all aspects of this study.

Ethical approval

Due to the use of a previously available question bank and no involvement of human subjects, the study is exempt from ethical approval.

Data availability statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Conflict of interest

The authors declare that this study was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Funding

The authors declare that this study received no funding.

Generative AI statement

The authors declare that no generative AI or AI-assisted technologies were used in the writing or preparation of this study.