PICOT questions and search strategies formulation: A novel approach using artificial intelligence automation

• Published in: Journal of nursing scholarship Vol. 57; no. 1; pp. 5 - 16
• Main Authors: Gosak, Lucija, Štiglic, Gregor, Pruinelli, Lisiane, Vrbnjak, Dominika
• Format: Journal Article
• Language: English
• Published: United States Blackwell Publishing Ltd 01.01.2025; John Wiley and Sons Inc
• Subjects: Accuracy; AI language models; Artificial Intelligence; Automation; Chatbots; Clinical nursing; Clinical outcomes; clinical questions; Clinical Scholarship; Decision making; Efficiency; Evidence-based nursing; Evidence-based practice; Health care; Humans; Information retrieval; Information Storage and Retrieval - methods; Language; Large language models; Librarians; Medical personnel; Nurses; Original; Relevance; Retrieval; Scripts; Search strategies; Skills; Subject heading schemes; AI language models; evidence‐based practice; clinical questions; search strategies; artificial intelligence
• ISSN: 1527-6546; 1547-5069; 1547-5069
• DOI: 10.1111/jnu.13036

Aim The aim of this study was to evaluate and compare artificial intelligence (AI)‐based large language models (LLMs) (ChatGPT‐3.5, Bing, and Bard) with human‐based formulations in generating relevant clinical queries, using comprehensive methodological evaluations. Methods To interact with the major LLMs ChatGPT‐3.5, Bing Chat, and Google Bard, scripts and prompts were designed to formulate PICOT (population, intervention, comparison, outcome, time) clinical questions and search strategies. Quality of the LLMs responses was assessed using a descriptive approach and independent assessment by two researchers. To determine the number of hits, PubMed, Web of Science, Cochrane Library, and CINAHL Ultimate search results were imported separately, without search restrictions, with the search strings generated by the three LLMs and an additional one by the expert. Hits from one of the scenarios were also exported for relevance evaluation. The use of a single scenario was chosen to provide a focused analysis. Cronbach's alpha and intraclass correlation coefficient (ICC) were also calculated. Results In five different scenarios, ChatGPT‐3.5 generated 11,859 hits, Bing 1,376,854, Bard 16,583, and an expert 5919 hits. We then used the first scenario to assess the relevance of the obtained results. The human expert search approach resulted in 65.22% (56/105) relevant articles. Bing was the most accurate AI‐based LLM with 70.79% (63/89), followed by ChatGPT‐3.5 with 21.05% (12/45), and Bard with 13.29% (42/316) relevant hits. Based on the assessment of two evaluators, ChatGPT‐3.5 received the highest score (M = 48.50; SD = 0.71). Results showed a high level of agreement between the two evaluators. Although ChatGPT‐3.5 showed a lower percentage of relevant hits compared to Bing, this reflects the nuanced evaluation criteria, where the subjective evaluation prioritized contextual accuracy and quality over mere relevance. Conclusion This study provides valuable insights into the ability of LLMs to formulate PICOT clinical questions and search strategies. AI‐based LLMs, such as ChatGPT‐3.5, demonstrate significant potential for augmenting clinical workflows, improving clinical query development, and supporting search strategies. However, the findings also highlight limitations that necessitate further refinement and continued human oversight. Clinical Relevance AI could assist nurses in formulating PICOT clinical questions and search strategies. AI‐based LLMs offer valuable support to healthcare professionals by improving the structure of clinical questions and enhancing search strategies, thereby significantly increasing the efficiency of information retrieval.