Dynamics of relevance judgment during physicians’ online information search process for patient treatment

Physicians rely on online sources of information more so than ever, to find the information they need when treating patients, but little is known about determining the relevance criteria when searching for online sources. Relevance judgment criteria can change with a user’s background knowledge on a search topic. Physicians are a subset of users who have received insufficient attention in this relevance criteria research topic. Due to the dynamic and multidimensional nature of relevance judgments, it is important to understand how relevance judgment criteria change in the course of information search. This study aims to determine the dynamics in relevance judgment criteria in physicians’ information search process. We observed ten internal medicine physicians while they searched the online clinical information resource to resolve a clinical question concerning patient treatment. They rated on a checklist to assess relevance judgment criteria at each of the three stages in the search process, namely, problem recognition (before searching), system interaction (searching for online resources), and document interaction (selecting the final documents after searching). We conducted pre-interview and post-interview to probe physicians search behaviours and reasons for the assessments. Topical relevance, accuracy, and credibility were found to be the most important criteria for all three search stages; however, ratings of some relevance judgment criteria, such as title, obtainability, personal preference, and understandability, changed markedly across the search process. The criteria regarding the information content tended to be more important during the problem recognition stage, while the criteria in the information format became more important as the search progressed. Physicians preferred recently published review articles in highly reliable journals with well-written article outlines, which allowed them to have quick and clear understanding of the contents. By identifying specific relevance criteria prominent at each search stage, our study revealed that physicians employ a unique set of criteria when judging the relevance. Given the fact that physicians are relying on online resources to find answers to patient care, this study can help in developing the effective clinical information systems for physicians and integrating into the Electronic Health Record (EHR) systems.