Factors affecting relevance judgment: a report from TREC Legal track

• Published in: Journal of documentation Vol. 67; no. 2; pp. 264 - 278
• Main Author: Chu, Heting
• Format: Journal Article
• Language: English
• Published: Bingley Emerald Group Publishing Limited 01.01.2011; Emerald
• Subjects: Aerospace Education; Criteria; Data collection; Documents; Exact sciences and technology; Human beings; Image retrieval; Information and communication sciences; Information retrieval; Information science. Documentation; Information Seeking; Information seeking behavior; Interactive; Judgements; Judgments; Legal; Library and information science. General aspects; Lists; Literature reviews; Online information retrieval; Questionnaires; R&D; Relevance; Research & development; Researchers; Sciences and techniques of general use; Searching; Studies; Tasks; Use and user studies. Information needs; Users (Information); Information retrieval; Assessment; Information management; Evaluation; Questionnaire
• ISSN: 0022-0418; 1758-7379
• DOI: 10.1108/00220411111109467

Purpose - This study intends to identify factors that affect relevance judgment of retrieved information as part of the 2007 TREC Legal track interactive task.Design methodology approach - Data were gathered and analyzed from the participants of the 2007 TREC Legal track interactive task using a questionnaire which includes not only a list of 80 relevance factors identified in prior research, but also a space for expressing their thoughts on relevance judgment in the process.Findings - This study finds that topicality remains a primary criterion, out of various options, for determining relevance, while specificity of the search request, task, or retrieved results also helps greatly in relevance judgment.Research limitations implications - Relevance research should focus on the topicality and specificity of what is being evaluated as well as conducted in real environments.Practical implications - If multiple relevance factors are presented to assessors, the total number in a list should be below ten to take account of the limited processing capacity of human beings' short-term memory. Otherwise, the assessors might either completely ignore or inadequately consider some of the relevance factors when making judgment decisions.Originality value - This study presents a method for reducing the artificiality of relevance research design, an apparent limitation in many related studies. Specifically, relevance judgment was made in this research as part of the 2007 TREC Legal track interactive task rather than a study devised for the sake of it. The assessors also served as searchers so that their searching experience would facilitate their subsequent relevance judgments.