TY - JOUR
T1 - An Analysis of Web-scale Discovery Services From the Perspective of User's Relevance Judgment
AU - Lee, Boram
AU - Chung, Eun Kyung
N1 - Publisher Copyright:
© 2016 Elsevier Inc.
PY - 2016/9/1
Y1 - 2016/9/1
N2 - Although web-scale discovery services have been implemented increasingly worldwide, there is a need for the exploration of search effectiveness from users' perspectives. In this sense, this study examined web-scale discovery services in the view of users' relevance judgment comparing to individual databases in the fields of Education and Library and Information Science. Using four search topics for the EBSCO Discovery Service (EDS), ERIC, ERC, LISA, and LISTA, their search effectiveness were measured in terms of modified precision, recall, and reciprocal rank based on the relevance judgments of four participants. Comparison of the measurements showed that the web-scale discovery service was less effective than individual databases. In particular, EDS's effectiveness was lower than that of ERIC and ERC in terms of modified precision and recall. The modified reciprocal rank of EDS was lower than those of most individual databases in the fields of Education and LIS. Moreover, the relationship between the rankings from EDS and those from four participants was weak, as indicated by Spearman rank-order correlation coefficients (0.141 in Education and 0.170 in LIS). In fact, the effectiveness in the fields of LIS and Education of EDS was lower than those individual databases to a degree dependent on the field.
AB - Although web-scale discovery services have been implemented increasingly worldwide, there is a need for the exploration of search effectiveness from users' perspectives. In this sense, this study examined web-scale discovery services in the view of users' relevance judgment comparing to individual databases in the fields of Education and Library and Information Science. Using four search topics for the EBSCO Discovery Service (EDS), ERIC, ERC, LISA, and LISTA, their search effectiveness were measured in terms of modified precision, recall, and reciprocal rank based on the relevance judgments of four participants. Comparison of the measurements showed that the web-scale discovery service was less effective than individual databases. In particular, EDS's effectiveness was lower than that of ERIC and ERC in terms of modified precision and recall. The modified reciprocal rank of EDS was lower than those of most individual databases in the fields of Education and LIS. Moreover, the relationship between the rankings from EDS and those from four participants was weak, as indicated by Spearman rank-order correlation coefficients (0.141 in Education and 0.170 in LIS). In fact, the effectiveness in the fields of LIS and Education of EDS was lower than those individual databases to a degree dependent on the field.
KW - Relevance judgment
KW - Relevance ranking
KW - Search effectiveness
KW - user's perspective
KW - User-oriented
KW - Web-scale discovery
UR - http://www.scopus.com/inward/record.url?scp=84991384380&partnerID=8YFLogxK
U2 - 10.1016/j.acalib.2016.06.016
DO - 10.1016/j.acalib.2016.06.016
M3 - Article
AN - SCOPUS:84991384380
VL - 42
SP - 529
EP - 534
JO - Journal of Academic Librarianship
JF - Journal of Academic Librarianship
SN - 0099-1333
IS - 5
ER -