A new algorithm for modeling online search behavior and studying ranking reliability variations

• Published in: Applied intelligence (Dordrecht, Netherlands) Vol. 52; no. 7; pp. 7529 - 7549
• Main Authors: Di Caprio, Debora, Santos-Arteaga, Francisco J., Tavana, Madjid
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.05.2022; Springer Nature B.V
• Subjects: Algorithms; Alternatives; Artificial Intelligence; Computer Science; Decision making; Decision trees; Information retrieval; Machines; Manufacturing; Mechanical Engineering; Processes; Ranking; Reliability analysis; Search engines; Information retrieval; Ranking reliability; Click-through rate; Online search algorithm; Decision tree
• ISSN: 0924-669X; 1573-7497
• DOI: 10.1007/s10489-021-02856-8

We design an information retrieval algorithm that mimics the stochastic behavior of decision-makers (DMs) when evaluating the alternatives displayed by an online search engine. The algorithm consists of a decision tree that incorporates all the 1024 decision nodes that may arise from the information retrieval process of DMs. We calibrate the behavior of the algorithm to the one observed from online users and run several sets of 1,000,000 queries. Each query lets DMs decide which subset of the ten alternatives composing the initial page of results to click, allowing us to evaluate their behavior as ranking reliability is assumed to decrease when DMs decide not to click on an alternative. We compare the click-through rates (CTRs) obtained when modifying the degree of ranking reliability derived from the alternatives displayed on the first page of search results. We illustrate how the stability of the CTR prevails among the top-ranked alternatives within relatively reliable scenarios while it drops when imposing large initial decrements in reliability. The resulting consequences regarding the importance of relative ranking positions are analyzed, the top three alternatives exhibiting a generally contained decrease in their CTRs that contrasts with the cumulative pattern arising from the fourth position onwards.