Document (#40072)

Author

Zhitomirsky-Geffet, M.
Bar-Ilan, J.
Levene, M.

Title

Testing the stability of "wisdom of crowds" judgments of search results over time and their similarity with the search engine rankings

Source

Aslib journal of information management. 68(2016) no.4, S.407-427

Year

2016

Abstract

Purpose - One of the under-explored aspects in the process of user information seeking behaviour is influence of time on relevance evaluation. It has been shown in previous studies that individual users might change their assessment of search results over time. It is also known that aggregated judgements of multiple individual users can lead to correct and reliable decisions; this phenomenon is known as the "wisdom of crowds". The purpose of this paper is to examine whether aggregated judgements will be more stable and thus more reliable over time than individual user judgements. Design/methodology/approach - In this study two simple measures are proposed to calculate the aggregated judgements of search results and compare their reliability and stability to individual user judgements. In addition, the aggregated "wisdom of crowds" judgements were used as a means to compare the differences between human assessments of search results and search engine's rankings. A large-scale user study was conducted with 87 participants who evaluated two different queries and four diverse result sets twice, with an interval of two months. Two types of judgements were considered in this study: relevance on a four-point scale, and ranking on a ten-point scale without ties. Findings - It was found that aggregated judgements are much more stable than individual user judgements, yet they are quite different from search engine rankings. Practical implications - The proposed "wisdom of crowds"-based approach provides a reliable reference point for the evaluation of search engines. This is also important for exploring the need of personalisation and adapting search engine's ranking over time to changes in users preferences. Originality/value - This is a first study that applies the notion of "wisdom of crowds" to examine an under-explored in the literature phenomenon of "change in time" in user evaluation of relevance.

Content

Vgl.: http://www.emeraldinsight.com/doi/abs/10.1108/AJIM-10-2015-0165.

Similar documents (author)

1. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Analysis of change in users' assessment of search results over time (2017) 5.50

   5.5045137 = sum of:
     5.5045137 = sum of:
       1.0581977 = weight(author_txt:ilan in 4593) [ClassicSimilarity], result of:
         1.0581977 = score(doc=4593,freq=1.0), product of:
           0.41854176 = queryWeight, product of:
             8.090549 = idf(docFreq=36, maxDocs=44421)
             0.05173218 = queryNorm
           2.5282967 = fieldWeight in 4593, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.090549 = idf(docFreq=36, maxDocs=44421)
             0.3125 = fieldNorm(doc=4593)
       1.3936292 = weight(author_txt:geffet in 4593) [ClassicSimilarity], result of:
         1.3936292 = score(doc=4593,freq=1.0), product of:
           0.50287384 = queryWeight, product of:
             1.0961251 = boost
             8.868255 = idf(docFreq=16, maxDocs=44421)
             0.05173218 = queryNorm
           2.7713296 = fieldWeight in 4593, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.868255 = idf(docFreq=16, maxDocs=44421)
             0.3125 = fieldNorm(doc=4593)
       1.3936292 = weight(author_txt:zhitomirsky in 4593) [ClassicSimilarity], result of:
         1.3936292 = score(doc=4593,freq=1.0), product of:
           0.50287384 = queryWeight, product of:
             1.0961251 = boost
             8.868255 = idf(docFreq=16, maxDocs=44421)
             0.05173218 = queryNorm
           2.7713296 = fieldWeight in 4593, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.868255 = idf(docFreq=16, maxDocs=44421)
             0.3125 = fieldNorm(doc=4593)
       1.6590577 = weight(author_txt:levene in 4593) [ClassicSimilarity], result of:
         1.6590577 = score(doc=4593,freq=1.0), product of:
           0.5648527 = queryWeight, product of:
             1.1617113 = boost
             9.398883 = idf(docFreq=9, maxDocs=44421)
             0.05173218 = queryNorm
           2.937151 = fieldWeight in 4593, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.398883 = idf(docFreq=9, maxDocs=44421)
             0.3125 = fieldNorm(doc=4593)
   

2. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Categorical relevance judgment (2018) 5.50

   5.5045137 = sum of:
     5.5045137 = sum of:
       1.0581977 = weight(author_txt:ilan in 457) [ClassicSimilarity], result of:
         1.0581977 = score(doc=457,freq=1.0), product of:
           0.41854176 = queryWeight, product of:
             8.090549 = idf(docFreq=36, maxDocs=44421)
             0.05173218 = queryNorm
           2.5282967 = fieldWeight in 457, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.090549 = idf(docFreq=36, maxDocs=44421)
             0.3125 = fieldNorm(doc=457)
       1.3936292 = weight(author_txt:geffet in 457) [ClassicSimilarity], result of:
         1.3936292 = score(doc=457,freq=1.0), product of:
           0.50287384 = queryWeight, product of:
             1.0961251 = boost
             8.868255 = idf(docFreq=16, maxDocs=44421)
             0.05173218 = queryNorm
           2.7713296 = fieldWeight in 457, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.868255 = idf(docFreq=16, maxDocs=44421)
             0.3125 = fieldNorm(doc=457)
       1.3936292 = weight(author_txt:zhitomirsky in 457) [ClassicSimilarity], result of:
         1.3936292 = score(doc=457,freq=1.0), product of:
           0.50287384 = queryWeight, product of:
             1.0961251 = boost
             8.868255 = idf(docFreq=16, maxDocs=44421)
             0.05173218 = queryNorm
           2.7713296 = fieldWeight in 457, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.868255 = idf(docFreq=16, maxDocs=44421)
             0.3125 = fieldNorm(doc=457)
       1.6590577 = weight(author_txt:levene in 457) [ClassicSimilarity], result of:
         1.6590577 = score(doc=457,freq=1.0), product of:
           0.5648527 = queryWeight, product of:
             1.1617113 = boost
             9.398883 = idf(docFreq=9, maxDocs=44421)
             0.05173218 = queryNorm
           2.937151 = fieldWeight in 457, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.398883 = idf(docFreq=9, maxDocs=44421)
             0.3125 = fieldNorm(doc=457)
   

3. Zhitomirsky-Geffet, M.; Bar-Ilan, J.: Towards maximal unification of semantically diverse ontologies for controversial domains (2014) 3.46

   3.46091 = sum of:
     3.46091 = product of:
       4.614547 = sum of:
         1.2698373 = weight(author_txt:ilan in 2634) [ClassicSimilarity], result of:
           1.2698373 = score(doc=2634,freq=1.0), product of:
             0.41854176 = queryWeight, product of:
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.05173218 = queryNorm
             3.033956 = fieldWeight in 2634, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.375 = fieldNorm(doc=2634)
         1.6723549 = weight(author_txt:geffet in 2634) [ClassicSimilarity], result of:
           1.6723549 = score(doc=2634,freq=1.0), product of:
             0.50287384 = queryWeight, product of:
               1.0961251 = boost
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.05173218 = queryNorm
             3.3255954 = fieldWeight in 2634, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.375 = fieldNorm(doc=2634)
         1.6723549 = weight(author_txt:zhitomirsky in 2634) [ClassicSimilarity], result of:
           1.6723549 = score(doc=2634,freq=1.0), product of:
             0.50287384 = queryWeight, product of:
               1.0961251 = boost
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.05173218 = queryNorm
             3.3255954 = fieldWeight in 2634, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.375 = fieldNorm(doc=2634)
       0.75 = coord(3/4)
   

4. Bar-Ilan, J.; Zhitomirsky-Geffet, M.; Miller, Y.; Shoham, S.: ¬The effects of background information and social interaction on image tagging (2010) 2.88

   2.884092 = sum of:
     2.884092 = product of:
       3.8454561 = sum of:
         1.0581977 = weight(author_txt:ilan in 440) [ClassicSimilarity], result of:
           1.0581977 = score(doc=440,freq=1.0), product of:
             0.41854176 = queryWeight, product of:
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.05173218 = queryNorm
             2.5282967 = fieldWeight in 440, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.3125 = fieldNorm(doc=440)
         1.3936292 = weight(author_txt:geffet in 440) [ClassicSimilarity], result of:
           1.3936292 = score(doc=440,freq=1.0), product of:
             0.50287384 = queryWeight, product of:
               1.0961251 = boost
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.05173218 = queryNorm
             2.7713296 = fieldWeight in 440, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.3125 = fieldNorm(doc=440)
         1.3936292 = weight(author_txt:zhitomirsky in 440) [ClassicSimilarity], result of:
           1.3936292 = score(doc=440,freq=1.0), product of:
             0.50287384 = queryWeight, product of:
               1.0961251 = boost
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.05173218 = queryNorm
             2.7713296 = fieldWeight in 440, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.3125 = fieldNorm(doc=440)
       0.75 = coord(3/4)
   

5. Zhitomirsky-Geffet, M.; Erez, E.S.; Bar-Ilan, J.: Toward multiviewpoint ontology construction by collaboration of non-experts and crowdsourcing : the case of the effect of diet on health (2017) 2.88

   2.884092 = sum of:
     2.884092 = product of:
       3.8454561 = sum of:
         1.0581977 = weight(author_txt:ilan in 4439) [ClassicSimilarity], result of:
           1.0581977 = score(doc=4439,freq=1.0), product of:
             0.41854176 = queryWeight, product of:
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.05173218 = queryNorm
             2.5282967 = fieldWeight in 4439, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.3125 = fieldNorm(doc=4439)
         1.3936292 = weight(author_txt:geffet in 4439) [ClassicSimilarity], result of:
           1.3936292 = score(doc=4439,freq=1.0), product of:
             0.50287384 = queryWeight, product of:
               1.0961251 = boost
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.05173218 = queryNorm
             2.7713296 = fieldWeight in 4439, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.3125 = fieldNorm(doc=4439)
         1.3936292 = weight(author_txt:zhitomirsky in 4439) [ClassicSimilarity], result of:
           1.3936292 = score(doc=4439,freq=1.0), product of:
             0.50287384 = queryWeight, product of:
               1.0961251 = boost
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.05173218 = queryNorm
             2.7713296 = fieldWeight in 4439, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.3125 = fieldNorm(doc=4439)
       0.75 = coord(3/4)
   

Similar documents (content)

1. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Analysis of change in users' assessment of search results over time (2017) 0.59

   0.5924231 = sum of:
     0.5924231 = product of:
       1.0578984 = sum of:
         0.03422767 = weight(abstract_txt:ranking in 4593) [ClassicSimilarity], result of:
           0.03422767 = score(doc=4593,freq=2.0), product of:
             0.069191426 = queryWeight, product of:
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.012362953 = queryNorm
             0.4946808 = fieldWeight in 4593, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.018616175 = weight(abstract_txt:evaluation in 4593) [ClassicSimilarity], result of:
           0.018616175 = score(doc=4593,freq=1.0), product of:
             0.066491805 = queryWeight, product of:
               1.2006145 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.012362953 = queryNorm
             0.279977 = fieldWeight in 4593, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.021997834 = weight(abstract_txt:study in 4593) [ClassicSimilarity], result of:
           0.021997834 = score(doc=4593,freq=4.0), product of:
             0.05152927 = queryWeight, product of:
               1.2204384 = boost
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.012362953 = queryNorm
             0.42689976 = fieldWeight in 4593, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.025990685 = weight(abstract_txt:results in 4593) [ClassicSimilarity], result of:
           0.025990685 = score(doc=4593,freq=5.0), product of:
             0.05346167 = queryWeight, product of:
               1.2431116 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.012362953 = queryNorm
             0.4861555 = fieldWeight in 4593, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.009994661 = weight(abstract_txt:this in 4593) [ClassicSimilarity], result of:
           0.009994661 = score(doc=4593,freq=3.0), product of:
             0.038369793 = queryWeight, product of:
               1.2898203 = boost
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.012362953 = queryNorm
             0.26048255 = fieldWeight in 4593, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.049614724 = weight(abstract_txt:relevance in 4593) [ClassicSimilarity], result of:
           0.049614724 = score(doc=4593,freq=4.0), product of:
             0.08051835 = queryWeight, product of:
               1.321195 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.012362953 = queryNorm
             0.6161915 = fieldWeight in 4593, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.041291796 = weight(abstract_txt:point in 4593) [ClassicSimilarity], result of:
           0.041291796 = score(doc=4593,freq=2.0), product of:
             0.0897581 = queryWeight, product of:
               1.3949425 = boost
               5.204693 = idf(docFreq=662, maxDocs=44421)
               0.012362953 = queryNorm
             0.4600342 = fieldWeight in 4593, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.204693 = idf(docFreq=662, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.058674917 = weight(abstract_txt:scale in 4593) [ClassicSimilarity], result of:
           0.058674917 = score(doc=4593,freq=3.0), product of:
             0.09910656 = queryWeight, product of:
               1.4657865 = boost
               5.4690194 = idf(docFreq=508, maxDocs=44421)
               0.012362953 = queryNorm
             0.5920387 = fieldWeight in 4593, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.4690194 = idf(docFreq=508, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.021087771 = weight(abstract_txt:over in 4593) [ClassicSimilarity], result of:
           0.021087771 = score(doc=4593,freq=1.0), product of:
             0.07952576 = queryWeight, product of:
               1.5161521 = boost
               4.242705 = idf(docFreq=1734, maxDocs=44421)
               0.012362953 = queryNorm
             0.26516905 = fieldWeight in 4593, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.242705 = idf(docFreq=1734, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.0672406 = weight(abstract_txt:rankings in 4593) [ClassicSimilarity], result of:
           0.0672406 = score(doc=4593,freq=1.0), product of:
             0.15652926 = queryWeight, product of:
               1.8421172 = boost
               6.8731537 = idf(docFreq=124, maxDocs=44421)
               0.012362953 = queryNorm
             0.4295721 = fieldWeight in 4593, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.8731537 = idf(docFreq=124, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.046188015 = weight(abstract_txt:user in 4593) [ClassicSimilarity], result of:
           0.046188015 = score(doc=4593,freq=5.0), product of:
             0.08978711 = queryWeight, product of:
               1.9730655 = boost
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.012362953 = queryNorm
             0.514417 = fieldWeight in 4593, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.041825995 = weight(abstract_txt:time in 4593) [ClassicSimilarity], result of:
           0.041825995 = score(doc=4593,freq=2.0), product of:
             0.114061385 = queryWeight, product of:
               2.2238414 = boost
               4.1487055 = idf(docFreq=1905, maxDocs=44421)
               0.012362953 = queryNorm
             0.36669722 = fieldWeight in 4593, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1487055 = idf(docFreq=1905, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.052524768 = weight(abstract_txt:search in 4593) [ClassicSimilarity], result of:
           0.052524768 = score(doc=4593,freq=3.0), product of:
             0.13276508 = queryWeight, product of:
               2.9384754 = boost
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.012362953 = queryNorm
             0.39562184 = fieldWeight in 4593, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.5686228 = weight(abstract_txt:judgements in 4593) [ClassicSimilarity], result of:
           0.5686228 = score(doc=4593,freq=5.0), product of:
             0.5479922 = queryWeight, product of:
               5.9699035 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.012362953 = queryNorm
             1.0376476 = fieldWeight in 4593, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
       0.56 = coord(14/25)
   

2. Tang, R.; Solomon, P.: Toward an understanding of the dynamics of relevance judgement : an analysis of one person's search behavior (1998) 0.21

   0.21106055 = sum of:
     0.21106055 = product of:
       0.7537877 = sum of:
         0.028576022 = weight(abstract_txt:study in 4268) [ClassicSimilarity], result of:
           0.028576022 = score(doc=4268,freq=3.0), product of:
             0.05152927 = queryWeight, product of:
               1.2204384 = boost
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.012362953 = queryNorm
             0.55455905 = fieldWeight in 4268, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.09375 = fieldNorm(doc=4268)
         0.017435081 = weight(abstract_txt:results in 4268) [ClassicSimilarity], result of:
           0.017435081 = score(doc=4268,freq=1.0), product of:
             0.05346167 = queryWeight, product of:
               1.2431116 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.012362953 = queryNorm
             0.32612303 = fieldWeight in 4268, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.09375 = fieldNorm(doc=4268)
         0.008655631 = weight(abstract_txt:this in 4268) [ClassicSimilarity], result of:
           0.008655631 = score(doc=4268,freq=1.0), product of:
             0.038369793 = queryWeight, product of:
               1.2898203 = boost
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.012362953 = queryNorm
             0.2255845 = fieldWeight in 4268, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.09375 = fieldNorm(doc=4268)
         0.08320642 = weight(abstract_txt:relevance in 4268) [ClassicSimilarity], result of:
           0.08320642 = score(doc=4268,freq=5.0), product of:
             0.08051835 = queryWeight, product of:
               1.321195 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.012362953 = queryNorm
             1.0333846 = fieldWeight in 4268, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.09375 = fieldNorm(doc=4268)
         0.030983863 = weight(abstract_txt:user in 4268) [ClassicSimilarity], result of:
           0.030983863 = score(doc=4268,freq=1.0), product of:
             0.08978711 = queryWeight, product of:
               1.9730655 = boost
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.012362953 = queryNorm
             0.34508142 = fieldWeight in 4268, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.09375 = fieldNorm(doc=4268)
         0.04548778 = weight(abstract_txt:search in 4268) [ClassicSimilarity], result of:
           0.04548778 = score(doc=4268,freq=1.0), product of:
             0.13276508 = queryWeight, product of:
               2.9384754 = boost
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.012362953 = queryNorm
             0.34261855 = fieldWeight in 4268, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.09375 = fieldNorm(doc=4268)
         0.5394429 = weight(abstract_txt:judgements in 4268) [ClassicSimilarity], result of:
           0.5394429 = score(doc=4268,freq=2.0), product of:
             0.5479922 = queryWeight, product of:
               5.9699035 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.012362953 = queryNorm
             0.98439884 = fieldWeight in 4268, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.09375 = fieldNorm(doc=4268)
       0.28 = coord(7/25)
   

3. Su, L.T.: ¬A comprehensive and systematic model of user evaluation of Web search engines : Il. An evaluation by undergraduates (2003) 0.21

   0.20966044 = sum of:
     0.20966044 = product of:
       0.5241511 = sum of:
         0.032578308 = weight(abstract_txt:evaluation in 3117) [ClassicSimilarity], result of:
           0.032578308 = score(doc=3117,freq=4.0), product of:
             0.066491805 = queryWeight, product of:
               1.2006145 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.012362953 = queryNorm
             0.48995975 = fieldWeight in 3117, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3117)
         0.016669346 = weight(abstract_txt:study in 3117) [ClassicSimilarity], result of:
           0.016669346 = score(doc=3117,freq=3.0), product of:
             0.05152927 = queryWeight, product of:
               1.2204384 = boost
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.012362953 = queryNorm
             0.32349277 = fieldWeight in 3117, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3117)
         0.01761576 = weight(abstract_txt:results in 3117) [ClassicSimilarity], result of:
           0.01761576 = score(doc=3117,freq=3.0), product of:
             0.05346167 = queryWeight, product of:
               1.2431116 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.012362953 = queryNorm
             0.3295026 = fieldWeight in 3117, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3117)
         0.005049118 = weight(abstract_txt:this in 3117) [ClassicSimilarity], result of:
           0.005049118 = score(doc=3117,freq=1.0), product of:
             0.038369793 = queryWeight, product of:
               1.2898203 = boost
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.012362953 = queryNorm
             0.13159096 = fieldWeight in 3117, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3117)
         0.030697541 = weight(abstract_txt:relevance in 3117) [ClassicSimilarity], result of:
           0.030697541 = score(doc=3117,freq=2.0), product of:
             0.08051835 = queryWeight, product of:
               1.321195 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.012362953 = queryNorm
             0.381249 = fieldWeight in 3117, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3117)
         0.047819093 = weight(abstract_txt:user in 3117) [ClassicSimilarity], result of:
           0.047819093 = score(doc=3117,freq=7.0), product of:
             0.08978711 = queryWeight, product of:
               1.9730655 = boost
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.012362953 = queryNorm
             0.53258306 = fieldWeight in 3117, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3117)
         0.035011716 = weight(abstract_txt:individual in 3117) [ClassicSimilarity], result of:
           0.035011716 = score(doc=3117,freq=1.0), product of:
             0.13129885 = queryWeight, product of:
               2.1780825 = boost
               4.8760076 = idf(docFreq=920, maxDocs=44421)
               0.012362953 = queryNorm
             0.26665667 = fieldWeight in 3117, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.8760076 = idf(docFreq=920, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3117)
         0.036597744 = weight(abstract_txt:time in 3117) [ClassicSimilarity], result of:
           0.036597744 = score(doc=3117,freq=2.0), product of:
             0.114061385 = queryWeight, product of:
               2.2238414 = boost
               4.1487055 = idf(docFreq=1905, maxDocs=44421)
               0.012362953 = queryNorm
             0.32086006 = fieldWeight in 3117, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1487055 = idf(docFreq=1905, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3117)
         0.07960362 = weight(abstract_txt:search in 3117) [ClassicSimilarity], result of:
           0.07960362 = score(doc=3117,freq=9.0), product of:
             0.13276508 = queryWeight, product of:
               2.9384754 = boost
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.012362953 = queryNorm
             0.5995825 = fieldWeight in 3117, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3117)
         0.22250886 = weight(abstract_txt:judgements in 3117) [ClassicSimilarity], result of:
           0.22250886 = score(doc=3117,freq=1.0), product of:
             0.5479922 = queryWeight, product of:
               5.9699035 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.012362953 = queryNorm
             0.40604383 = fieldWeight in 3117, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3117)
       0.4 = coord(10/25)
   

4. Dennis, S.; Bruza, P.; McArthur, R.: Web searching : a process-oriented experimental study of three interactive search paradigms (2002) 0.20

   0.19538039 = sum of:
     0.19538039 = product of:
       0.5427233 = sum of:
         0.010998917 = weight(abstract_txt:study in 1200) [ClassicSimilarity], result of:
           0.010998917 = score(doc=1200,freq=1.0), product of:
             0.05152927 = queryWeight, product of:
               1.2204384 = boost
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.012362953 = queryNorm
             0.21344988 = fieldWeight in 1200, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.0625 = fieldNorm(doc=1200)
         0.011623388 = weight(abstract_txt:results in 1200) [ClassicSimilarity], result of:
           0.011623388 = score(doc=1200,freq=1.0), product of:
             0.05346167 = queryWeight, product of:
               1.2431116 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.012362953 = queryNorm
             0.21741535 = fieldWeight in 1200, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.0625 = fieldNorm(doc=1200)
         0.008160607 = weight(abstract_txt:this in 1200) [ClassicSimilarity], result of:
           0.008160607 = score(doc=1200,freq=2.0), product of:
             0.038369793 = queryWeight, product of:
               1.2898203 = boost
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.012362953 = queryNorm
             0.21268311 = fieldWeight in 1200, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.0625 = fieldNorm(doc=1200)
         0.049614724 = weight(abstract_txt:relevance in 1200) [ClassicSimilarity], result of:
           0.049614724 = score(doc=1200,freq=4.0), product of:
             0.08051835 = queryWeight, product of:
               1.321195 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.012362953 = queryNorm
             0.6161915 = fieldWeight in 1200, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=1200)
         0.021087771 = weight(abstract_txt:over in 1200) [ClassicSimilarity], result of:
           0.021087771 = score(doc=1200,freq=1.0), product of:
             0.07952576 = queryWeight, product of:
               1.5161521 = boost
               4.242705 = idf(docFreq=1734, maxDocs=44421)
               0.012362953 = queryNorm
             0.26516905 = fieldWeight in 1200, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.242705 = idf(docFreq=1734, maxDocs=44421)
               0.0625 = fieldNorm(doc=1200)
         0.035777085 = weight(abstract_txt:user in 1200) [ClassicSimilarity], result of:
           0.035777085 = score(doc=1200,freq=3.0), product of:
             0.08978711 = queryWeight, product of:
               1.9730655 = boost
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.012362953 = queryNorm
             0.3984657 = fieldWeight in 1200, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.0625 = fieldNorm(doc=1200)
         0.041825995 = weight(abstract_txt:time in 1200) [ClassicSimilarity], result of:
           0.041825995 = score(doc=1200,freq=2.0), product of:
             0.114061385 = queryWeight, product of:
               2.2238414 = boost
               4.1487055 = idf(docFreq=1905, maxDocs=44421)
               0.012362953 = queryNorm
             0.36669722 = fieldWeight in 1200, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1487055 = idf(docFreq=1905, maxDocs=44421)
               0.0625 = fieldNorm(doc=1200)
         0.10933902 = weight(abstract_txt:search in 1200) [ClassicSimilarity], result of:
           0.10933902 = score(doc=1200,freq=13.0), product of:
             0.13276508 = queryWeight, product of:
               2.9384754 = boost
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.012362953 = queryNorm
             0.82355255 = fieldWeight in 1200, product of:
               3.6055512 = tf(freq=13.0), with freq of:
                 13.0 = termFreq=13.0
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.0625 = fieldNorm(doc=1200)
         0.25429583 = weight(abstract_txt:judgements in 1200) [ClassicSimilarity], result of:
           0.25429583 = score(doc=1200,freq=1.0), product of:
             0.5479922 = queryWeight, product of:
               5.9699035 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.012362953 = queryNorm
             0.46405008 = fieldWeight in 1200, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.0625 = fieldNorm(doc=1200)
       0.36 = coord(9/25)
   

5. Bar-Ilan, J.; Levene, M.; Mat-Hassan, M.: Methods for evaluating dynamic changes in search engine rankings : a case study (2006) 0.19

   0.1934947 = sum of:
     0.1934947 = product of:
       0.48373675 = sum of:
         0.02994921 = weight(abstract_txt:ranking in 741) [ClassicSimilarity], result of:
           0.02994921 = score(doc=741,freq=2.0), product of:
             0.069191426 = queryWeight, product of:
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.012362953 = queryNorm
             0.43284568 = fieldWeight in 741, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.0546875 = fieldNorm(doc=741)
         0.030166185 = weight(abstract_txt:compare in 741) [ClassicSimilarity], result of:
           0.030166185 = score(doc=741,freq=2.0), product of:
             0.069525205 = queryWeight, product of:
               1.0024091 = boost
               5.6101575 = idf(docFreq=441, maxDocs=44421)
               0.012362953 = queryNorm
             0.43388847 = fieldWeight in 741, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.6101575 = idf(docFreq=441, maxDocs=44421)
               0.0546875 = fieldNorm(doc=741)
         0.026908519 = weight(abstract_txt:results in 741) [ClassicSimilarity], result of:
           0.026908519 = score(doc=741,freq=7.0), product of:
             0.05346167 = queryWeight, product of:
               1.2431116 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.012362953 = queryNorm
             0.50332355 = fieldWeight in 741, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.0546875 = fieldNorm(doc=741)
         0.044507384 = weight(abstract_txt:stable in 741) [ClassicSimilarity], result of:
           0.044507384 = score(doc=741,freq=1.0), product of:
             0.113525495 = queryWeight, product of:
               1.2809157 = boost
               7.168868 = idf(docFreq=92, maxDocs=44421)
               0.012362953 = queryNorm
             0.39204746 = fieldWeight in 741, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.168868 = idf(docFreq=92, maxDocs=44421)
               0.0546875 = fieldNorm(doc=741)
         0.007140531 = weight(abstract_txt:this in 741) [ClassicSimilarity], result of:
           0.007140531 = score(doc=741,freq=2.0), product of:
             0.038369793 = queryWeight, product of:
               1.2898203 = boost
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.012362953 = queryNorm
             0.18609773 = fieldWeight in 741, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.0546875 = fieldNorm(doc=741)
         0.025547994 = weight(abstract_txt:point in 741) [ClassicSimilarity], result of:
           0.025547994 = score(doc=741,freq=1.0), product of:
             0.0897581 = queryWeight, product of:
               1.3949425 = boost
               5.204693 = idf(docFreq=662, maxDocs=44421)
               0.012362953 = queryNorm
             0.28463164 = fieldWeight in 741, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.204693 = idf(docFreq=662, maxDocs=44421)
               0.0546875 = fieldNorm(doc=741)
         0.0369036 = weight(abstract_txt:over in 741) [ClassicSimilarity], result of:
           0.0369036 = score(doc=741,freq=4.0), product of:
             0.07952576 = queryWeight, product of:
               1.5161521 = boost
               4.242705 = idf(docFreq=1734, maxDocs=44421)
               0.012362953 = queryNorm
             0.46404585 = fieldWeight in 741, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.242705 = idf(docFreq=1734, maxDocs=44421)
               0.0546875 = fieldNorm(doc=741)
         0.166412 = weight(abstract_txt:rankings in 741) [ClassicSimilarity], result of:
           0.166412 = score(doc=741,freq=8.0), product of:
             0.15652926 = queryWeight, product of:
               1.8421172 = boost
               6.8731537 = idf(docFreq=124, maxDocs=44421)
               0.012362953 = queryNorm
             1.0631367 = fieldWeight in 741, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               6.8731537 = idf(docFreq=124, maxDocs=44421)
               0.0546875 = fieldNorm(doc=741)
         0.036597744 = weight(abstract_txt:time in 741) [ClassicSimilarity], result of:
           0.036597744 = score(doc=741,freq=2.0), product of:
             0.114061385 = queryWeight, product of:
               2.2238414 = boost
               4.1487055 = idf(docFreq=1905, maxDocs=44421)
               0.012362953 = queryNorm
             0.32086006 = fieldWeight in 741, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1487055 = idf(docFreq=1905, maxDocs=44421)
               0.0546875 = fieldNorm(doc=741)
         0.07960362 = weight(abstract_txt:search in 741) [ClassicSimilarity], result of:
           0.07960362 = score(doc=741,freq=9.0), product of:
             0.13276508 = queryWeight, product of:
               2.9384754 = boost
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.012362953 = queryNorm
             0.5995825 = fieldWeight in 741, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.0546875 = fieldNorm(doc=741)
       0.4 = coord(10/25)