Document (#41458)

Author

Zhitomirsky-Geffet, M.
Bar-Ilan, J.
Levene, M.

Title

Categorical relevance judgment

Source

Journal of the Association for Information Science and Technology. 69(2018) no.9, S.1084-1094

Year

2018

Abstract

In this study we aim to explore users' behavior when assessing search results relevance based on the hypothesis of categorical thinking. To investigate how users categories search engine results, we perform several experiments where users are asked to group a list of 20 search results into several categories, while attaching a relevance judgment to each formed category. Moreover, to determine how users change their minds over time, each experiment was repeated three times under the same conditions, with a gap of one month between rounds. The results show that on average users form 4-5 categories. Within each round the size of a category decreases with the relevance of a category. To measure the agreement between the search engine's ranking and the users' relevance judgments, we defined two novel similarity measures, the average concordance and the MinMax swap ratio. Similarity is shown to be the highest for the third round as the users' opinion stabilizes. Qualitative analysis uncovered some interesting points that users tended to categories results by type and reliability of their source, and particularly, found commercial sites less trustworthy, and attached high relevance to Wikipedia when their prior domain knowledge was limited.

Content

Vgl.: https://onlinelibrary.wiley.com/doi/10.1002/asi.24035.

Similar documents (author)

1. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Testing the stability of "wisdom of crowds" judgments of search results over time and their similarity with the search engine rankings (2016) 5.50

   5.5045137 = sum of:
     5.5045137 = sum of:
       1.0581977 = weight(author_txt:ilan in 4071) [ClassicSimilarity], result of:
         1.0581977 = score(doc=4071,freq=1.0), product of:
           0.41854176 = queryWeight, product of:
             8.090549 = idf(docFreq=36, maxDocs=44421)
             0.05173218 = queryNorm
           2.5282967 = fieldWeight in 4071, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.090549 = idf(docFreq=36, maxDocs=44421)
             0.3125 = fieldNorm(doc=4071)
       1.3936292 = weight(author_txt:geffet in 4071) [ClassicSimilarity], result of:
         1.3936292 = score(doc=4071,freq=1.0), product of:
           0.50287384 = queryWeight, product of:
             1.0961251 = boost
             8.868255 = idf(docFreq=16, maxDocs=44421)
             0.05173218 = queryNorm
           2.7713296 = fieldWeight in 4071, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.868255 = idf(docFreq=16, maxDocs=44421)
             0.3125 = fieldNorm(doc=4071)
       1.3936292 = weight(author_txt:zhitomirsky in 4071) [ClassicSimilarity], result of:
         1.3936292 = score(doc=4071,freq=1.0), product of:
           0.50287384 = queryWeight, product of:
             1.0961251 = boost
             8.868255 = idf(docFreq=16, maxDocs=44421)
             0.05173218 = queryNorm
           2.7713296 = fieldWeight in 4071, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.868255 = idf(docFreq=16, maxDocs=44421)
             0.3125 = fieldNorm(doc=4071)
       1.6590577 = weight(author_txt:levene in 4071) [ClassicSimilarity], result of:
         1.6590577 = score(doc=4071,freq=1.0), product of:
           0.5648527 = queryWeight, product of:
             1.1617113 = boost
             9.398883 = idf(docFreq=9, maxDocs=44421)
             0.05173218 = queryNorm
           2.937151 = fieldWeight in 4071, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.398883 = idf(docFreq=9, maxDocs=44421)
             0.3125 = fieldNorm(doc=4071)
   

2. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Analysis of change in users' assessment of search results over time (2017) 5.50

   5.5045137 = sum of:
     5.5045137 = sum of:
       1.0581977 = weight(author_txt:ilan in 4593) [ClassicSimilarity], result of:
         1.0581977 = score(doc=4593,freq=1.0), product of:
           0.41854176 = queryWeight, product of:
             8.090549 = idf(docFreq=36, maxDocs=44421)
             0.05173218 = queryNorm
           2.5282967 = fieldWeight in 4593, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.090549 = idf(docFreq=36, maxDocs=44421)
             0.3125 = fieldNorm(doc=4593)
       1.3936292 = weight(author_txt:geffet in 4593) [ClassicSimilarity], result of:
         1.3936292 = score(doc=4593,freq=1.0), product of:
           0.50287384 = queryWeight, product of:
             1.0961251 = boost
             8.868255 = idf(docFreq=16, maxDocs=44421)
             0.05173218 = queryNorm
           2.7713296 = fieldWeight in 4593, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.868255 = idf(docFreq=16, maxDocs=44421)
             0.3125 = fieldNorm(doc=4593)
       1.3936292 = weight(author_txt:zhitomirsky in 4593) [ClassicSimilarity], result of:
         1.3936292 = score(doc=4593,freq=1.0), product of:
           0.50287384 = queryWeight, product of:
             1.0961251 = boost
             8.868255 = idf(docFreq=16, maxDocs=44421)
             0.05173218 = queryNorm
           2.7713296 = fieldWeight in 4593, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.868255 = idf(docFreq=16, maxDocs=44421)
             0.3125 = fieldNorm(doc=4593)
       1.6590577 = weight(author_txt:levene in 4593) [ClassicSimilarity], result of:
         1.6590577 = score(doc=4593,freq=1.0), product of:
           0.5648527 = queryWeight, product of:
             1.1617113 = boost
             9.398883 = idf(docFreq=9, maxDocs=44421)
             0.05173218 = queryNorm
           2.937151 = fieldWeight in 4593, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.398883 = idf(docFreq=9, maxDocs=44421)
             0.3125 = fieldNorm(doc=4593)
   

3. Zhitomirsky-Geffet, M.; Bar-Ilan, J.: Towards maximal unification of semantically diverse ontologies for controversial domains (2014) 3.46

   3.46091 = sum of:
     3.46091 = product of:
       4.614547 = sum of:
         1.2698373 = weight(author_txt:ilan in 2634) [ClassicSimilarity], result of:
           1.2698373 = score(doc=2634,freq=1.0), product of:
             0.41854176 = queryWeight, product of:
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.05173218 = queryNorm
             3.033956 = fieldWeight in 2634, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.375 = fieldNorm(doc=2634)
         1.6723549 = weight(author_txt:geffet in 2634) [ClassicSimilarity], result of:
           1.6723549 = score(doc=2634,freq=1.0), product of:
             0.50287384 = queryWeight, product of:
               1.0961251 = boost
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.05173218 = queryNorm
             3.3255954 = fieldWeight in 2634, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.375 = fieldNorm(doc=2634)
         1.6723549 = weight(author_txt:zhitomirsky in 2634) [ClassicSimilarity], result of:
           1.6723549 = score(doc=2634,freq=1.0), product of:
             0.50287384 = queryWeight, product of:
               1.0961251 = boost
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.05173218 = queryNorm
             3.3255954 = fieldWeight in 2634, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.375 = fieldNorm(doc=2634)
       0.75 = coord(3/4)
   

4. Bar-Ilan, J.; Zhitomirsky-Geffet, M.; Miller, Y.; Shoham, S.: ¬The effects of background information and social interaction on image tagging (2010) 2.88

   2.884092 = sum of:
     2.884092 = product of:
       3.8454561 = sum of:
         1.0581977 = weight(author_txt:ilan in 440) [ClassicSimilarity], result of:
           1.0581977 = score(doc=440,freq=1.0), product of:
             0.41854176 = queryWeight, product of:
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.05173218 = queryNorm
             2.5282967 = fieldWeight in 440, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.3125 = fieldNorm(doc=440)
         1.3936292 = weight(author_txt:geffet in 440) [ClassicSimilarity], result of:
           1.3936292 = score(doc=440,freq=1.0), product of:
             0.50287384 = queryWeight, product of:
               1.0961251 = boost
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.05173218 = queryNorm
             2.7713296 = fieldWeight in 440, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.3125 = fieldNorm(doc=440)
         1.3936292 = weight(author_txt:zhitomirsky in 440) [ClassicSimilarity], result of:
           1.3936292 = score(doc=440,freq=1.0), product of:
             0.50287384 = queryWeight, product of:
               1.0961251 = boost
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.05173218 = queryNorm
             2.7713296 = fieldWeight in 440, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.3125 = fieldNorm(doc=440)
       0.75 = coord(3/4)
   

5. Zhitomirsky-Geffet, M.; Erez, E.S.; Bar-Ilan, J.: Toward multiviewpoint ontology construction by collaboration of non-experts and crowdsourcing : the case of the effect of diet on health (2017) 2.88

   2.884092 = sum of:
     2.884092 = product of:
       3.8454561 = sum of:
         1.0581977 = weight(author_txt:ilan in 4439) [ClassicSimilarity], result of:
           1.0581977 = score(doc=4439,freq=1.0), product of:
             0.41854176 = queryWeight, product of:
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.05173218 = queryNorm
             2.5282967 = fieldWeight in 4439, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.3125 = fieldNorm(doc=4439)
         1.3936292 = weight(author_txt:geffet in 4439) [ClassicSimilarity], result of:
           1.3936292 = score(doc=4439,freq=1.0), product of:
             0.50287384 = queryWeight, product of:
               1.0961251 = boost
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.05173218 = queryNorm
             2.7713296 = fieldWeight in 4439, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.3125 = fieldNorm(doc=4439)
         1.3936292 = weight(author_txt:zhitomirsky in 4439) [ClassicSimilarity], result of:
           1.3936292 = score(doc=4439,freq=1.0), product of:
             0.50287384 = queryWeight, product of:
               1.0961251 = boost
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.05173218 = queryNorm
             2.7713296 = fieldWeight in 4439, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.868255 = idf(docFreq=16, maxDocs=44421)
               0.3125 = fieldNorm(doc=4439)
       0.75 = coord(3/4)
   

Similar documents (content)

1. Khoo, C.S.G.; Ng, K.; Ou, S.: ¬An exploratory study of human clustering of Web pages (2003) 0.26

   0.26291746 = sum of:
     0.26291746 = product of:
       0.6572936 = sum of:
         0.020523606 = weight(abstract_txt:several in 3741) [ClassicSimilarity], result of:
           0.020523606 = score(doc=3741,freq=1.0), product of:
             0.096261404 = queryWeight, product of:
               1.167841 = boost
               4.548416 = idf(docFreq=1277, maxDocs=44421)
               0.01812209 = queryNorm
             0.213207 = fieldWeight in 3741, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.548416 = idf(docFreq=1277, maxDocs=44421)
               0.046875 = fieldNorm(doc=3741)
         0.014494339 = weight(abstract_txt:their in 3741) [ClassicSimilarity], result of:
           0.014494339 = score(doc=3741,freq=2.0), product of:
             0.06935894 = queryWeight, product of:
               1.2141 = boost
               3.1523883 = idf(docFreq=5161, maxDocs=44421)
               0.01812209 = queryNorm
             0.20897579 = fieldWeight in 3741, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.1523883 = idf(docFreq=5161, maxDocs=44421)
               0.046875 = fieldNorm(doc=3741)
         0.042956285 = weight(abstract_txt:similarity in 3741) [ClassicSimilarity], result of:
           0.042956285 = score(doc=3741,freq=1.0), product of:
             0.15750737 = queryWeight, product of:
               1.4938537 = boost
               5.8181453 = idf(docFreq=358, maxDocs=44421)
               0.01812209 = queryNorm
             0.27272555 = fieldWeight in 3741, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.8181453 = idf(docFreq=358, maxDocs=44421)
               0.046875 = fieldNorm(doc=3741)
         0.06342617 = weight(abstract_txt:average in 3741) [ClassicSimilarity], result of:
           0.06342617 = score(doc=3741,freq=2.0), product of:
             0.16210088 = queryWeight, product of:
               1.5154803 = boost
               5.9023747 = idf(docFreq=329, maxDocs=44421)
               0.01812209 = queryNorm
             0.39127588 = fieldWeight in 3741, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.9023747 = idf(docFreq=329, maxDocs=44421)
               0.046875 = fieldNorm(doc=3741)
         0.03230331 = weight(abstract_txt:each in 3741) [ClassicSimilarity], result of:
           0.03230331 = score(doc=3741,freq=2.0), product of:
             0.118340895 = queryWeight, product of:
               1.5858812 = boost
               4.1177115 = idf(docFreq=1965, maxDocs=44421)
               0.01812209 = queryNorm
             0.2729683 = fieldWeight in 3741, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1177115 = idf(docFreq=1965, maxDocs=44421)
               0.046875 = fieldNorm(doc=3741)
         0.060223628 = weight(abstract_txt:search in 3741) [ClassicSimilarity], result of:
           0.060223628 = score(doc=3741,freq=8.0), product of:
             0.12429138 = queryWeight, product of:
               1.8766925 = boost
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.01812209 = queryNorm
             0.4845358 = fieldWeight in 3741, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.046875 = fieldNorm(doc=3741)
         0.032460757 = weight(abstract_txt:results in 3741) [ClassicSimilarity], result of:
           0.032460757 = score(doc=3741,freq=2.0), product of:
             0.14076419 = queryWeight, product of:
               2.2329228 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.01812209 = queryNorm
             0.23060378 = fieldWeight in 3741, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.046875 = fieldNorm(doc=3741)
         0.19149849 = weight(abstract_txt:categories in 3741) [ClassicSimilarity], result of:
           0.19149849 = score(doc=3741,freq=10.0), product of:
             0.24949966 = queryWeight, product of:
               2.658934 = boost
               5.177905 = idf(docFreq=680, maxDocs=44421)
               0.01812209 = queryNorm
             0.76753 = fieldWeight in 3741, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               5.177905 = idf(docFreq=680, maxDocs=44421)
               0.046875 = fieldNorm(doc=3741)
         0.08855938 = weight(abstract_txt:users in 3741) [ClassicSimilarity], result of:
           0.08855938 = score(doc=3741,freq=5.0), product of:
             0.23684786 = queryWeight, product of:
               3.6637206 = boost
               3.5672934 = idf(docFreq=3408, maxDocs=44421)
               0.01812209 = queryNorm
             0.3739083 = fieldWeight in 3741, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.5672934 = idf(docFreq=3408, maxDocs=44421)
               0.046875 = fieldNorm(doc=3741)
         0.11084763 = weight(abstract_txt:relevance in 3741) [ClassicSimilarity], result of:
           0.11084763 = score(doc=3741,freq=2.0), product of:
             0.33920673 = queryWeight, product of:
               3.7970862 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.01812209 = queryNorm
             0.32678488 = fieldWeight in 3741, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.046875 = fieldNorm(doc=3741)
       0.4 = coord(10/25)
   

2. Zhu, S.; Xie, X.; Ye, Z.; Ai, Q.; Liu, Y.: Comparing point-wise and pair-wise relevance judgment with brain signals (2024) 0.22

   0.22037348 = sum of:
     0.22037348 = product of:
       0.78704816 = sum of:
         0.027364807 = weight(abstract_txt:several in 2340) [ClassicSimilarity], result of:
           0.027364807 = score(doc=2340,freq=1.0), product of:
             0.096261404 = queryWeight, product of:
               1.167841 = boost
               4.548416 = idf(docFreq=1277, maxDocs=44421)
               0.01812209 = queryNorm
             0.284276 = fieldWeight in 2340, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.548416 = idf(docFreq=1277, maxDocs=44421)
               0.0625 = fieldNorm(doc=2340)
         0.013665394 = weight(abstract_txt:their in 2340) [ClassicSimilarity], result of:
           0.013665394 = score(doc=2340,freq=1.0), product of:
             0.06935894 = queryWeight, product of:
               1.2141 = boost
               3.1523883 = idf(docFreq=5161, maxDocs=44421)
               0.01812209 = queryNorm
             0.19702427 = fieldWeight in 2340, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.1523883 = idf(docFreq=5161, maxDocs=44421)
               0.0625 = fieldNorm(doc=2340)
         0.030455854 = weight(abstract_txt:each in 2340) [ClassicSimilarity], result of:
           0.030455854 = score(doc=2340,freq=1.0), product of:
             0.118340895 = queryWeight, product of:
               1.5858812 = boost
               4.1177115 = idf(docFreq=1965, maxDocs=44421)
               0.01812209 = queryNorm
             0.25735697 = fieldWeight in 2340, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1177115 = idf(docFreq=1965, maxDocs=44421)
               0.0625 = fieldNorm(doc=2340)
         0.049172387 = weight(abstract_txt:search in 2340) [ClassicSimilarity], result of:
           0.049172387 = score(doc=2340,freq=3.0), product of:
             0.12429138 = queryWeight, product of:
               1.8766925 = boost
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.01812209 = queryNorm
             0.39562184 = fieldWeight in 2340, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.0625 = fieldNorm(doc=2340)
         0.27933228 = weight(abstract_txt:judgment in 2340) [ClassicSimilarity], result of:
           0.27933228 = score(doc=2340,freq=6.0), product of:
             0.24928008 = queryWeight, product of:
               1.8793229 = boost
               7.319441 = idf(docFreq=79, maxDocs=44421)
               0.01812209 = queryNorm
             1.120556 = fieldWeight in 2340, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               7.319441 = idf(docFreq=79, maxDocs=44421)
               0.0625 = fieldNorm(doc=2340)
         0.09146374 = weight(abstract_txt:users in 2340) [ClassicSimilarity], result of:
           0.09146374 = score(doc=2340,freq=3.0), product of:
             0.23684786 = queryWeight, product of:
               3.6637206 = boost
               3.5672934 = idf(docFreq=3408, maxDocs=44421)
               0.01812209 = queryNorm
             0.38617083 = fieldWeight in 2340, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.5672934 = idf(docFreq=3408, maxDocs=44421)
               0.0625 = fieldNorm(doc=2340)
         0.29559368 = weight(abstract_txt:relevance in 2340) [ClassicSimilarity], result of:
           0.29559368 = score(doc=2340,freq=8.0), product of:
             0.33920673 = queryWeight, product of:
               3.7970862 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.01812209 = queryNorm
             0.87142634 = fieldWeight in 2340, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=2340)
       0.28 = coord(7/25)
   

3. Desai, M.; Spink, A.: ¬A algorithm to cluster documents based on relevance (2005) 0.18

   0.18480784 = sum of:
     0.18480784 = product of:
       0.660028 = sum of:
         0.029311744 = weight(abstract_txt:when in 2035) [ClassicSimilarity], result of:
           0.029311744 = score(doc=2035,freq=2.0), product of:
             0.07998497 = queryWeight, product of:
               1.0645396 = boost
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.01812209 = queryNorm
             0.36646566 = fieldWeight in 2035, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.0625 = fieldNorm(doc=2035)
         0.030455854 = weight(abstract_txt:each in 2035) [ClassicSimilarity], result of:
           0.030455854 = score(doc=2035,freq=1.0), product of:
             0.118340895 = queryWeight, product of:
               1.5858812 = boost
               4.1177115 = idf(docFreq=1965, maxDocs=44421)
               0.01812209 = queryNorm
             0.25735697 = fieldWeight in 2035, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1177115 = idf(docFreq=1965, maxDocs=44421)
               0.0625 = fieldNorm(doc=2035)
         0.05677938 = weight(abstract_txt:search in 2035) [ClassicSimilarity], result of:
           0.05677938 = score(doc=2035,freq=4.0), product of:
             0.12429138 = queryWeight, product of:
               1.8766925 = boost
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.01812209 = queryNorm
             0.45682475 = fieldWeight in 2035, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.0625 = fieldNorm(doc=2035)
         0.1750489 = weight(abstract_txt:categorical in 2035) [ClassicSimilarity], result of:
           0.1750489 = score(doc=2035,freq=1.0), product of:
             0.33171377 = queryWeight, product of:
               2.1679006 = boost
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.01812209 = queryNorm
             0.5277107 = fieldWeight in 2035, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.0625 = fieldNorm(doc=2035)
         0.043281008 = weight(abstract_txt:results in 2035) [ClassicSimilarity], result of:
           0.043281008 = score(doc=2035,freq=2.0), product of:
             0.14076419 = queryWeight, product of:
               2.2329228 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.01812209 = queryNorm
             0.30747172 = fieldWeight in 2035, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.0625 = fieldNorm(doc=2035)
         0.09146374 = weight(abstract_txt:users in 2035) [ClassicSimilarity], result of:
           0.09146374 = score(doc=2035,freq=3.0), product of:
             0.23684786 = queryWeight, product of:
               3.6637206 = boost
               3.5672934 = idf(docFreq=3408, maxDocs=44421)
               0.01812209 = queryNorm
             0.38617083 = fieldWeight in 2035, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.5672934 = idf(docFreq=3408, maxDocs=44421)
               0.0625 = fieldNorm(doc=2035)
         0.23368734 = weight(abstract_txt:relevance in 2035) [ClassicSimilarity], result of:
           0.23368734 = score(doc=2035,freq=5.0), product of:
             0.33920673 = queryWeight, product of:
               3.7970862 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.01812209 = queryNorm
             0.68892306 = fieldWeight in 2035, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=2035)
       0.28 = coord(7/25)
   

4. Balatsoukas, P.; Demian, P.: Effects of granularity of search results on the relevance judgment behavior of engineers : building systems for retrieval and understanding of context (2010) 0.18

   0.17787068 = sum of:
     0.17787068 = product of:
       0.6352524 = sum of:
         0.020726534 = weight(abstract_txt:when in 406) [ClassicSimilarity], result of:
           0.020726534 = score(doc=406,freq=1.0), product of:
             0.07998497 = queryWeight, product of:
               1.0645396 = boost
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.01812209 = queryNorm
             0.25913036 = fieldWeight in 406, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.0625 = fieldNorm(doc=406)
         0.019325785 = weight(abstract_txt:their in 406) [ClassicSimilarity], result of:
           0.019325785 = score(doc=406,freq=2.0), product of:
             0.06935894 = queryWeight, product of:
               1.2141 = boost
               3.1523883 = idf(docFreq=5161, maxDocs=44421)
               0.01812209 = queryNorm
             0.2786344 = fieldWeight in 406, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.1523883 = idf(docFreq=5161, maxDocs=44421)
               0.0625 = fieldNorm(doc=406)
         0.040149085 = weight(abstract_txt:search in 406) [ClassicSimilarity], result of:
           0.040149085 = score(doc=406,freq=2.0), product of:
             0.12429138 = queryWeight, product of:
               1.8766925 = boost
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.01812209 = queryNorm
             0.3230239 = fieldWeight in 406, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.0625 = fieldNorm(doc=406)
         0.22807385 = weight(abstract_txt:judgment in 406) [ClassicSimilarity], result of:
           0.22807385 = score(doc=406,freq=4.0), product of:
             0.24928008 = queryWeight, product of:
               1.8793229 = boost
               7.319441 = idf(docFreq=79, maxDocs=44421)
               0.01812209 = queryNorm
             0.9149301 = fieldWeight in 406, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.319441 = idf(docFreq=79, maxDocs=44421)
               0.0625 = fieldNorm(doc=406)
         0.043281008 = weight(abstract_txt:results in 406) [ClassicSimilarity], result of:
           0.043281008 = score(doc=406,freq=2.0), product of:
             0.14076419 = queryWeight, product of:
               2.2329228 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.01812209 = queryNorm
             0.30747172 = fieldWeight in 406, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.0625 = fieldNorm(doc=406)
         0.07467983 = weight(abstract_txt:users in 406) [ClassicSimilarity], result of:
           0.07467983 = score(doc=406,freq=2.0), product of:
             0.23684786 = queryWeight, product of:
               3.6637206 = boost
               3.5672934 = idf(docFreq=3408, maxDocs=44421)
               0.01812209 = queryNorm
             0.31530717 = fieldWeight in 406, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.5672934 = idf(docFreq=3408, maxDocs=44421)
               0.0625 = fieldNorm(doc=406)
         0.20901631 = weight(abstract_txt:relevance in 406) [ClassicSimilarity], result of:
           0.20901631 = score(doc=406,freq=4.0), product of:
             0.33920673 = queryWeight, product of:
               3.7970862 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.01812209 = queryNorm
             0.6161915 = fieldWeight in 406, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=406)
       0.28 = coord(7/25)
   

5. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Analysis of change in users' assessment of search results over time (2017) 0.18

   0.17570744 = sum of:
     0.17570744 = product of:
       0.6275266 = sum of:
         0.020726534 = weight(abstract_txt:when in 4593) [ClassicSimilarity], result of:
           0.020726534 = score(doc=4593,freq=1.0), product of:
             0.07998497 = queryWeight, product of:
               1.0645396 = boost
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.01812209 = queryNorm
             0.25913036 = fieldWeight in 4593, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.013665394 = weight(abstract_txt:their in 4593) [ClassicSimilarity], result of:
           0.013665394 = score(doc=4593,freq=1.0), product of:
             0.06935894 = queryWeight, product of:
               1.2141 = boost
               3.1523883 = idf(docFreq=5161, maxDocs=44421)
               0.01812209 = queryNorm
             0.19702427 = fieldWeight in 4593, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.1523883 = idf(docFreq=5161, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.049172387 = weight(abstract_txt:search in 4593) [ClassicSimilarity], result of:
           0.049172387 = score(doc=4593,freq=3.0), product of:
             0.12429138 = queryWeight, product of:
               1.8766925 = boost
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.01812209 = queryNorm
             0.39562184 = fieldWeight in 4593, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.654598 = idf(docFreq=3123, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.1750489 = weight(abstract_txt:categorical in 4593) [ClassicSimilarity], result of:
           0.1750489 = score(doc=4593,freq=1.0), product of:
             0.33171377 = queryWeight, product of:
               2.1679006 = boost
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.01812209 = queryNorm
             0.5277107 = fieldWeight in 4593, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.068433285 = weight(abstract_txt:results in 4593) [ClassicSimilarity], result of:
           0.068433285 = score(doc=4593,freq=5.0), product of:
             0.14076419 = queryWeight, product of:
               2.2329228 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.01812209 = queryNorm
             0.4861555 = fieldWeight in 4593, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.09146374 = weight(abstract_txt:users in 4593) [ClassicSimilarity], result of:
           0.09146374 = score(doc=4593,freq=3.0), product of:
             0.23684786 = queryWeight, product of:
               3.6637206 = boost
               3.5672934 = idf(docFreq=3408, maxDocs=44421)
               0.01812209 = queryNorm
             0.38617083 = fieldWeight in 4593, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.5672934 = idf(docFreq=3408, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.20901631 = weight(abstract_txt:relevance in 4593) [ClassicSimilarity], result of:
           0.20901631 = score(doc=4593,freq=4.0), product of:
             0.33920673 = queryWeight, product of:
               3.7970862 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.01812209 = queryNorm
             0.6161915 = fieldWeight in 4593, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
       0.28 = coord(7/25)