Document (#10817)

Author

Yao, Y.Y.

Title

Measuring retrieval effectiveness based on user preference of documents

Source

Journal of the American Society for Information Science. 46(1995) no.2, S.133-145

Year

1995

Abstract

The notion of user preference is adopted for the representation, interpretation, and measurement of the relevance or usefulness of documents. Use judgements on documents may be formally describes by a weak order (i.e. user ranking) and measured using an ordinal scale. Within this framework, a new measure of system performance is suggested based on the distance between user ranking and system ranking. It only uses the relative order of documents and therefore confirms to the valid use of an ordinal scale measuring relevance. It is also applicable to multilevel relevance judgements and ranked system output. The appropriateness of the proposed measure is demonstrated through an axiomatic approach. The inherent relationships between the new measure and many existing measures provide further supporting evidence

Similar documents (content)

1. Bartell, B.T.; Cottrell, G.W.; Belew, R.K.: Optimizing similarity using multi-query relevance feedback (1998) 0.23

   0.23017848 = sum of:
     0.23017848 = product of:
       0.6393846 = sum of:
         0.046072464 = weight(abstract_txt:ranked in 2152) [ClassicSimilarity], result of:
           0.046072464 = score(doc=2152,freq=1.0), product of:
             0.11467108 = queryWeight, product of:
               6.428468 = idf(docFreq=194, maxDocs=44421)
               0.01783801 = queryNorm
             0.40177926 = fieldWeight in 2152, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.428468 = idf(docFreq=194, maxDocs=44421)
               0.0625 = fieldNorm(doc=2152)
         0.014334957 = weight(abstract_txt:between in 2152) [ClassicSimilarity], result of:
           0.014334957 = score(doc=2152,freq=1.0), product of:
             0.06633874 = queryWeight, product of:
               1.0756516 = boost
               3.4573963 = idf(docFreq=3804, maxDocs=44421)
               0.01783801 = queryNorm
             0.21608727 = fieldWeight in 2152, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4573963 = idf(docFreq=3804, maxDocs=44421)
               0.0625 = fieldNorm(doc=2152)
         0.019961921 = weight(abstract_txt:system in 2152) [ClassicSimilarity], result of:
           0.019961921 = score(doc=2152,freq=1.0), product of:
             0.09469672 = queryWeight, product of:
               1.573987 = boost
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.01783801 = queryNorm
             0.21079844 = fieldWeight in 2152, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.0625 = fieldNorm(doc=2152)
         0.11948827 = weight(abstract_txt:preference in 2152) [ClassicSimilarity], result of:
           0.11948827 = score(doc=2152,freq=1.0), product of:
             0.27272177 = queryWeight, product of:
               2.1809611 = boost
               7.01012 = idf(docFreq=108, maxDocs=44421)
               0.01783801 = queryNorm
             0.4381325 = fieldWeight in 2152, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.01012 = idf(docFreq=108, maxDocs=44421)
               0.0625 = fieldNorm(doc=2152)
         0.034596313 = weight(abstract_txt:user in 2152) [ClassicSimilarity], result of:
           0.034596313 = score(doc=2152,freq=1.0), product of:
             0.15038326 = queryWeight, product of:
               2.290355 = boost
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.01783801 = queryNorm
             0.23005427 = fieldWeight in 2152, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.0625 = fieldNorm(doc=2152)
         0.062324278 = weight(abstract_txt:relevance in 2152) [ClassicSimilarity], result of:
           0.062324278 = score(doc=2152,freq=1.0), product of:
             0.20228866 = queryWeight, product of:
               2.3004851 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.01783801 = queryNorm
             0.30809575 = fieldWeight in 2152, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=2152)
         0.16716616 = weight(abstract_txt:measure in 2152) [ClassicSimilarity], result of:
           0.16716616 = score(doc=2152,freq=4.0), product of:
             0.24600595 = queryWeight, product of:
               2.5369182 = boost
               5.4361663 = idf(docFreq=525, maxDocs=44421)
               0.01783801 = queryNorm
             0.6795208 = fieldWeight in 2152, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.4361663 = idf(docFreq=525, maxDocs=44421)
               0.0625 = fieldNorm(doc=2152)
         0.084232815 = weight(abstract_txt:documents in 2152) [ClassicSimilarity], result of:
           0.084232815 = score(doc=2152,freq=3.0), product of:
             0.18870935 = queryWeight, product of:
               2.5656638 = boost
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.01783801 = queryNorm
             0.4463627 = fieldWeight in 2152, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.0625 = fieldNorm(doc=2152)
         0.09120746 = weight(abstract_txt:ranking in 2152) [ClassicSimilarity], result of:
           0.09120746 = score(doc=2152,freq=1.0), product of:
             0.26074758 = queryWeight, product of:
               2.6118233 = boost
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.01783801 = queryNorm
             0.34979215 = fieldWeight in 2152, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.0625 = fieldNorm(doc=2152)
       0.36 = coord(9/25)
   

2. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Analysis of change in users' assessment of search results over time (2017) 0.18

   0.1840803 = sum of:
     0.1840803 = product of:
       0.7670013 = sum of:
         0.02027269 = weight(abstract_txt:between in 4593) [ClassicSimilarity], result of:
           0.02027269 = score(doc=4593,freq=2.0), product of:
             0.06633874 = queryWeight, product of:
               1.0756516 = boost
               3.4573963 = idf(docFreq=3804, maxDocs=44421)
               0.01783801 = queryNorm
             0.30559355 = fieldWeight in 4593, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4573963 = idf(docFreq=3804, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.09827385 = weight(abstract_txt:scale in 4593) [ClassicSimilarity], result of:
           0.09827385 = score(doc=4593,freq=3.0), product of:
             0.16599227 = queryWeight, product of:
               1.7015 = boost
               5.4690194 = idf(docFreq=508, maxDocs=44421)
               0.01783801 = queryNorm
             0.5920387 = fieldWeight in 4593, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.4690194 = idf(docFreq=508, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.077359706 = weight(abstract_txt:user in 4593) [ClassicSimilarity], result of:
           0.077359706 = score(doc=4593,freq=5.0), product of:
             0.15038326 = queryWeight, product of:
               2.290355 = boost
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.01783801 = queryNorm
             0.514417 = fieldWeight in 4593, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.124648556 = weight(abstract_txt:relevance in 4593) [ClassicSimilarity], result of:
           0.124648556 = score(doc=4593,freq=4.0), product of:
             0.20228866 = queryWeight, product of:
               2.3004851 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.01783801 = queryNorm
             0.6161915 = fieldWeight in 4593, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.3174596 = weight(abstract_txt:judgements in 4593) [ClassicSimilarity], result of:
           0.3174596 = score(doc=4593,freq=5.0), product of:
             0.30594164 = queryWeight, product of:
               2.3099754 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.01783801 = queryNorm
             1.0376476 = fieldWeight in 4593, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.12898682 = weight(abstract_txt:ranking in 4593) [ClassicSimilarity], result of:
           0.12898682 = score(doc=4593,freq=2.0), product of:
             0.26074758 = queryWeight, product of:
               2.6118233 = boost
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.01783801 = queryNorm
             0.4946808 = fieldWeight in 4593, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
       0.24 = coord(6/25)
   

3. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Testing the stability of "wisdom of crowds" judgments of search results over time and their similarity with the search engine rankings (2016) 0.18

   0.18064293 = sum of:
     0.18064293 = product of:
       0.75267893 = sum of:
         0.012543088 = weight(abstract_txt:between in 4071) [ClassicSimilarity], result of:
           0.012543088 = score(doc=4071,freq=1.0), product of:
             0.06633874 = queryWeight, product of:
               1.0756516 = boost
               3.4573963 = idf(docFreq=3804, maxDocs=44421)
               0.01783801 = queryNorm
             0.18907636 = fieldWeight in 4071, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4573963 = idf(docFreq=3804, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4071)
         0.08598962 = weight(abstract_txt:scale in 4071) [ClassicSimilarity], result of:
           0.08598962 = score(doc=4071,freq=3.0), product of:
             0.16599227 = queryWeight, product of:
               1.7015 = boost
               5.4690194 = idf(docFreq=508, maxDocs=44421)
               0.01783801 = queryNorm
             0.51803386 = fieldWeight in 4071, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.4690194 = idf(docFreq=508, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4071)
         0.0741504 = weight(abstract_txt:user in 4071) [ClassicSimilarity], result of:
           0.0741504 = score(doc=4071,freq=6.0), product of:
             0.15038326 = queryWeight, product of:
               2.290355 = boost
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.01783801 = queryNorm
             0.49307615 = fieldWeight in 4071, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4071)
         0.094455205 = weight(abstract_txt:relevance in 4071) [ClassicSimilarity], result of:
           0.094455205 = score(doc=4071,freq=3.0), product of:
             0.20228866 = queryWeight, product of:
               2.3004851 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.01783801 = queryNorm
             0.46693277 = fieldWeight in 4071, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4071)
         0.37267718 = weight(abstract_txt:judgements in 4071) [ClassicSimilarity], result of:
           0.37267718 = score(doc=4071,freq=9.0), product of:
             0.30594164 = queryWeight, product of:
               2.3099754 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.01783801 = queryNorm
             1.2181315 = fieldWeight in 4071, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4071)
         0.112863466 = weight(abstract_txt:ranking in 4071) [ClassicSimilarity], result of:
           0.112863466 = score(doc=4071,freq=2.0), product of:
             0.26074758 = queryWeight, product of:
               2.6118233 = boost
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.01783801 = queryNorm
             0.43284568 = fieldWeight in 4071, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4071)
       0.24 = coord(6/25)
   

4. Wong, S.K.M.; Yao, Y.Y.; Salton, G.; Buckley, C.: Evaluation of an adaptive linear model (1991) 0.17

   0.16853729 = sum of:
     0.16853729 = product of:
       0.8426864 = sum of:
         0.23897654 = weight(abstract_txt:preference in 4835) [ClassicSimilarity], result of:
           0.23897654 = score(doc=4835,freq=1.0), product of:
             0.27272177 = queryWeight, product of:
               2.1809611 = boost
               7.01012 = idf(docFreq=108, maxDocs=44421)
               0.01783801 = queryNorm
             0.876265 = fieldWeight in 4835, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.01012 = idf(docFreq=108, maxDocs=44421)
               0.125 = fieldNorm(doc=4835)
         0.09785315 = weight(abstract_txt:user in 4835) [ClassicSimilarity], result of:
           0.09785315 = score(doc=4835,freq=2.0), product of:
             0.15038326 = queryWeight, product of:
               2.290355 = boost
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.01783801 = queryNorm
             0.65069175 = fieldWeight in 4835, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.125 = fieldNorm(doc=4835)
         0.124648556 = weight(abstract_txt:relevance in 4835) [ClassicSimilarity], result of:
           0.124648556 = score(doc=4835,freq=1.0), product of:
             0.20228866 = queryWeight, product of:
               2.3004851 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.01783801 = queryNorm
             0.6161915 = fieldWeight in 4835, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.125 = fieldNorm(doc=4835)
         0.2839445 = weight(abstract_txt:judgements in 4835) [ClassicSimilarity], result of:
           0.2839445 = score(doc=4835,freq=1.0), product of:
             0.30594164 = queryWeight, product of:
               2.3099754 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.01783801 = queryNorm
             0.92810017 = fieldWeight in 4835, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.125 = fieldNorm(doc=4835)
         0.09726368 = weight(abstract_txt:documents in 4835) [ClassicSimilarity], result of:
           0.09726368 = score(doc=4835,freq=1.0), product of:
             0.18870935 = queryWeight, product of:
               2.5656638 = boost
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.01783801 = queryNorm
             0.51541525 = fieldWeight in 4835, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.125 = fieldNorm(doc=4835)
       0.2 = coord(5/25)
   

5. Cosijn, E.: Relevance judgments and measurements (2009) 0.16

   0.16418627 = sum of:
     0.16418627 = product of:
       0.5863795 = sum of:
         0.057590578 = weight(abstract_txt:ranked in 842) [ClassicSimilarity], result of:
           0.057590578 = score(doc=842,freq=1.0), product of:
             0.11467108 = queryWeight, product of:
               6.428468 = idf(docFreq=194, maxDocs=44421)
               0.01783801 = queryNorm
             0.5022241 = fieldWeight in 842, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.428468 = idf(docFreq=194, maxDocs=44421)
               0.078125 = fieldNorm(doc=842)
         0.031036092 = weight(abstract_txt:between in 842) [ClassicSimilarity], result of:
           0.031036092 = score(doc=842,freq=3.0), product of:
             0.06633874 = queryWeight, product of:
               1.0756516 = boost
               3.4573963 = idf(docFreq=3804, maxDocs=44421)
               0.01783801 = queryNorm
             0.46784264 = fieldWeight in 842, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.4573963 = idf(docFreq=3804, maxDocs=44421)
               0.078125 = fieldNorm(doc=842)
         0.07190987 = weight(abstract_txt:measurement in 842) [ClassicSimilarity], result of:
           0.07190987 = score(doc=842,freq=1.0), product of:
             0.13296743 = queryWeight, product of:
               1.0768263 = boost
               6.922344 = idf(docFreq=118, maxDocs=44421)
               0.01783801 = queryNorm
             0.54080814 = fieldWeight in 842, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.922344 = idf(docFreq=118, maxDocs=44421)
               0.078125 = fieldNorm(doc=842)
         0.024952404 = weight(abstract_txt:system in 842) [ClassicSimilarity], result of:
           0.024952404 = score(doc=842,freq=1.0), product of:
             0.09469672 = queryWeight, product of:
               1.573987 = boost
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.01783801 = queryNorm
             0.26349807 = fieldWeight in 842, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.078125 = fieldNorm(doc=842)
         0.061158217 = weight(abstract_txt:user in 842) [ClassicSimilarity], result of:
           0.061158217 = score(doc=842,freq=2.0), product of:
             0.15038326 = queryWeight, product of:
               2.290355 = boost
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.01783801 = queryNorm
             0.40668234 = fieldWeight in 842, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.078125 = fieldNorm(doc=842)
         0.19082834 = weight(abstract_txt:relevance in 842) [ClassicSimilarity], result of:
           0.19082834 = score(doc=842,freq=6.0), product of:
             0.20228866 = queryWeight, product of:
               2.3004851 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.01783801 = queryNorm
             0.94334674 = fieldWeight in 842, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.078125 = fieldNorm(doc=842)
         0.148904 = weight(abstract_txt:documents in 842) [ClassicSimilarity], result of:
           0.148904 = score(doc=842,freq=6.0), product of:
             0.18870935 = queryWeight, product of:
               2.5656638 = boost
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.01783801 = queryNorm
             0.7890653 = fieldWeight in 842, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.078125 = fieldNorm(doc=842)
       0.28 = coord(7/25)