Document (#13223)

Author

Good, I.J.

Title

¬The decision-theory approach to the evaluation of information-retrieval systems

Source

Information storage review. 3(1967), S.31-34

Year

1967

Abstract

It is argued that the evaluation of information-retrieval systems should ultimately be based on the principle of rationality, the maximization of expected utility. In full generality this would involve an estimation of both the cost and value of a system, but the emphasis in this paper is on the problem of value, in terms of which the effiency of the system could be defined. One implication of the discussion is that it is not legitimate to superimpose the 2x2 contingency tables that refer to select/discarded and relevant/irrelevant, correspondending to each request,but it might be all right to superimpose them after applying a monotonic function to the entries. In particular, it is questionable whether a useful statistic is the ratio of the total number of relevant selected documents to the total number of relevant ones, over a sample of requests.

Theme

Retrievalstudien

Similar documents (content)

1. Lancaster, F.W.: Evaluating the performance of a large computerized information system (1985) 0.11

   0.111996636 = sum of:
     0.111996636 = product of:
       0.3499895 = sum of:
         0.04713868 = weight(abstract_txt:request in 4649) [ClassicSimilarity], result of:
           0.04713868 = score(doc=4649,freq=1.0), product of:
             0.1449175 = queryWeight, product of:
               1.0024486 = boost
               6.939294 = idf(docFreq=116, maxDocs=44421)
               0.0208326 = queryNorm
             0.3252794 = fieldWeight in 4649, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.939294 = idf(docFreq=116, maxDocs=44421)
               0.046875 = fieldNorm(doc=4649)
         0.023709275 = weight(abstract_txt:retrieval in 4649) [ClassicSimilarity], result of:
           0.023709275 = score(doc=4649,freq=4.0), product of:
             0.07274526 = queryWeight, product of:
               1.0044285 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0208326 = queryNorm
             0.3259219 = fieldWeight in 4649, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.046875 = fieldNorm(doc=4649)
         0.012516865 = weight(abstract_txt:that in 4649) [ClassicSimilarity], result of:
           0.012516865 = score(doc=4649,freq=5.0), product of:
             0.050495192 = queryWeight, product of:
               1.0249138 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0208326 = queryNorm
             0.2478823 = fieldWeight in 4649, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.046875 = fieldNorm(doc=4649)
         0.050842967 = weight(abstract_txt:ultimately in 4649) [ClassicSimilarity], result of:
           0.050842967 = score(doc=4649,freq=1.0), product of:
             0.15241341 = queryWeight, product of:
               1.0280477 = boost
               7.1165 = idf(docFreq=97, maxDocs=44421)
               0.0208326 = queryNorm
             0.33358592 = fieldWeight in 4649, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.1165 = idf(docFreq=97, maxDocs=44421)
               0.046875 = fieldNorm(doc=4649)
         0.05774124 = weight(abstract_txt:irrelevant in 4649) [ClassicSimilarity], result of:
           0.05774124 = score(doc=4649,freq=1.0), product of:
             0.16590519 = queryWeight, product of:
               1.0725849 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.0208326 = queryNorm
             0.34803757 = fieldWeight in 4649, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.046875 = fieldNorm(doc=4649)
         0.03456696 = weight(abstract_txt:number in 4649) [ClassicSimilarity], result of:
           0.03456696 = score(doc=4649,freq=3.0), product of:
             0.10294683 = queryWeight, product of:
               1.1948769 = boost
               4.1356745 = idf(docFreq=1930, maxDocs=44421)
               0.0208326 = queryNorm
             0.3357749 = fieldWeight in 4649, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.1356745 = idf(docFreq=1930, maxDocs=44421)
               0.046875 = fieldNorm(doc=4649)
         0.050724592 = weight(abstract_txt:evaluation in 4649) [ClassicSimilarity], result of:
           0.050724592 = score(doc=4649,freq=4.0), product of:
             0.120782755 = queryWeight, product of:
               1.2942529 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0208326 = queryNorm
             0.4199655 = fieldWeight in 4649, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.046875 = fieldNorm(doc=4649)
         0.07274894 = weight(abstract_txt:relevant in 4649) [ClassicSimilarity], result of:
           0.07274894 = score(doc=4649,freq=3.0), product of:
             0.19353239 = queryWeight, product of:
               2.0064998 = boost
               4.6298943 = idf(docFreq=1177, maxDocs=44421)
               0.0208326 = queryNorm
             0.37590057 = fieldWeight in 4649, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.6298943 = idf(docFreq=1177, maxDocs=44421)
               0.046875 = fieldNorm(doc=4649)
       0.32 = coord(8/25)
   

2. Blagden, J.F.: How much noise in a role-free and link-free co-ordinate indexing system? (1966) 0.08

   0.0786727 = sum of:
     0.0786727 = product of:
       0.32780293 = sum of:
         0.019757727 = weight(abstract_txt:retrieval in 2717) [ClassicSimilarity], result of:
           0.019757727 = score(doc=2717,freq=1.0), product of:
             0.07274526 = queryWeight, product of:
               1.0044285 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0208326 = queryNorm
             0.27160156 = fieldWeight in 2717, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.078125 = fieldNorm(doc=2717)
         0.00932952 = weight(abstract_txt:that in 2717) [ClassicSimilarity], result of:
           0.00932952 = score(doc=2717,freq=1.0), product of:
             0.050495192 = queryWeight, product of:
               1.0249138 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0208326 = queryNorm
             0.18476056 = fieldWeight in 2717, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.078125 = fieldNorm(doc=2717)
         0.1360974 = weight(abstract_txt:irrelevant in 2717) [ClassicSimilarity], result of:
           0.1360974 = score(doc=2717,freq=2.0), product of:
             0.16590519 = queryWeight, product of:
               1.0725849 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.0208326 = queryNorm
             0.8203324 = fieldWeight in 2717, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.078125 = fieldNorm(doc=2717)
         0.033262074 = weight(abstract_txt:number in 2717) [ClassicSimilarity], result of:
           0.033262074 = score(doc=2717,freq=1.0), product of:
             0.10294683 = queryWeight, product of:
               1.1948769 = boost
               4.1356745 = idf(docFreq=1930, maxDocs=44421)
               0.0208326 = queryNorm
             0.32309955 = fieldWeight in 2717, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1356745 = idf(docFreq=1930, maxDocs=44421)
               0.078125 = fieldNorm(doc=2717)
         0.04227049 = weight(abstract_txt:evaluation in 2717) [ClassicSimilarity], result of:
           0.04227049 = score(doc=2717,freq=1.0), product of:
             0.120782755 = queryWeight, product of:
               1.2942529 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0208326 = queryNorm
             0.34997123 = fieldWeight in 2717, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.078125 = fieldNorm(doc=2717)
         0.08708572 = weight(abstract_txt:total in 2717) [ClassicSimilarity], result of:
           0.08708572 = score(doc=2717,freq=1.0), product of:
             0.1955591 = queryWeight, product of:
               1.6468562 = boost
               5.7000527 = idf(docFreq=403, maxDocs=44421)
               0.0208326 = queryNorm
             0.4453166 = fieldWeight in 2717, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7000527 = idf(docFreq=403, maxDocs=44421)
               0.078125 = fieldNorm(doc=2717)
       0.24 = coord(6/25)
   

3. Plaunt, C.; Norgard, B.A.: ¬An association-based method for automatic indexing with a controlled vocabulary (1998) 0.08

   0.078069836 = sum of:
     0.078069836 = product of:
       0.32529098 = sum of:
         0.015806181 = weight(abstract_txt:retrieval in 2794) [ClassicSimilarity], result of:
           0.015806181 = score(doc=2794,freq=1.0), product of:
             0.07274526 = queryWeight, product of:
               1.0044285 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0208326 = queryNorm
             0.21728125 = fieldWeight in 2794, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0625 = fieldNorm(doc=2794)
         0.010555147 = weight(abstract_txt:that in 2794) [ClassicSimilarity], result of:
           0.010555147 = score(doc=2794,freq=2.0), product of:
             0.050495192 = queryWeight, product of:
               1.0249138 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0208326 = queryNorm
             0.20903271 = fieldWeight in 2794, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=2794)
         0.08280024 = weight(abstract_txt:ratio in 2794) [ClassicSimilarity], result of:
           0.08280024 = score(doc=2794,freq=1.0), product of:
             0.17415307 = queryWeight, product of:
               1.098923 = boost
               7.607123 = idf(docFreq=59, maxDocs=44421)
               0.0208326 = queryNorm
             0.47544518 = fieldWeight in 2794, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.607123 = idf(docFreq=59, maxDocs=44421)
               0.0625 = fieldNorm(doc=2794)
         0.12631084 = weight(abstract_txt:statistic in 2794) [ClassicSimilarity], result of:
           0.12631084 = score(doc=2794,freq=1.0), product of:
             0.23078302 = queryWeight, product of:
               1.265038 = boost
               8.757029 = idf(docFreq=18, maxDocs=44421)
               0.0208326 = queryNorm
             0.5473143 = fieldWeight in 2794, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.757029 = idf(docFreq=18, maxDocs=44421)
               0.0625 = fieldNorm(doc=2794)
         0.033816393 = weight(abstract_txt:evaluation in 2794) [ClassicSimilarity], result of:
           0.033816393 = score(doc=2794,freq=1.0), product of:
             0.120782755 = queryWeight, product of:
               1.2942529 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0208326 = queryNorm
             0.279977 = fieldWeight in 2794, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0625 = fieldNorm(doc=2794)
         0.05600216 = weight(abstract_txt:relevant in 2794) [ClassicSimilarity], result of:
           0.05600216 = score(doc=2794,freq=1.0), product of:
             0.19353239 = queryWeight, product of:
               2.0064998 = boost
               4.6298943 = idf(docFreq=1177, maxDocs=44421)
               0.0208326 = queryNorm
             0.2893684 = fieldWeight in 2794, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6298943 = idf(docFreq=1177, maxDocs=44421)
               0.0625 = fieldNorm(doc=2794)
       0.24 = coord(6/25)
   

4. Kantor, P.; Kim, M.H.; Ibraev, U.; Atasoy, K.: Estimating the number of relevant documents in enormous collections (1999) 0.07

   0.073015235 = sum of:
     0.073015235 = product of:
       0.36507618 = sum of:
         0.022353316 = weight(abstract_txt:retrieval in 690) [ClassicSimilarity], result of:
           0.022353316 = score(doc=690,freq=2.0), product of:
             0.07274526 = queryWeight, product of:
               1.0044285 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0208326 = queryNorm
             0.3072821 = fieldWeight in 690, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0625 = fieldNorm(doc=690)
         0.016689153 = weight(abstract_txt:that in 690) [ClassicSimilarity], result of:
           0.016689153 = score(doc=690,freq=5.0), product of:
             0.050495192 = queryWeight, product of:
               1.0249138 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0208326 = queryNorm
             0.33050975 = fieldWeight in 690, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=690)
         0.05950101 = weight(abstract_txt:number in 690) [ClassicSimilarity], result of:
           0.05950101 = score(doc=690,freq=5.0), product of:
             0.10294683 = queryWeight, product of:
               1.1948769 = boost
               4.1356745 = idf(docFreq=1930, maxDocs=44421)
               0.0208326 = queryNorm
             0.5779781 = fieldWeight in 690, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.1356745 = idf(docFreq=1930, maxDocs=44421)
               0.0625 = fieldNorm(doc=690)
         0.09852624 = weight(abstract_txt:total in 690) [ClassicSimilarity], result of:
           0.09852624 = score(doc=690,freq=2.0), product of:
             0.1955591 = queryWeight, product of:
               1.6468562 = boost
               5.7000527 = idf(docFreq=403, maxDocs=44421)
               0.0208326 = queryNorm
             0.5038182 = fieldWeight in 690, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.7000527 = idf(docFreq=403, maxDocs=44421)
               0.0625 = fieldNorm(doc=690)
         0.16800646 = weight(abstract_txt:relevant in 690) [ClassicSimilarity], result of:
           0.16800646 = score(doc=690,freq=9.0), product of:
             0.19353239 = queryWeight, product of:
               2.0064998 = boost
               4.6298943 = idf(docFreq=1177, maxDocs=44421)
               0.0208326 = queryNorm
             0.8681052 = fieldWeight in 690, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               4.6298943 = idf(docFreq=1177, maxDocs=44421)
               0.0625 = fieldNorm(doc=690)
       0.2 = coord(5/25)
   

5. Bletsas, A.; Sahalos, J.N.: Hirsch index rankings require scaling and higher moment (2009) 0.07

   0.06891363 = sum of:
     0.06891363 = product of:
       0.34456813 = sum of:
         0.016159201 = weight(abstract_txt:that in 128) [ClassicSimilarity], result of:
           0.016159201 = score(doc=128,freq=3.0), product of:
             0.050495192 = queryWeight, product of:
               1.0249138 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0208326 = queryNorm
             0.32001466 = fieldWeight in 128, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.078125 = fieldNorm(doc=128)
         0.11569233 = weight(abstract_txt:estimation in 128) [ClassicSimilarity], result of:
           0.11569233 = score(doc=128,freq=1.0), product of:
             0.18757421 = queryWeight, product of:
               1.1404815 = boost
               7.894805 = idf(docFreq=44, maxDocs=44421)
               0.0208326 = queryNorm
             0.61678165 = fieldWeight in 128, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.894805 = idf(docFreq=44, maxDocs=44421)
               0.078125 = fieldNorm(doc=128)
         0.057611607 = weight(abstract_txt:number in 128) [ClassicSimilarity], result of:
           0.057611607 = score(doc=128,freq=3.0), product of:
             0.10294683 = queryWeight, product of:
               1.1948769 = boost
               4.1356745 = idf(docFreq=1930, maxDocs=44421)
               0.0208326 = queryNorm
             0.55962485 = fieldWeight in 128, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.1356745 = idf(docFreq=1930, maxDocs=44421)
               0.078125 = fieldNorm(doc=128)
         0.068019256 = weight(abstract_txt:value in 128) [ClassicSimilarity], result of:
           0.068019256 = score(doc=128,freq=3.0), product of:
             0.11499905 = queryWeight, product of:
               1.262885 = boost
               4.3710623 = idf(docFreq=1525, maxDocs=44421)
               0.0208326 = queryNorm
             0.5914767 = fieldWeight in 128, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.3710623 = idf(docFreq=1525, maxDocs=44421)
               0.078125 = fieldNorm(doc=128)
         0.08708572 = weight(abstract_txt:total in 128) [ClassicSimilarity], result of:
           0.08708572 = score(doc=128,freq=1.0), product of:
             0.1955591 = queryWeight, product of:
               1.6468562 = boost
               5.7000527 = idf(docFreq=403, maxDocs=44421)
               0.0208326 = queryNorm
             0.4453166 = fieldWeight in 128, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7000527 = idf(docFreq=403, maxDocs=44421)
               0.078125 = fieldNorm(doc=128)
       0.2 = coord(5/25)