Document (#32436)

Author

Wacholder, N.
Kelly, D.
Kantor, P.
Rittman, R.
Sun, Y.
Bai, B.
Small, S.
Yamrom, B.
Strzalkowski, T.

Title

¬A model for quantitative evaluation of an end-to-end question-answering system

Source

Journal of the American Society for Information Science and Technology. 58(2007) no.8, S.1082-1099

Year

2007

Abstract

We describe a procedure for quantitative evaluation of interactive question-answering systems and illustrate it with application to the High-Quality Interactive QuestionAnswering (HITIQA) system. Our objectives were (a) to design a method to realistically and reliably assess interactive question-answering systems by comparing the quality of reports produced using different systems, (b) to conduct a pilot test of this method, and (c) to perform a formative evaluation of the HITIQA system. Far more important than the specific information gathered from this pilot evaluation is the development of (a) a protocol for evaluating an emerging technology, (b) reusable assessment instruments, and (c) the knowledge gained in conducting the evaluation. We conclude that this method, which uses a surprisingly small number of subjects and does not rely on predetermined relevance judgments, measures the impact of system change on work produced by users. Therefore this method can be used to compare the product of interactive systems that use different underlying technologies.

Object

HITIQA

Similar documents (author)

1. Kelly, D.; Wacholder, N.; Rittman, R.; Sun, Y.; Kantor, P.; Small, S.; Strzalkowski, T.: Using interview data to identify evaluation criteria for interactive, analytical question-answering systems (2007) 4.96

   4.9565516 = sum of:
     4.9565516 = sum of:
       0.77087075 = weight(author_txt:kantor in 1332) [ClassicSimilarity], result of:
         0.77087075 = score(doc=1332,freq=1.0), product of:
           0.37717953 = queryWeight, product of:
             8.175107 = idf(docFreq=33, maxDocs=44421)
             0.046137564 = queryNorm
           2.0437768 = fieldWeight in 1332, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.175107 = idf(docFreq=33, maxDocs=44421)
             0.25 = fieldNorm(doc=1332)
       0.87365896 = weight(author_txt:kelly in 1332) [ClassicSimilarity], result of:
         0.87365896 = score(doc=1332,freq=1.0), product of:
           0.41000426 = queryWeight, product of:
             1.0426058 = boost
             8.523414 = idf(docFreq=23, maxDocs=44421)
             0.046137564 = queryNorm
           2.1308534 = fieldWeight in 1332, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.523414 = idf(docFreq=23, maxDocs=44421)
             0.25 = fieldNorm(doc=1332)
       1.004366 = weight(author_txt:small in 1332) [ClassicSimilarity], result of:
         1.004366 = score(doc=1332,freq=1.0), product of:
           0.44994047 = queryWeight, product of:
             1.0922033 = boost
             8.928879 = idf(docFreq=15, maxDocs=44421)
             0.046137564 = queryNorm
           2.2322197 = fieldWeight in 1332, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.928879 = idf(docFreq=15, maxDocs=44421)
             0.25 = fieldNorm(doc=1332)
       1.1361889 = weight(author_txt:strzalkowski in 1332) [ClassicSimilarity], result of:
         1.1361889 = score(doc=1332,freq=1.0), product of:
           0.4884957 = queryWeight, product of:
             1.1380368 = boost
             9.303573 = idf(docFreq=10, maxDocs=44421)
             0.046137564 = queryNorm
           2.3258932 = fieldWeight in 1332, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.303573 = idf(docFreq=10, maxDocs=44421)
             0.25 = fieldNorm(doc=1332)
       1.1714666 = weight(author_txt:wacholder in 1332) [ClassicSimilarity], result of:
         1.1714666 = score(doc=1332,freq=1.0), product of:
           0.4985557 = queryWeight, product of:
             1.1496954 = boost
             9.398883 = idf(docFreq=9, maxDocs=44421)
             0.046137564 = queryNorm
           2.3497207 = fieldWeight in 1332, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.398883 = idf(docFreq=9, maxDocs=44421)
             0.25 = fieldNorm(doc=1332)
   

2. Ng, K.B.; Kantor, P.B.; Strzalkowski, T.; Wacholder, N.; Tang, R.; Bai, B.; Rittman,; Song, P.; Sun, Y.: Automated judgment of document qualities (2006) 1.62

   1.6162263 = sum of:
     1.6162263 = product of:
       2.6937103 = sum of:
         0.6745119 = weight(author_txt:kantor in 307) [ClassicSimilarity], result of:
           0.6745119 = score(doc=307,freq=1.0), product of:
             0.37717953 = queryWeight, product of:
               8.175107 = idf(docFreq=33, maxDocs=44421)
               0.046137564 = queryNorm
             1.7883047 = fieldWeight in 307, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.175107 = idf(docFreq=33, maxDocs=44421)
               0.21875 = fieldNorm(doc=307)
         0.9941652 = weight(author_txt:strzalkowski in 307) [ClassicSimilarity], result of:
           0.9941652 = score(doc=307,freq=1.0), product of:
             0.4884957 = queryWeight, product of:
               1.1380368 = boost
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.046137564 = queryNorm
             2.0351565 = fieldWeight in 307, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.21875 = fieldNorm(doc=307)
         1.0250334 = weight(author_txt:wacholder in 307) [ClassicSimilarity], result of:
           1.0250334 = score(doc=307,freq=1.0), product of:
             0.4985557 = queryWeight, product of:
               1.1496954 = boost
               9.398883 = idf(docFreq=9, maxDocs=44421)
               0.046137564 = queryNorm
             2.0560057 = fieldWeight in 307, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.398883 = idf(docFreq=9, maxDocs=44421)
               0.21875 = fieldNorm(doc=307)
       0.6 = coord(3/5)
   

3. Wacholder, N.: Interactive query formulation (2011) 0.59

   0.58573335 = sum of:
     0.58573335 = product of:
       2.9286666 = sum of:
         2.9286666 = weight(author_txt:wacholder in 196) [ClassicSimilarity], result of:
           2.9286666 = score(doc=196,freq=1.0), product of:
             0.4985557 = queryWeight, product of:
               1.1496954 = boost
               9.398883 = idf(docFreq=9, maxDocs=44421)
               0.046137564 = queryNorm
             5.874302 = fieldWeight in 196, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.398883 = idf(docFreq=9, maxDocs=44421)
               0.625 = fieldNorm(doc=196)
       0.2 = coord(1/5)
   

4. Strzalkowski, T.: Natural language information retrieval (1995) 0.57

   0.56809443 = sum of:
     0.56809443 = product of:
       2.8404722 = sum of:
         2.8404722 = weight(author_txt:strzalkowski in 1982) [ClassicSimilarity], result of:
           2.8404722 = score(doc=1982,freq=1.0), product of:
             0.4884957 = queryWeight, product of:
               1.1380368 = boost
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.046137564 = queryNorm
             5.814733 = fieldWeight in 1982, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.625 = fieldNorm(doc=1982)
       0.2 = coord(1/5)
   

5. Strzalkowski, T.: Reversible grammar in natural langugae processing (19??) 0.57

   0.56809443 = sum of:
     0.56809443 = product of:
       2.8404722 = sum of:
         2.8404722 = weight(author_txt:strzalkowski in 5361) [ClassicSimilarity], result of:
           2.8404722 = score(doc=5361,freq=1.0), product of:
             0.4884957 = queryWeight, product of:
               1.1380368 = boost
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.046137564 = queryNorm
             5.814733 = fieldWeight in 5361, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.625 = fieldNorm(doc=5361)
       0.2 = coord(1/5)
   

Similar documents (content)

1. Sun, Y.; Kantor, P.B.; Morse, E.L.: Using cross-evaluation to evaluate interactive QA systems (2011) 0.62

   0.6231339 = sum of:
     0.6231339 = product of:
       1.2981956 = sum of:
         0.07328382 = weight(abstract_txt:judgments in 744) [ClassicSimilarity], result of:
           0.07328382 = score(doc=744,freq=1.0), product of:
             0.13631849 = queryWeight, product of:
               1.0404173 = boost
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.019040743 = queryNorm
             0.53759265 = fieldWeight in 744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.078125 = fieldNorm(doc=744)
         0.09711723 = weight(abstract_txt:surprisingly in 744) [ClassicSimilarity], result of:
           0.09711723 = score(doc=744,freq=1.0), product of:
             0.16446756 = queryWeight, product of:
               1.1428002 = boost
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.019040743 = queryNorm
             0.59049475 = fieldWeight in 744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.078125 = fieldNorm(doc=744)
         0.15103878 = weight(abstract_txt:predetermined in 744) [ClassicSimilarity], result of:
           0.15103878 = score(doc=744,freq=1.0), product of:
             0.22077082 = queryWeight, product of:
               1.3240398 = boost
               8.757029 = idf(docFreq=18, maxDocs=44421)
               0.019040743 = queryNorm
             0.6841428 = fieldWeight in 744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.757029 = idf(docFreq=18, maxDocs=44421)
               0.078125 = fieldNorm(doc=744)
         0.04506565 = weight(abstract_txt:quality in 744) [ClassicSimilarity], result of:
           0.04506565 = score(doc=744,freq=1.0), product of:
             0.12420043 = queryWeight, product of:
               1.4044515 = boost
               4.6444306 = idf(docFreq=1160, maxDocs=44421)
               0.019040743 = queryNorm
             0.36284614 = fieldWeight in 744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6444306 = idf(docFreq=1160, maxDocs=44421)
               0.078125 = fieldNorm(doc=744)
         0.021709649 = weight(abstract_txt:this in 744) [ClassicSimilarity], result of:
           0.021709649 = score(doc=744,freq=3.0), product of:
             0.06667517 = queryWeight, product of:
               1.4552655 = boost
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.019040743 = queryNorm
             0.3256032 = fieldWeight in 744, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.078125 = fieldNorm(doc=744)
         0.03451758 = weight(abstract_txt:system in 744) [ClassicSimilarity], result of:
           0.03451758 = score(doc=744,freq=1.0), product of:
             0.13099746 = queryWeight, product of:
               2.039819 = boost
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.019040743 = queryNorm
             0.26349807 = fieldWeight in 744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.078125 = fieldNorm(doc=744)
         0.07985005 = weight(abstract_txt:systems in 744) [ClassicSimilarity], result of:
           0.07985005 = score(doc=744,freq=5.0), product of:
             0.13399734 = queryWeight, product of:
               2.0630429 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.019040743 = queryNorm
             0.59590775 = fieldWeight in 744, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.078125 = fieldNorm(doc=744)
         0.09496652 = weight(abstract_txt:question in 744) [ClassicSimilarity], result of:
           0.09496652 = score(doc=744,freq=1.0), product of:
             0.23368822 = queryWeight, product of:
               2.3594415 = boost
               5.2016807 = idf(docFreq=664, maxDocs=44421)
               0.019040743 = queryNorm
             0.4063813 = fieldWeight in 744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.2016807 = idf(docFreq=664, maxDocs=44421)
               0.078125 = fieldNorm(doc=744)
         0.14188313 = weight(abstract_txt:method in 744) [ClassicSimilarity], result of:
           0.14188313 = score(doc=744,freq=3.0), product of:
             0.23306808 = queryWeight, product of:
               2.7208312 = boost
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.019040743 = queryNorm
             0.6087626 = fieldWeight in 744, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.078125 = fieldNorm(doc=744)
         0.19743425 = weight(abstract_txt:answering in 744) [ClassicSimilarity], result of:
           0.19743425 = score(doc=744,freq=1.0), product of:
             0.38066077 = queryWeight, product of:
               3.0113406 = boost
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.019040743 = queryNorm
             0.5186619 = fieldWeight in 744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.078125 = fieldNorm(doc=744)
         0.2183637 = weight(abstract_txt:interactive in 744) [ClassicSimilarity], result of:
           0.2183637 = score(doc=744,freq=2.0), product of:
             0.35564196 = queryWeight, product of:
               3.360986 = boost
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.019040743 = queryNorm
             0.6139987 = fieldWeight in 744, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.078125 = fieldNorm(doc=744)
         0.14296514 = weight(abstract_txt:evaluation in 744) [ClassicSimilarity], result of:
           0.14296514 = score(doc=744,freq=2.0), product of:
             0.28885695 = queryWeight, product of:
               3.386543 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.019040743 = queryNorm
             0.49493405 = fieldWeight in 744, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.078125 = fieldNorm(doc=744)
       0.48 = coord(12/25)
   

2. Kelly, D.; Wacholder, N.; Rittman, R.; Sun, Y.; Kantor, P.; Small, S.; Strzalkowski, T.: Using interview data to identify evaluation criteria for interactive, analytical question-answering systems (2007) 0.35

   0.34574485 = sum of:
     0.34574485 = product of:
       1.0804527 = sum of:
         0.04506565 = weight(abstract_txt:quality in 1332) [ClassicSimilarity], result of:
           0.04506565 = score(doc=1332,freq=1.0), product of:
             0.12420043 = queryWeight, product of:
               1.4044515 = boost
               4.6444306 = idf(docFreq=1160, maxDocs=44421)
               0.019040743 = queryNorm
             0.36284614 = fieldWeight in 1332, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6444306 = idf(docFreq=1160, maxDocs=44421)
               0.078125 = fieldNorm(doc=1332)
         0.017725853 = weight(abstract_txt:this in 1332) [ClassicSimilarity], result of:
           0.017725853 = score(doc=1332,freq=2.0), product of:
             0.06667517 = queryWeight, product of:
               1.4552655 = boost
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.019040743 = queryNorm
             0.26585388 = fieldWeight in 1332, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.078125 = fieldNorm(doc=1332)
         0.048815224 = weight(abstract_txt:system in 1332) [ClassicSimilarity], result of:
           0.048815224 = score(doc=1332,freq=2.0), product of:
             0.13099746 = queryWeight, product of:
               2.039819 = boost
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.019040743 = queryNorm
             0.37264252 = fieldWeight in 1332, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.078125 = fieldNorm(doc=1332)
         0.07142006 = weight(abstract_txt:systems in 1332) [ClassicSimilarity], result of:
           0.07142006 = score(doc=1332,freq=4.0), product of:
             0.13399734 = queryWeight, product of:
               2.0630429 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.019040743 = queryNorm
             0.5329961 = fieldWeight in 1332, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.078125 = fieldNorm(doc=1332)
         0.13430293 = weight(abstract_txt:question in 1332) [ClassicSimilarity], result of:
           0.13430293 = score(doc=1332,freq=2.0), product of:
             0.23368822 = queryWeight, product of:
               2.3594415 = boost
               5.2016807 = idf(docFreq=664, maxDocs=44421)
               0.019040743 = queryNorm
             0.5747099 = fieldWeight in 1332, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.2016807 = idf(docFreq=664, maxDocs=44421)
               0.078125 = fieldNorm(doc=1332)
         0.27921417 = weight(abstract_txt:answering in 1332) [ClassicSimilarity], result of:
           0.27921417 = score(doc=1332,freq=2.0), product of:
             0.38066077 = queryWeight, product of:
               3.0113406 = boost
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.019040743 = queryNorm
             0.7334987 = fieldWeight in 1332, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.078125 = fieldNorm(doc=1332)
         0.30881292 = weight(abstract_txt:interactive in 1332) [ClassicSimilarity], result of:
           0.30881292 = score(doc=1332,freq=4.0), product of:
             0.35564196 = queryWeight, product of:
               3.360986 = boost
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.019040743 = queryNorm
             0.8683253 = fieldWeight in 1332, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.078125 = fieldNorm(doc=1332)
         0.17509583 = weight(abstract_txt:evaluation in 1332) [ClassicSimilarity], result of:
           0.17509583 = score(doc=1332,freq=3.0), product of:
             0.28885695 = queryWeight, product of:
               3.386543 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.019040743 = queryNorm
             0.606168 = fieldWeight in 1332, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.078125 = fieldNorm(doc=1332)
       0.32 = coord(8/25)
   

3. López-Ostenero, F.; Peinado, V.; Gonzalo, J.; Verdejo, F.: Interactive question answering : Is Cross-Language harder than monolingual searching? (2008) 0.31

   0.31350258 = sum of:
     0.31350258 = product of:
       0.87084043 = sum of:
         0.022049379 = weight(abstract_txt:different in 3023) [ClassicSimilarity], result of:
           0.022049379 = score(doc=3023,freq=1.0), product of:
             0.07711821 = queryWeight, product of:
               1.1066841 = boost
               3.6597328 = idf(docFreq=3107, maxDocs=44421)
               0.019040743 = queryNorm
             0.28591663 = fieldWeight in 3023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6597328 = idf(docFreq=3107, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.017725853 = weight(abstract_txt:this in 3023) [ClassicSimilarity], result of:
           0.017725853 = score(doc=3023,freq=2.0), product of:
             0.06667517 = queryWeight, product of:
               1.4552655 = boost
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.019040743 = queryNorm
             0.26585388 = fieldWeight in 3023, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.094853476 = weight(abstract_txt:quantitative in 3023) [ClassicSimilarity], result of:
           0.094853476 = score(doc=3023,freq=1.0), product of:
             0.20398343 = queryWeight, product of:
               1.7998762 = boost
               5.9520745 = idf(docFreq=313, maxDocs=44421)
               0.019040743 = queryNorm
             0.46500582 = fieldWeight in 3023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.9520745 = idf(docFreq=313, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.03451758 = weight(abstract_txt:system in 3023) [ClassicSimilarity], result of:
           0.03451758 = score(doc=3023,freq=1.0), product of:
             0.13099746 = queryWeight, product of:
               2.039819 = boost
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.019040743 = queryNorm
             0.26349807 = fieldWeight in 3023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.05050161 = weight(abstract_txt:systems in 3023) [ClassicSimilarity], result of:
           0.05050161 = score(doc=3023,freq=2.0), product of:
             0.13399734 = queryWeight, product of:
               2.0630429 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.019040743 = queryNorm
             0.37688518 = fieldWeight in 3023, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.13430293 = weight(abstract_txt:question in 3023) [ClassicSimilarity], result of:
           0.13430293 = score(doc=3023,freq=2.0), product of:
             0.23368822 = queryWeight, product of:
               2.3594415 = boost
               5.2016807 = idf(docFreq=664, maxDocs=44421)
               0.019040743 = queryNorm
             0.5747099 = fieldWeight in 3023, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.2016807 = idf(docFreq=664, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.19743425 = weight(abstract_txt:answering in 3023) [ClassicSimilarity], result of:
           0.19743425 = score(doc=3023,freq=1.0), product of:
             0.38066077 = queryWeight, product of:
               3.0113406 = boost
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.019040743 = queryNorm
             0.5186619 = fieldWeight in 3023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.2183637 = weight(abstract_txt:interactive in 3023) [ClassicSimilarity], result of:
           0.2183637 = score(doc=3023,freq=2.0), product of:
             0.35564196 = queryWeight, product of:
               3.360986 = boost
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.019040743 = queryNorm
             0.6139987 = fieldWeight in 3023, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.10109162 = weight(abstract_txt:evaluation in 3023) [ClassicSimilarity], result of:
           0.10109162 = score(doc=3023,freq=1.0), product of:
             0.28885695 = queryWeight, product of:
               3.386543 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.019040743 = queryNorm
             0.34997123 = fieldWeight in 3023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
       0.36 = coord(9/25)
   

4. Lin, J.; Katz, B.: Building a reusable test collection for question answering (2006) 0.31

   0.31121546 = sum of:
     0.31121546 = product of:
       0.97254837 = sum of:
         0.022049379 = weight(abstract_txt:different in 45) [ClassicSimilarity], result of:
           0.022049379 = score(doc=45,freq=1.0), product of:
             0.07711821 = queryWeight, product of:
               1.1066841 = boost
               3.6597328 = idf(docFreq=3107, maxDocs=44421)
               0.019040743 = queryNorm
             0.28591663 = fieldWeight in 45, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6597328 = idf(docFreq=3107, maxDocs=44421)
               0.078125 = fieldNorm(doc=45)
         0.14357932 = weight(abstract_txt:reusable in 45) [ClassicSimilarity], result of:
           0.14357932 = score(doc=45,freq=1.0), product of:
             0.21344073 = queryWeight, product of:
               1.3018737 = boost
               8.610425 = idf(docFreq=21, maxDocs=44421)
               0.019040743 = queryNorm
             0.67268944 = fieldWeight in 45, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.610425 = idf(docFreq=21, maxDocs=44421)
               0.078125 = fieldNorm(doc=45)
         0.021709649 = weight(abstract_txt:this in 45) [ClassicSimilarity], result of:
           0.021709649 = score(doc=45,freq=3.0), product of:
             0.06667517 = queryWeight, product of:
               1.4552655 = boost
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.019040743 = queryNorm
             0.3256032 = fieldWeight in 45, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.078125 = fieldNorm(doc=45)
         0.048815224 = weight(abstract_txt:system in 45) [ClassicSimilarity], result of:
           0.048815224 = score(doc=45,freq=2.0), product of:
             0.13099746 = queryWeight, product of:
               2.039819 = boost
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.019040743 = queryNorm
             0.37264252 = fieldWeight in 45, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.078125 = fieldNorm(doc=45)
         0.05050161 = weight(abstract_txt:systems in 45) [ClassicSimilarity], result of:
           0.05050161 = score(doc=45,freq=2.0), product of:
             0.13399734 = queryWeight, product of:
               2.0630429 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.019040743 = queryNorm
             0.37688518 = fieldWeight in 45, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.078125 = fieldNorm(doc=45)
         0.18993305 = weight(abstract_txt:question in 45) [ClassicSimilarity], result of:
           0.18993305 = score(doc=45,freq=4.0), product of:
             0.23368822 = queryWeight, product of:
               2.3594415 = boost
               5.2016807 = idf(docFreq=664, maxDocs=44421)
               0.019040743 = queryNorm
             0.8127626 = fieldWeight in 45, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.2016807 = idf(docFreq=664, maxDocs=44421)
               0.078125 = fieldNorm(doc=45)
         0.3948685 = weight(abstract_txt:answering in 45) [ClassicSimilarity], result of:
           0.3948685 = score(doc=45,freq=4.0), product of:
             0.38066077 = queryWeight, product of:
               3.0113406 = boost
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.019040743 = queryNorm
             1.0373238 = fieldWeight in 45, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.078125 = fieldNorm(doc=45)
         0.10109162 = weight(abstract_txt:evaluation in 45) [ClassicSimilarity], result of:
           0.10109162 = score(doc=45,freq=1.0), product of:
             0.28885695 = queryWeight, product of:
               3.386543 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.019040743 = queryNorm
             0.34997123 = fieldWeight in 45, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.078125 = fieldNorm(doc=45)
       0.32 = coord(8/25)
   

5. Blandford, A.; Adams, A.; Attfield, S.; Buchanan, G.; Gow, J.; Makri, S.; Rimmer, J.; Warwick, C.: ¬The PRET A Rapporter framework : evaluating digital libraries from the perspective of information work (2008) 0.24

   0.24393593 = sum of:
     0.24393593 = product of:
       0.7622998 = sum of:
         0.06507068 = weight(abstract_txt:conduct in 3021) [ClassicSimilarity], result of:
           0.06507068 = score(doc=3021,freq=1.0), product of:
             0.12593302 = queryWeight, product of:
               6.613871 = idf(docFreq=161, maxDocs=44421)
               0.019040743 = queryNorm
             0.5167087 = fieldWeight in 3021, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.613871 = idf(docFreq=161, maxDocs=44421)
               0.078125 = fieldNorm(doc=3021)
         0.077480875 = weight(abstract_txt:conducting in 3021) [ClassicSimilarity], result of:
           0.077480875 = score(doc=3021,freq=1.0), product of:
             0.14147478 = queryWeight, product of:
               1.0599117 = boost
               7.01012 = idf(docFreq=108, maxDocs=44421)
               0.019040743 = queryNorm
             0.5476656 = fieldWeight in 3021, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.01012 = idf(docFreq=108, maxDocs=44421)
               0.078125 = fieldNorm(doc=3021)
         0.022049379 = weight(abstract_txt:different in 3021) [ClassicSimilarity], result of:
           0.022049379 = score(doc=3021,freq=1.0), product of:
             0.07711821 = queryWeight, product of:
               1.1066841 = boost
               3.6597328 = idf(docFreq=3107, maxDocs=44421)
               0.019040743 = queryNorm
             0.28591663 = fieldWeight in 3021, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6597328 = idf(docFreq=3107, maxDocs=44421)
               0.078125 = fieldNorm(doc=3021)
         0.017725853 = weight(abstract_txt:this in 3021) [ClassicSimilarity], result of:
           0.017725853 = score(doc=3021,freq=2.0), product of:
             0.06667517 = queryWeight, product of:
               1.4552655 = boost
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.019040743 = queryNorm
             0.26585388 = fieldWeight in 3021, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.078125 = fieldNorm(doc=3021)
         0.059786197 = weight(abstract_txt:system in 3021) [ClassicSimilarity], result of:
           0.059786197 = score(doc=3021,freq=3.0), product of:
             0.13099746 = queryWeight, product of:
               2.039819 = boost
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.019040743 = queryNorm
             0.45639202 = fieldWeight in 3021, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.078125 = fieldNorm(doc=3021)
         0.07985005 = weight(abstract_txt:systems in 3021) [ClassicSimilarity], result of:
           0.07985005 = score(doc=3021,freq=5.0), product of:
             0.13399734 = queryWeight, product of:
               2.0630429 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.019040743 = queryNorm
             0.59590775 = fieldWeight in 3021, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.078125 = fieldNorm(doc=3021)
         0.15440646 = weight(abstract_txt:interactive in 3021) [ClassicSimilarity], result of:
           0.15440646 = score(doc=3021,freq=1.0), product of:
             0.35564196 = queryWeight, product of:
               3.360986 = boost
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.019040743 = queryNorm
             0.43416265 = fieldWeight in 3021, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.078125 = fieldNorm(doc=3021)
         0.28593028 = weight(abstract_txt:evaluation in 3021) [ClassicSimilarity], result of:
           0.28593028 = score(doc=3021,freq=8.0), product of:
             0.28885695 = queryWeight, product of:
               3.386543 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.019040743 = queryNorm
             0.9898681 = fieldWeight in 3021, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.078125 = fieldNorm(doc=3021)
       0.32 = coord(8/25)