Document (#32436)

Author

Wacholder, N.
Kelly, D.
Kantor, P.
Rittman, R.
Sun, Y.
Bai, B.
Small, S.
Yamrom, B.
Strzalkowski, T.

Title

¬A model for quantitative evaluation of an end-to-end question-answering system

Source

Journal of the American Society for Information Science and Technology. 58(2007) no.8, S.1082-1099

Year

2007

Abstract

We describe a procedure for quantitative evaluation of interactive question-answering systems and illustrate it with application to the High-Quality Interactive QuestionAnswering (HITIQA) system. Our objectives were (a) to design a method to realistically and reliably assess interactive question-answering systems by comparing the quality of reports produced using different systems, (b) to conduct a pilot test of this method, and (c) to perform a formative evaluation of the HITIQA system. Far more important than the specific information gathered from this pilot evaluation is the development of (a) a protocol for evaluating an emerging technology, (b) reusable assessment instruments, and (c) the knowledge gained in conducting the evaluation. We conclude that this method, which uses a surprisingly small number of subjects and does not rely on predetermined relevance judgments, measures the impact of system change on work produced by users. Therefore this method can be used to compare the product of interactive systems that use different underlying technologies.

Object

HITIQA

Similar documents (author)

1. Kelly, D.; Wacholder, N.; Rittman, R.; Sun, Y.; Kantor, P.; Small, S.; Strzalkowski, T.: Using interview data to identify evaluation criteria for interactive, analytical question-answering systems (2007) 4.95

   4.953991 = sum of:
     4.953991 = sum of:
       0.7703651 = weight(author_txt:kantor in 332) [ClassicSimilarity], result of:
         0.7703651 = score(doc=332,freq=1.0), product of:
           0.37714344 = queryWeight, product of:
             8.1705265 = idf(docFreq=33, maxDocs=44218)
             0.046159014 = queryNorm
           2.0426316 = fieldWeight in 332, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.1705265 = idf(docFreq=33, maxDocs=44218)
             0.25 = fieldNorm(doc=332)
       0.8731459 = weight(author_txt:kelly in 332) [ClassicSimilarity], result of:
         0.8731459 = score(doc=332,freq=1.0), product of:
           0.40998378 = queryWeight, product of:
             1.0426296 = boost
             8.518833 = idf(docFreq=23, maxDocs=44218)
             0.046159014 = queryNorm
           2.1297083 = fieldWeight in 332, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.518833 = idf(docFreq=23, maxDocs=44218)
             0.25 = fieldNorm(doc=332)
       1.0038497 = weight(author_txt:small in 332) [ClassicSimilarity], result of:
         1.0038497 = score(doc=332,freq=1.0), product of:
           0.44994003 = queryWeight, product of:
             1.092255 = boost
             8.924298 = idf(docFreq=15, maxDocs=44218)
             0.046159014 = queryNorm
           2.2310746 = fieldWeight in 332, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.924298 = idf(docFreq=15, maxDocs=44218)
             0.25 = fieldNorm(doc=332)
       1.1356752 = weight(author_txt:strzalkowski in 332) [ClassicSimilarity], result of:
         1.1356752 = score(doc=332,freq=1.0), product of:
           0.48851538 = queryWeight, product of:
             1.1381142 = boost
             9.298992 = idf(docFreq=10, maxDocs=44218)
             0.046159014 = queryNorm
           2.324748 = fieldWeight in 332, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.298992 = idf(docFreq=10, maxDocs=44218)
             0.25 = fieldNorm(doc=332)
       1.1709547 = weight(author_txt:wacholder in 332) [ClassicSimilarity], result of:
         1.1709547 = score(doc=332,freq=1.0), product of:
           0.49858078 = queryWeight, product of:
             1.1497793 = boost
             9.394302 = idf(docFreq=9, maxDocs=44218)
             0.046159014 = queryNorm
           2.3485756 = fieldWeight in 332, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.394302 = idf(docFreq=9, maxDocs=44218)
             0.25 = fieldNorm(doc=332)
   

2. Ng, K.B.; Kantor, P.B.; Strzalkowski, T.; Wacholder, N.; Tang, R.; Bai, B.; Rittman,; Song, P.; Sun, Y.: Automated judgment of document qualities (2006) 1.62

   1.6154224 = sum of:
     1.6154224 = product of:
       2.6923704 = sum of:
         0.6740695 = weight(author_txt:kantor in 182) [ClassicSimilarity], result of:
           0.6740695 = score(doc=182,freq=1.0), product of:
             0.37714344 = queryWeight, product of:
               8.1705265 = idf(docFreq=33, maxDocs=44218)
               0.046159014 = queryNorm
             1.7873027 = fieldWeight in 182, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.1705265 = idf(docFreq=33, maxDocs=44218)
               0.21875 = fieldNorm(doc=182)
         0.9937157 = weight(author_txt:strzalkowski in 182) [ClassicSimilarity], result of:
           0.9937157 = score(doc=182,freq=1.0), product of:
             0.48851538 = queryWeight, product of:
               1.1381142 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.046159014 = queryNorm
             2.0341544 = fieldWeight in 182, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.21875 = fieldNorm(doc=182)
         1.0245854 = weight(author_txt:wacholder in 182) [ClassicSimilarity], result of:
           1.0245854 = score(doc=182,freq=1.0), product of:
             0.49858078 = queryWeight, product of:
               1.1497793 = boost
               9.394302 = idf(docFreq=9, maxDocs=44218)
               0.046159014 = queryNorm
             2.0550036 = fieldWeight in 182, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.394302 = idf(docFreq=9, maxDocs=44218)
               0.21875 = fieldNorm(doc=182)
       0.6 = coord(3/5)
   

3. Wacholder, N.: Interactive query formulation (2011) 0.59

   0.58547735 = sum of:
     0.58547735 = product of:
       2.9273868 = sum of:
         2.9273868 = weight(author_txt:wacholder in 4196) [ClassicSimilarity], result of:
           2.9273868 = score(doc=4196,freq=1.0), product of:
             0.49858078 = queryWeight, product of:
               1.1497793 = boost
               9.394302 = idf(docFreq=9, maxDocs=44218)
               0.046159014 = queryNorm
             5.871439 = fieldWeight in 4196, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.394302 = idf(docFreq=9, maxDocs=44218)
               0.625 = fieldNorm(doc=4196)
       0.2 = coord(1/5)
   

4. Strzalkowski, T.: Natural language information retrieval (1995) 0.57

   0.5678376 = sum of:
     0.5678376 = product of:
       2.8391879 = sum of:
         2.8391879 = weight(author_txt:strzalkowski in 1914) [ClassicSimilarity], result of:
           2.8391879 = score(doc=1914,freq=1.0), product of:
             0.48851538 = queryWeight, product of:
               1.1381142 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.046159014 = queryNorm
             5.81187 = fieldWeight in 1914, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.625 = fieldNorm(doc=1914)
       0.2 = coord(1/5)
   

5. Strzalkowski, T.: Reversible grammar in natural langugae processing (19??) 0.57

   0.5678376 = sum of:
     0.5678376 = product of:
       2.8391879 = sum of:
         2.8391879 = weight(author_txt:strzalkowski in 5293) [ClassicSimilarity], result of:
           2.8391879 = score(doc=5293,freq=1.0), product of:
             0.48851538 = queryWeight, product of:
               1.1381142 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.046159014 = queryNorm
             5.81187 = fieldWeight in 5293, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.625 = fieldNorm(doc=5293)
       0.2 = coord(1/5)
   

Similar documents (content)

1. Sun, Y.; Kantor, P.B.; Morse, E.L.: Using cross-evaluation to evaluate interactive QA systems (2011) 0.62

   0.62481016 = sum of:
     0.62481016 = product of:
       1.3016878 = sum of:
         0.07345063 = weight(abstract_txt:judgments in 4744) [ClassicSimilarity], result of:
           0.07345063 = score(doc=4744,freq=1.0), product of:
             0.13639726 = queryWeight, product of:
               1.0389754 = boost
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.019045858 = queryNorm
             0.5385052 = fieldWeight in 4744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.078125 = fieldNorm(doc=4744)
         0.09666833 = weight(abstract_txt:surprisingly in 4744) [ClassicSimilarity], result of:
           0.09666833 = score(doc=4744,freq=1.0), product of:
             0.16380663 = queryWeight, product of:
               1.1385921 = boost
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.019045858 = queryNorm
             0.5901369 = fieldWeight in 4744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.078125 = fieldNorm(doc=4744)
         0.1531822 = weight(abstract_txt:predetermined in 4744) [ClassicSimilarity], result of:
           0.1531822 = score(doc=4744,freq=1.0), product of:
             0.22264563 = queryWeight, product of:
               1.3274236 = boost
               8.806516 = idf(docFreq=17, maxDocs=44218)
               0.019045858 = queryNorm
             0.688009 = fieldWeight in 4744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.806516 = idf(docFreq=17, maxDocs=44218)
               0.078125 = fieldNorm(doc=4744)
         0.04518397 = weight(abstract_txt:quality in 4744) [ClassicSimilarity], result of:
           0.04518397 = score(doc=4744,freq=1.0), product of:
             0.124301076 = queryWeight, product of:
               1.402668 = boost
               4.6528544 = idf(docFreq=1145, maxDocs=44218)
               0.019045858 = queryNorm
             0.36350426 = fieldWeight in 4744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6528544 = idf(docFreq=1145, maxDocs=44218)
               0.078125 = fieldNorm(doc=4744)
         0.021832224 = weight(abstract_txt:this in 4744) [ClassicSimilarity], result of:
           0.021832224 = score(doc=4744,freq=3.0), product of:
             0.06686316 = queryWeight, product of:
               1.4548756 = boost
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.019045858 = queryNorm
             0.32652098 = fieldWeight in 4744, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.078125 = fieldNorm(doc=4744)
         0.034406286 = weight(abstract_txt:system in 4744) [ClassicSimilarity], result of:
           0.034406286 = score(doc=4744,freq=1.0), product of:
             0.13059318 = queryWeight, product of:
               2.033259 = boost
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.019045858 = queryNorm
             0.2634616 = fieldWeight in 4744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.078125 = fieldNorm(doc=4744)
         0.0796749 = weight(abstract_txt:systems in 4744) [ClassicSimilarity], result of:
           0.0796749 = score(doc=4744,freq=5.0), product of:
             0.13367586 = queryWeight, product of:
               2.0571167 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.019045858 = queryNorm
             0.5960306 = fieldWeight in 4744, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.078125 = fieldNorm(doc=4744)
         0.09502779 = weight(abstract_txt:question in 4744) [ClassicSimilarity], result of:
           0.09502779 = score(doc=4744,freq=1.0), product of:
             0.2335695 = queryWeight, product of:
               2.3548927 = boost
               5.207682 = idf(docFreq=657, maxDocs=44218)
               0.019045858 = queryNorm
             0.40685016 = fieldWeight in 4744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.207682 = idf(docFreq=657, maxDocs=44218)
               0.078125 = fieldNorm(doc=4744)
         0.14168672 = weight(abstract_txt:method in 4744) [ClassicSimilarity], result of:
           0.14168672 = score(doc=4744,freq=3.0), product of:
             0.2326346 = queryWeight, product of:
               2.7137485 = boost
               4.50095 = idf(docFreq=1333, maxDocs=44218)
               0.019045858 = queryNorm
             0.60905266 = fieldWeight in 4744, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.50095 = idf(docFreq=1333, maxDocs=44218)
               0.078125 = fieldNorm(doc=4744)
         0.1999318 = weight(abstract_txt:answering in 4744) [ClassicSimilarity], result of:
           0.1999318 = score(doc=4744,freq=1.0), product of:
             0.38350436 = queryWeight, product of:
               3.0175066 = boost
               6.6730065 = idf(docFreq=151, maxDocs=44218)
               0.019045858 = queryNorm
             0.5213286 = fieldWeight in 4744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.6730065 = idf(docFreq=151, maxDocs=44218)
               0.078125 = fieldNorm(doc=4744)
         0.21746425 = weight(abstract_txt:interactive in 4744) [ClassicSimilarity], result of:
           0.21746425 = score(doc=4744,freq=2.0), product of:
             0.35433215 = queryWeight, product of:
               3.349174 = boost
               5.5548496 = idf(docFreq=464, maxDocs=44218)
               0.019045858 = queryNorm
             0.61372995 = fieldWeight in 4744, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5548496 = idf(docFreq=464, maxDocs=44218)
               0.078125 = fieldNorm(doc=4744)
         0.14317882 = weight(abstract_txt:evaluation in 4744) [ClassicSimilarity], result of:
           0.14317882 = score(doc=4744,freq=2.0), product of:
             0.28887358 = queryWeight, product of:
               3.3809702 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.019045858 = queryNorm
             0.49564525 = fieldWeight in 4744, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.078125 = fieldNorm(doc=4744)
       0.48 = coord(12/25)
   

2. Kelly, D.; Wacholder, N.; Rittman, R.; Sun, Y.; Kantor, P.; Small, S.; Strzalkowski, T.: Using interview data to identify evaluation criteria for interactive, analytical question-answering systems (2007) 0.35

   0.34654891 = sum of:
     0.34654891 = product of:
       1.0829654 = sum of:
         0.04518397 = weight(abstract_txt:quality in 332) [ClassicSimilarity], result of:
           0.04518397 = score(doc=332,freq=1.0), product of:
             0.124301076 = queryWeight, product of:
               1.402668 = boost
               4.6528544 = idf(docFreq=1145, maxDocs=44218)
               0.019045858 = queryNorm
             0.36350426 = fieldWeight in 332, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6528544 = idf(docFreq=1145, maxDocs=44218)
               0.078125 = fieldNorm(doc=332)
         0.017825937 = weight(abstract_txt:this in 332) [ClassicSimilarity], result of:
           0.017825937 = score(doc=332,freq=2.0), product of:
             0.06686316 = queryWeight, product of:
               1.4548756 = boost
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.019045858 = queryNorm
             0.2666033 = fieldWeight in 332, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.078125 = fieldNorm(doc=332)
         0.048657842 = weight(abstract_txt:system in 332) [ClassicSimilarity], result of:
           0.048657842 = score(doc=332,freq=2.0), product of:
             0.13059318 = queryWeight, product of:
               2.033259 = boost
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.019045858 = queryNorm
             0.372591 = fieldWeight in 332, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.078125 = fieldNorm(doc=332)
         0.071263395 = weight(abstract_txt:systems in 332) [ClassicSimilarity], result of:
           0.071263395 = score(doc=332,freq=4.0), product of:
             0.13367586 = queryWeight, product of:
               2.0571167 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.019045858 = queryNorm
             0.53310597 = fieldWeight in 332, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.078125 = fieldNorm(doc=332)
         0.1343896 = weight(abstract_txt:question in 332) [ClassicSimilarity], result of:
           0.1343896 = score(doc=332,freq=2.0), product of:
             0.2335695 = queryWeight, product of:
               2.3548927 = boost
               5.207682 = idf(docFreq=657, maxDocs=44218)
               0.019045858 = queryNorm
             0.57537305 = fieldWeight in 332, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.207682 = idf(docFreq=657, maxDocs=44218)
               0.078125 = fieldNorm(doc=332)
         0.2827463 = weight(abstract_txt:answering in 332) [ClassicSimilarity], result of:
           0.2827463 = score(doc=332,freq=2.0), product of:
             0.38350436 = queryWeight, product of:
               3.0175066 = boost
               6.6730065 = idf(docFreq=151, maxDocs=44218)
               0.019045858 = queryNorm
             0.73727006 = fieldWeight in 332, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.6730065 = idf(docFreq=151, maxDocs=44218)
               0.078125 = fieldNorm(doc=332)
         0.3075409 = weight(abstract_txt:interactive in 332) [ClassicSimilarity], result of:
           0.3075409 = score(doc=332,freq=4.0), product of:
             0.35433215 = queryWeight, product of:
               3.349174 = boost
               5.5548496 = idf(docFreq=464, maxDocs=44218)
               0.019045858 = queryNorm
             0.86794525 = fieldWeight in 332, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.5548496 = idf(docFreq=464, maxDocs=44218)
               0.078125 = fieldNorm(doc=332)
         0.17535752 = weight(abstract_txt:evaluation in 332) [ClassicSimilarity], result of:
           0.17535752 = score(doc=332,freq=3.0), product of:
             0.28887358 = queryWeight, product of:
               3.3809702 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.019045858 = queryNorm
             0.607039 = fieldWeight in 332, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.078125 = fieldNorm(doc=332)
       0.32 = coord(8/25)
   

3. López-Ostenero, F.; Peinado, V.; Gonzalo, J.; Verdejo, F.: Interactive question answering : Is Cross-Language harder than monolingual searching? (2008) 0.31

   0.3142919 = sum of:
     0.3142919 = product of:
       0.87303305 = sum of:
         0.022091554 = weight(abstract_txt:different in 2023) [ClassicSimilarity], result of:
           0.022091554 = score(doc=2023,freq=1.0), product of:
             0.077144116 = queryWeight, product of:
               1.1050167 = boost
               3.6655018 = idf(docFreq=3075, maxDocs=44218)
               0.019045858 = queryNorm
             0.28636733 = fieldWeight in 2023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6655018 = idf(docFreq=3075, maxDocs=44218)
               0.078125 = fieldNorm(doc=2023)
         0.017825937 = weight(abstract_txt:this in 2023) [ClassicSimilarity], result of:
           0.017825937 = score(doc=2023,freq=2.0), product of:
             0.06686316 = queryWeight, product of:
               1.4548756 = boost
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.019045858 = queryNorm
             0.2666033 = fieldWeight in 2023, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.078125 = fieldNorm(doc=2023)
         0.09529007 = weight(abstract_txt:quantitative in 2023) [ClassicSimilarity], result of:
           0.09529007 = score(doc=2023,freq=1.0), product of:
             0.20441702 = queryWeight, product of:
               1.7987714 = boost
               5.9667873 = idf(docFreq=307, maxDocs=44218)
               0.019045858 = queryNorm
             0.46615526 = fieldWeight in 2023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.9667873 = idf(docFreq=307, maxDocs=44218)
               0.078125 = fieldNorm(doc=2023)
         0.034406286 = weight(abstract_txt:system in 2023) [ClassicSimilarity], result of:
           0.034406286 = score(doc=2023,freq=1.0), product of:
             0.13059318 = queryWeight, product of:
               2.033259 = boost
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.019045858 = queryNorm
             0.2634616 = fieldWeight in 2023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.078125 = fieldNorm(doc=2023)
         0.050390836 = weight(abstract_txt:systems in 2023) [ClassicSimilarity], result of:
           0.050390836 = score(doc=2023,freq=2.0), product of:
             0.13367586 = queryWeight, product of:
               2.0571167 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.019045858 = queryNorm
             0.37696287 = fieldWeight in 2023, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.078125 = fieldNorm(doc=2023)
         0.1343896 = weight(abstract_txt:question in 2023) [ClassicSimilarity], result of:
           0.1343896 = score(doc=2023,freq=2.0), product of:
             0.2335695 = queryWeight, product of:
               2.3548927 = boost
               5.207682 = idf(docFreq=657, maxDocs=44218)
               0.019045858 = queryNorm
             0.57537305 = fieldWeight in 2023, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.207682 = idf(docFreq=657, maxDocs=44218)
               0.078125 = fieldNorm(doc=2023)
         0.1999318 = weight(abstract_txt:answering in 2023) [ClassicSimilarity], result of:
           0.1999318 = score(doc=2023,freq=1.0), product of:
             0.38350436 = queryWeight, product of:
               3.0175066 = boost
               6.6730065 = idf(docFreq=151, maxDocs=44218)
               0.019045858 = queryNorm
             0.5213286 = fieldWeight in 2023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.6730065 = idf(docFreq=151, maxDocs=44218)
               0.078125 = fieldNorm(doc=2023)
         0.21746425 = weight(abstract_txt:interactive in 2023) [ClassicSimilarity], result of:
           0.21746425 = score(doc=2023,freq=2.0), product of:
             0.35433215 = queryWeight, product of:
               3.349174 = boost
               5.5548496 = idf(docFreq=464, maxDocs=44218)
               0.019045858 = queryNorm
             0.61372995 = fieldWeight in 2023, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5548496 = idf(docFreq=464, maxDocs=44218)
               0.078125 = fieldNorm(doc=2023)
         0.10124271 = weight(abstract_txt:evaluation in 2023) [ClassicSimilarity], result of:
           0.10124271 = score(doc=2023,freq=1.0), product of:
             0.28887358 = queryWeight, product of:
               3.3809702 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.019045858 = queryNorm
             0.35047412 = fieldWeight in 2023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.078125 = fieldNorm(doc=2023)
       0.36 = coord(9/25)
   

4. Lin, J.; Katz, B.: Building a reusable test collection for question answering (2006) 0.31

   0.31266618 = sum of:
     0.31266618 = product of:
       0.97708184 = sum of:
         0.022091554 = weight(abstract_txt:different in 5045) [ClassicSimilarity], result of:
           0.022091554 = score(doc=5045,freq=1.0), product of:
             0.077144116 = queryWeight, product of:
               1.1050167 = boost
               3.6655018 = idf(docFreq=3075, maxDocs=44218)
               0.019045858 = queryNorm
             0.28636733 = fieldWeight in 5045, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6655018 = idf(docFreq=3075, maxDocs=44218)
               0.078125 = fieldNorm(doc=5045)
         0.14294745 = weight(abstract_txt:reusable in 5045) [ClassicSimilarity], result of:
           0.14294745 = score(doc=5045,freq=1.0), product of:
             0.21261452 = queryWeight, product of:
               1.297176 = boost
               8.6058445 = idf(docFreq=21, maxDocs=44218)
               0.019045858 = queryNorm
             0.6723316 = fieldWeight in 5045, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.6058445 = idf(docFreq=21, maxDocs=44218)
               0.078125 = fieldNorm(doc=5045)
         0.021832224 = weight(abstract_txt:this in 5045) [ClassicSimilarity], result of:
           0.021832224 = score(doc=5045,freq=3.0), product of:
             0.06686316 = queryWeight, product of:
               1.4548756 = boost
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.019045858 = queryNorm
             0.32652098 = fieldWeight in 5045, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.078125 = fieldNorm(doc=5045)
         0.048657842 = weight(abstract_txt:system in 5045) [ClassicSimilarity], result of:
           0.048657842 = score(doc=5045,freq=2.0), product of:
             0.13059318 = queryWeight, product of:
               2.033259 = boost
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.019045858 = queryNorm
             0.372591 = fieldWeight in 5045, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.078125 = fieldNorm(doc=5045)
         0.050390836 = weight(abstract_txt:systems in 5045) [ClassicSimilarity], result of:
           0.050390836 = score(doc=5045,freq=2.0), product of:
             0.13367586 = queryWeight, product of:
               2.0571167 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.019045858 = queryNorm
             0.37696287 = fieldWeight in 5045, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.078125 = fieldNorm(doc=5045)
         0.19005558 = weight(abstract_txt:question in 5045) [ClassicSimilarity], result of:
           0.19005558 = score(doc=5045,freq=4.0), product of:
             0.2335695 = queryWeight, product of:
               2.3548927 = boost
               5.207682 = idf(docFreq=657, maxDocs=44218)
               0.019045858 = queryNorm
             0.8137003 = fieldWeight in 5045, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.207682 = idf(docFreq=657, maxDocs=44218)
               0.078125 = fieldNorm(doc=5045)
         0.3998636 = weight(abstract_txt:answering in 5045) [ClassicSimilarity], result of:
           0.3998636 = score(doc=5045,freq=4.0), product of:
             0.38350436 = queryWeight, product of:
               3.0175066 = boost
               6.6730065 = idf(docFreq=151, maxDocs=44218)
               0.019045858 = queryNorm
             1.0426573 = fieldWeight in 5045, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.6730065 = idf(docFreq=151, maxDocs=44218)
               0.078125 = fieldNorm(doc=5045)
         0.10124271 = weight(abstract_txt:evaluation in 5045) [ClassicSimilarity], result of:
           0.10124271 = score(doc=5045,freq=1.0), product of:
             0.28887358 = queryWeight, product of:
               3.3809702 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.019045858 = queryNorm
             0.35047412 = fieldWeight in 5045, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.078125 = fieldNorm(doc=5045)
       0.32 = coord(8/25)
   

5. Blandford, A.; Adams, A.; Attfield, S.; Buchanan, G.; Gow, J.; Makri, S.; Rimmer, J.; Warwick, C.: ¬The PRET A Rapporter framework : evaluating digital libraries from the perspective of information work (2008) 0.24

   0.24400945 = sum of:
     0.24400945 = product of:
       0.76252955 = sum of:
         0.065490715 = weight(abstract_txt:conduct in 2021) [ClassicSimilarity], result of:
           0.065490715 = score(doc=2021,freq=1.0), product of:
             0.12635578 = queryWeight, product of:
               6.634292 = idf(docFreq=157, maxDocs=44218)
               0.019045858 = queryNorm
             0.51830405 = fieldWeight in 2021, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.634292 = idf(docFreq=157, maxDocs=44218)
               0.078125 = fieldNorm(doc=2021)
         0.07772493 = weight(abstract_txt:conducting in 2021) [ClassicSimilarity], result of:
           0.07772493 = score(doc=2021,freq=1.0), product of:
             0.14163879 = queryWeight, product of:
               1.0587503 = boost
               7.0240583 = idf(docFreq=106, maxDocs=44218)
               0.019045858 = queryNorm
             0.5487546 = fieldWeight in 2021, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.0240583 = idf(docFreq=106, maxDocs=44218)
               0.078125 = fieldNorm(doc=2021)
         0.022091554 = weight(abstract_txt:different in 2021) [ClassicSimilarity], result of:
           0.022091554 = score(doc=2021,freq=1.0), product of:
             0.077144116 = queryWeight, product of:
               1.1050167 = boost
               3.6655018 = idf(docFreq=3075, maxDocs=44218)
               0.019045858 = queryNorm
             0.28636733 = fieldWeight in 2021, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6655018 = idf(docFreq=3075, maxDocs=44218)
               0.078125 = fieldNorm(doc=2021)
         0.017825937 = weight(abstract_txt:this in 2021) [ClassicSimilarity], result of:
           0.017825937 = score(doc=2021,freq=2.0), product of:
             0.06686316 = queryWeight, product of:
               1.4548756 = boost
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.019045858 = queryNorm
             0.2666033 = fieldWeight in 2021, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.078125 = fieldNorm(doc=2021)
         0.05959344 = weight(abstract_txt:system in 2021) [ClassicSimilarity], result of:
           0.05959344 = score(doc=2021,freq=3.0), product of:
             0.13059318 = queryWeight, product of:
               2.033259 = boost
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.019045858 = queryNorm
             0.45632887 = fieldWeight in 2021, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.078125 = fieldNorm(doc=2021)
         0.0796749 = weight(abstract_txt:systems in 2021) [ClassicSimilarity], result of:
           0.0796749 = score(doc=2021,freq=5.0), product of:
             0.13367586 = queryWeight, product of:
               2.0571167 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.019045858 = queryNorm
             0.5960306 = fieldWeight in 2021, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.078125 = fieldNorm(doc=2021)
         0.15377045 = weight(abstract_txt:interactive in 2021) [ClassicSimilarity], result of:
           0.15377045 = score(doc=2021,freq=1.0), product of:
             0.35433215 = queryWeight, product of:
               3.349174 = boost
               5.5548496 = idf(docFreq=464, maxDocs=44218)
               0.019045858 = queryNorm
             0.43397263 = fieldWeight in 2021, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5548496 = idf(docFreq=464, maxDocs=44218)
               0.078125 = fieldNorm(doc=2021)
         0.28635764 = weight(abstract_txt:evaluation in 2021) [ClassicSimilarity], result of:
           0.28635764 = score(doc=2021,freq=8.0), product of:
             0.28887358 = queryWeight, product of:
               3.3809702 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.019045858 = queryNorm
             0.9912905 = fieldWeight in 2021, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.078125 = fieldNorm(doc=2021)
       0.32 = coord(8/25)