Document (#32333)

Author

Kelly, D.
Wacholder, N.
Rittman, R.
Sun, Y.
Kantor, P.
Small, S.
Strzalkowski, T.

Title

Using interview data to identify evaluation criteria for interactive, analytical question-answering systems

Source

Journal of the American Society for Information Science and Technology. 58(2007) no.7, S.1032-1043

Year

2007

Abstract

The purpose of this work is to identify potential evaluation criteria for interactive, analytical question-answering (QA) systems by analyzing evaluative comments made by users of such a system. Qualitative data collected from intelligence analysts during interviews and focus groups were analyzed to identify common themes related to performance, use, and usability. These data were collected as part of an intensive, three-day evaluation workshop of the High-Quality Interactive Question Answering (HITIQA) system. Inductive coding and memoing were used to identify and categorize these data. Results suggest potential evaluation criteria for interactive, analytical QA systems, which can be used to guide the development and design of future systems and evaluations. This work contributes to studies of QA systems, information seeking and use behaviors, and interactive searching.

Similar documents (author)

1. Wacholder, N.; Kelly, D.; Kantor, P.; Rittman, R.; Sun, Y.; Bai, B.; Small, S.; Yamrom, B.; Strzalkowski, T.: ¬A model for quantitative evaluation of an end-to-end question-answering system (2007) 4.34

   4.3369823 = sum of:
     4.3369823 = sum of:
       0.6745119 = weight(author_txt:kantor in 1435) [ClassicSimilarity], result of:
         0.6745119 = score(doc=1435,freq=1.0), product of:
           0.37717953 = queryWeight, product of:
             8.175107 = idf(docFreq=33, maxDocs=44421)
             0.046137564 = queryNorm
           1.7883047 = fieldWeight in 1435, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.175107 = idf(docFreq=33, maxDocs=44421)
             0.21875 = fieldNorm(doc=1435)
       0.76445156 = weight(author_txt:kelly in 1435) [ClassicSimilarity], result of:
         0.76445156 = score(doc=1435,freq=1.0), product of:
           0.41000426 = queryWeight, product of:
             1.0426058 = boost
             8.523414 = idf(docFreq=23, maxDocs=44421)
             0.046137564 = queryNorm
           1.8644967 = fieldWeight in 1435, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.523414 = idf(docFreq=23, maxDocs=44421)
             0.21875 = fieldNorm(doc=1435)
       0.87882024 = weight(author_txt:small in 1435) [ClassicSimilarity], result of:
         0.87882024 = score(doc=1435,freq=1.0), product of:
           0.44994047 = queryWeight, product of:
             1.0922033 = boost
             8.928879 = idf(docFreq=15, maxDocs=44421)
             0.046137564 = queryNorm
           1.9531922 = fieldWeight in 1435, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.928879 = idf(docFreq=15, maxDocs=44421)
             0.21875 = fieldNorm(doc=1435)
       0.9941652 = weight(author_txt:strzalkowski in 1435) [ClassicSimilarity], result of:
         0.9941652 = score(doc=1435,freq=1.0), product of:
           0.4884957 = queryWeight, product of:
             1.1380368 = boost
             9.303573 = idf(docFreq=10, maxDocs=44421)
             0.046137564 = queryNorm
           2.0351565 = fieldWeight in 1435, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.303573 = idf(docFreq=10, maxDocs=44421)
             0.21875 = fieldNorm(doc=1435)
       1.0250334 = weight(author_txt:wacholder in 1435) [ClassicSimilarity], result of:
         1.0250334 = score(doc=1435,freq=1.0), product of:
           0.4985557 = queryWeight, product of:
             1.1496954 = boost
             9.398883 = idf(docFreq=9, maxDocs=44421)
             0.046137564 = queryNorm
           2.0560057 = fieldWeight in 1435, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.398883 = idf(docFreq=9, maxDocs=44421)
             0.21875 = fieldNorm(doc=1435)
   

2. Ng, K.B.; Kantor, P.B.; Strzalkowski, T.; Wacholder, N.; Tang, R.; Bai, B.; Rittman,; Song, P.; Sun, Y.: Automated judgment of document qualities (2006) 1.62

   1.6162263 = sum of:
     1.6162263 = product of:
       2.6937103 = sum of:
         0.6745119 = weight(author_txt:kantor in 307) [ClassicSimilarity], result of:
           0.6745119 = score(doc=307,freq=1.0), product of:
             0.37717953 = queryWeight, product of:
               8.175107 = idf(docFreq=33, maxDocs=44421)
               0.046137564 = queryNorm
             1.7883047 = fieldWeight in 307, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.175107 = idf(docFreq=33, maxDocs=44421)
               0.21875 = fieldNorm(doc=307)
         0.9941652 = weight(author_txt:strzalkowski in 307) [ClassicSimilarity], result of:
           0.9941652 = score(doc=307,freq=1.0), product of:
             0.4884957 = queryWeight, product of:
               1.1380368 = boost
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.046137564 = queryNorm
             2.0351565 = fieldWeight in 307, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.21875 = fieldNorm(doc=307)
         1.0250334 = weight(author_txt:wacholder in 307) [ClassicSimilarity], result of:
           1.0250334 = score(doc=307,freq=1.0), product of:
             0.4985557 = queryWeight, product of:
               1.1496954 = boost
               9.398883 = idf(docFreq=9, maxDocs=44421)
               0.046137564 = queryNorm
             2.0560057 = fieldWeight in 307, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.398883 = idf(docFreq=9, maxDocs=44421)
               0.21875 = fieldNorm(doc=307)
       0.6 = coord(3/5)
   

3. Wacholder, N.: Interactive query formulation (2011) 0.59

   0.58573335 = sum of:
     0.58573335 = product of:
       2.9286666 = sum of:
         2.9286666 = weight(author_txt:wacholder in 196) [ClassicSimilarity], result of:
           2.9286666 = score(doc=196,freq=1.0), product of:
             0.4985557 = queryWeight, product of:
               1.1496954 = boost
               9.398883 = idf(docFreq=9, maxDocs=44421)
               0.046137564 = queryNorm
             5.874302 = fieldWeight in 196, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.398883 = idf(docFreq=9, maxDocs=44421)
               0.625 = fieldNorm(doc=196)
       0.2 = coord(1/5)
   

4. Strzalkowski, T.: Natural language information retrieval (1995) 0.57

   0.56809443 = sum of:
     0.56809443 = product of:
       2.8404722 = sum of:
         2.8404722 = weight(author_txt:strzalkowski in 1982) [ClassicSimilarity], result of:
           2.8404722 = score(doc=1982,freq=1.0), product of:
             0.4884957 = queryWeight, product of:
               1.1380368 = boost
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.046137564 = queryNorm
             5.814733 = fieldWeight in 1982, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.625 = fieldNorm(doc=1982)
       0.2 = coord(1/5)
   

5. Strzalkowski, T.: Reversible grammar in natural langugae processing (19??) 0.57

   0.56809443 = sum of:
     0.56809443 = product of:
       2.8404722 = sum of:
         2.8404722 = weight(author_txt:strzalkowski in 5361) [ClassicSimilarity], result of:
           2.8404722 = score(doc=5361,freq=1.0), product of:
             0.4884957 = queryWeight, product of:
               1.1380368 = boost
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.046137564 = queryNorm
             5.814733 = fieldWeight in 5361, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.625 = fieldNorm(doc=5361)
       0.2 = coord(1/5)
   

Similar documents (content)

1. Wacholder, N.; Kelly, D.; Kantor, P.; Rittman, R.; Sun, Y.; Bai, B.; Small, S.; Yamrom, B.; Strzalkowski, T.: ¬A model for quantitative evaluation of an end-to-end question-answering system (2007) 0.40

   0.40142334 = sum of:
     0.40142334 = product of:
       1.1150647 = sum of:
         0.016429005 = weight(abstract_txt:used in 1435) [ClassicSimilarity], result of:
           0.016429005 = score(doc=1435,freq=1.0), product of:
             0.06263888 = queryWeight, product of:
               1.0377169 = boost
               3.3572001 = idf(docFreq=4205, maxDocs=44421)
               0.017979924 = queryNorm
             0.26228127 = fieldWeight in 1435, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3572001 = idf(docFreq=4205, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
         0.02885375 = weight(abstract_txt:system in 1435) [ClassicSimilarity], result of:
           0.02885375 = score(doc=1435,freq=3.0), product of:
             0.06322142 = queryWeight, product of:
               1.0425311 = boost
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.017979924 = queryNorm
             0.45639202 = fieldWeight in 1435, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
         0.023626238 = weight(abstract_txt:work in 1435) [ClassicSimilarity], result of:
           0.023626238 = score(doc=1435,freq=1.0), product of:
             0.07980551 = queryWeight, product of:
               1.171314 = boost
               3.7894108 = idf(docFreq=2729, maxDocs=44421)
               0.017979924 = queryNorm
             0.29604772 = fieldWeight in 1435, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.7894108 = idf(docFreq=2729, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
         0.032127414 = weight(abstract_txt:were in 1435) [ClassicSimilarity], result of:
           0.032127414 = score(doc=1435,freq=1.0), product of:
             0.11212886 = queryWeight, product of:
               1.7004397 = boost
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.017979924 = queryNorm
             0.28652224 = fieldWeight in 1435, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
         0.12963335 = weight(abstract_txt:question in 1435) [ClassicSimilarity], result of:
           0.12963335 = score(doc=1435,freq=2.0), product of:
             0.22556311 = queryWeight, product of:
               2.4117737 = boost
               5.2016807 = idf(docFreq=664, maxDocs=44421)
               0.017979924 = queryNorm
             0.5747099 = fieldWeight in 1435, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.2016807 = idf(docFreq=664, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
         0.086171076 = weight(abstract_txt:systems in 1435) [ClassicSimilarity], result of:
           0.086171076 = score(doc=1435,freq=4.0), product of:
             0.161673 = queryWeight, product of:
               2.6360016 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.017979924 = queryNorm
             0.5329961 = fieldWeight in 1435, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
         0.15612285 = weight(abstract_txt:evaluation in 1435) [ClassicSimilarity], result of:
           0.15612285 = score(doc=1435,freq=4.0), product of:
             0.22305098 = queryWeight, product of:
               2.7693253 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.017979924 = queryNorm
             0.69994247 = fieldWeight in 1435, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
         0.2695062 = weight(abstract_txt:answering in 1435) [ClassicSimilarity], result of:
           0.2695062 = score(doc=1435,freq=2.0), product of:
             0.3674256 = queryWeight, product of:
               3.078132 = boost
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.017979924 = queryNorm
             0.7334987 = fieldWeight in 1435, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
         0.3725948 = weight(abstract_txt:interactive in 1435) [ClassicSimilarity], result of:
           0.3725948 = score(doc=1435,freq=4.0), product of:
             0.4290959 = queryWeight, product of:
               4.2944155 = boost
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.017979924 = queryNorm
             0.8683253 = fieldWeight in 1435, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
       0.36 = coord(9/25)
   

2. Pomerantz, J.: ¬A linguistic analysis of question taxonomies (2005) 0.24

   0.24184899 = sum of:
     0.24184899 = product of:
       0.8637464 = sum of:
         0.028351488 = weight(abstract_txt:work in 4465) [ClassicSimilarity], result of:
           0.028351488 = score(doc=4465,freq=1.0), product of:
             0.07980551 = queryWeight, product of:
               1.171314 = boost
               3.7894108 = idf(docFreq=2729, maxDocs=44421)
               0.017979924 = queryNorm
             0.35525727 = fieldWeight in 4465, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.7894108 = idf(docFreq=2729, maxDocs=44421)
               0.09375 = fieldNorm(doc=4465)
         0.051203493 = weight(abstract_txt:potential in 4465) [ClassicSimilarity], result of:
           0.051203493 = score(doc=4465,freq=1.0), product of:
             0.118353754 = queryWeight, product of:
               1.4264216 = boost
               4.61473 = idf(docFreq=1195, maxDocs=44421)
               0.017979924 = queryNorm
             0.43263093 = fieldWeight in 4465, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.61473 = idf(docFreq=1195, maxDocs=44421)
               0.09375 = fieldNorm(doc=4465)
         0.0385529 = weight(abstract_txt:were in 4465) [ClassicSimilarity], result of:
           0.0385529 = score(doc=4465,freq=1.0), product of:
             0.11212886 = queryWeight, product of:
               1.7004397 = boost
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.017979924 = queryNorm
             0.3438267 = fieldWeight in 4465, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.09375 = fieldNorm(doc=4465)
         0.24596202 = weight(abstract_txt:question in 4465) [ClassicSimilarity], result of:
           0.24596202 = score(doc=4465,freq=5.0), product of:
             0.22556311 = queryWeight, product of:
               2.4117737 = boost
               5.2016807 = idf(docFreq=664, maxDocs=44421)
               0.017979924 = queryNorm
             1.0904355 = fieldWeight in 4465, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               5.2016807 = idf(docFreq=664, maxDocs=44421)
               0.09375 = fieldNorm(doc=4465)
         0.051702645 = weight(abstract_txt:systems in 4465) [ClassicSimilarity], result of:
           0.051702645 = score(doc=4465,freq=1.0), product of:
             0.161673 = queryWeight, product of:
               2.6360016 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.017979924 = queryNorm
             0.31979766 = fieldWeight in 4465, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.09375 = fieldNorm(doc=4465)
         0.124566376 = weight(abstract_txt:identify in 4465) [ClassicSimilarity], result of:
           0.124566376 = score(doc=4465,freq=1.0), product of:
             0.2697281 = queryWeight, product of:
               3.0453343 = boost
               4.9261017 = idf(docFreq=875, maxDocs=44421)
               0.017979924 = queryNorm
             0.46182203 = fieldWeight in 4465, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.9261017 = idf(docFreq=875, maxDocs=44421)
               0.09375 = fieldNorm(doc=4465)
         0.32340744 = weight(abstract_txt:answering in 4465) [ClassicSimilarity], result of:
           0.32340744 = score(doc=4465,freq=2.0), product of:
             0.3674256 = queryWeight, product of:
               3.078132 = boost
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.017979924 = queryNorm
             0.8801985 = fieldWeight in 4465, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.09375 = fieldNorm(doc=4465)
       0.28 = coord(7/25)
   

3. Sun, Y.; Kantor, P.B.; Morse, E.L.: Using cross-evaluation to evaluate interactive QA systems (2011) 0.22

   0.21994674 = sum of:
     0.21994674 = product of:
       0.7855241 = sum of:
         0.016429005 = weight(abstract_txt:used in 744) [ClassicSimilarity], result of:
           0.016429005 = score(doc=744,freq=1.0), product of:
             0.06263888 = queryWeight, product of:
               1.0377169 = boost
               3.3572001 = idf(docFreq=4205, maxDocs=44421)
               0.017979924 = queryNorm
             0.26228127 = fieldWeight in 744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3572001 = idf(docFreq=4205, maxDocs=44421)
               0.078125 = fieldNorm(doc=744)
         0.016658721 = weight(abstract_txt:system in 744) [ClassicSimilarity], result of:
           0.016658721 = score(doc=744,freq=1.0), product of:
             0.06322142 = queryWeight, product of:
               1.0425311 = boost
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.017979924 = queryNorm
             0.26349807 = fieldWeight in 744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.078125 = fieldNorm(doc=744)
         0.091664635 = weight(abstract_txt:question in 744) [ClassicSimilarity], result of:
           0.091664635 = score(doc=744,freq=1.0), product of:
             0.22556311 = queryWeight, product of:
               2.4117737 = boost
               5.2016807 = idf(docFreq=664, maxDocs=44421)
               0.017979924 = queryNorm
             0.4063813 = fieldWeight in 744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.2016807 = idf(docFreq=664, maxDocs=44421)
               0.078125 = fieldNorm(doc=744)
         0.09634219 = weight(abstract_txt:systems in 744) [ClassicSimilarity], result of:
           0.09634219 = score(doc=744,freq=5.0), product of:
             0.161673 = queryWeight, product of:
               2.6360016 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.017979924 = queryNorm
             0.59590775 = fieldWeight in 744, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.078125 = fieldNorm(doc=744)
         0.11039553 = weight(abstract_txt:evaluation in 744) [ClassicSimilarity], result of:
           0.11039553 = score(doc=744,freq=2.0), product of:
             0.22305098 = queryWeight, product of:
               2.7693253 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.017979924 = queryNorm
             0.49493405 = fieldWeight in 744, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.078125 = fieldNorm(doc=744)
         0.19056965 = weight(abstract_txt:answering in 744) [ClassicSimilarity], result of:
           0.19056965 = score(doc=744,freq=1.0), product of:
             0.3674256 = queryWeight, product of:
               3.078132 = boost
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.017979924 = queryNorm
             0.5186619 = fieldWeight in 744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.078125 = fieldNorm(doc=744)
         0.26346433 = weight(abstract_txt:interactive in 744) [ClassicSimilarity], result of:
           0.26346433 = score(doc=744,freq=2.0), product of:
             0.4290959 = queryWeight, product of:
               4.2944155 = boost
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.017979924 = queryNorm
             0.6139987 = fieldWeight in 744, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.078125 = fieldNorm(doc=744)
       0.28 = coord(7/25)
   

4. Efthimiadis, E.N.: Interactive query expansion : a user-based evaluation in a relevance feedback environment (2000) 0.19

   0.18614718 = sum of:
     0.18614718 = product of:
       0.58171 = sum of:
         0.04538303 = weight(abstract_txt:evaluations in 6701) [ClassicSimilarity], result of:
           0.04538303 = score(doc=6701,freq=1.0), product of:
             0.12415287 = queryWeight, product of:
               1.0330474 = boost
               6.684188 = idf(docFreq=150, maxDocs=44421)
               0.017979924 = queryNorm
             0.36554152 = fieldWeight in 6701, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.684188 = idf(docFreq=150, maxDocs=44421)
               0.0546875 = fieldNorm(doc=6701)
         0.011661104 = weight(abstract_txt:system in 6701) [ClassicSimilarity], result of:
           0.011661104 = score(doc=6701,freq=1.0), product of:
             0.06322142 = queryWeight, product of:
               1.0425311 = boost
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.017979924 = queryNorm
             0.18444863 = fieldWeight in 6701, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.0546875 = fieldNorm(doc=6701)
         0.05028736 = weight(abstract_txt:were in 6701) [ClassicSimilarity], result of:
           0.05028736 = score(doc=6701,freq=5.0), product of:
             0.11212886 = queryWeight, product of:
               1.7004397 = boost
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.017979924 = queryNorm
             0.44847828 = fieldWeight in 6701, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.0546875 = fieldNorm(doc=6701)
         0.053731464 = weight(abstract_txt:collected in 6701) [ClassicSimilarity], result of:
           0.053731464 = score(doc=6701,freq=1.0), product of:
             0.17506133 = queryWeight, product of:
               1.7348104 = boost
               5.612423 = idf(docFreq=440, maxDocs=44421)
               0.017979924 = queryNorm
             0.30692938 = fieldWeight in 6701, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.612423 = idf(docFreq=440, maxDocs=44421)
               0.0546875 = fieldNorm(doc=6701)
         0.031749982 = weight(abstract_txt:data in 6701) [ClassicSimilarity], result of:
           0.031749982 = score(doc=6701,freq=2.0), product of:
             0.12327265 = queryWeight, product of:
               2.0587578 = boost
               3.3302255 = idf(docFreq=4320, maxDocs=44421)
               0.017979924 = queryNorm
             0.257559 = fieldWeight in 6701, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.3302255 = idf(docFreq=4320, maxDocs=44421)
               0.0546875 = fieldNorm(doc=6701)
         0.042652506 = weight(abstract_txt:systems in 6701) [ClassicSimilarity], result of:
           0.042652506 = score(doc=6701,freq=2.0), product of:
             0.161673 = queryWeight, product of:
               2.6360016 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.017979924 = queryNorm
             0.2638196 = fieldWeight in 6701, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.0546875 = fieldNorm(doc=6701)
         0.054643 = weight(abstract_txt:evaluation in 6701) [ClassicSimilarity], result of:
           0.054643 = score(doc=6701,freq=1.0), product of:
             0.22305098 = queryWeight, product of:
               2.7693253 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.017979924 = queryNorm
             0.24497987 = fieldWeight in 6701, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0546875 = fieldNorm(doc=6701)
         0.29160157 = weight(abstract_txt:interactive in 6701) [ClassicSimilarity], result of:
           0.29160157 = score(doc=6701,freq=5.0), product of:
             0.4290959 = queryWeight, product of:
               4.2944155 = boost
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.017979924 = queryNorm
             0.67957205 = fieldWeight in 6701, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.0546875 = fieldNorm(doc=6701)
       0.32 = coord(8/25)
   

5. López-Ostenero, F.; Peinado, V.; Gonzalo, J.; Verdejo, F.: Interactive question answering : Is Cross-Language harder than monolingual searching? (2008) 0.18

   0.17743671 = sum of:
     0.17743671 = product of:
       0.7393196 = sum of:
         0.016658721 = weight(abstract_txt:system in 3023) [ClassicSimilarity], result of:
           0.016658721 = score(doc=3023,freq=1.0), product of:
             0.06322142 = queryWeight, product of:
               1.0425311 = boost
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.017979924 = queryNorm
             0.26349807 = fieldWeight in 3023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.372775 = idf(docFreq=4140, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.12963335 = weight(abstract_txt:question in 3023) [ClassicSimilarity], result of:
           0.12963335 = score(doc=3023,freq=2.0), product of:
             0.22556311 = queryWeight, product of:
               2.4117737 = boost
               5.2016807 = idf(docFreq=664, maxDocs=44421)
               0.017979924 = queryNorm
             0.5747099 = fieldWeight in 3023, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.2016807 = idf(docFreq=664, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.060932156 = weight(abstract_txt:systems in 3023) [ClassicSimilarity], result of:
           0.060932156 = score(doc=3023,freq=2.0), product of:
             0.161673 = queryWeight, product of:
               2.6360016 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.017979924 = queryNorm
             0.37688518 = fieldWeight in 3023, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.078061424 = weight(abstract_txt:evaluation in 3023) [ClassicSimilarity], result of:
           0.078061424 = score(doc=3023,freq=1.0), product of:
             0.22305098 = queryWeight, product of:
               2.7693253 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.017979924 = queryNorm
             0.34997123 = fieldWeight in 3023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.19056965 = weight(abstract_txt:answering in 3023) [ClassicSimilarity], result of:
           0.19056965 = score(doc=3023,freq=1.0), product of:
             0.3674256 = queryWeight, product of:
               3.078132 = boost
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.017979924 = queryNorm
             0.5186619 = fieldWeight in 3023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.26346433 = weight(abstract_txt:interactive in 3023) [ClassicSimilarity], result of:
           0.26346433 = score(doc=3023,freq=2.0), product of:
             0.4290959 = queryWeight, product of:
               4.2944155 = boost
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.017979924 = queryNorm
             0.6139987 = fieldWeight in 3023, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
       0.24 = coord(6/25)