Document (#36745)

Author

Sun, Y.
Kantor, P.B.
Morse, E.L.

Title

Using cross-evaluation to evaluate interactive QA systems

Source

Journal of the American Society for Information Science and Technology. 62(2011) no.9, S.1653-1665

Year

2011

Abstract

In this article, we report on an experiment to assess the possibility of rigorous evaluation of interactive question-answering (QA) systems using the cross-evaluation method. This method takes into account the effects of tasks and context, and of the users of the systems. Statistical techniques are used to remove these effects, isolating the effect of the system itself. The results show that this approach yields meaningful measurements of the impact of systems on user task performance, using a surprisingly small number of subjects and without relying on predetermined judgments of the quality, or of the relevance of materials. We conclude that the method is indeed effective for comparing end-to-end QA systems, and for comparing interactive systems with high efficiency.

Similar documents (author)

1. Morse, P.M.: Search theory and browsing (1970) 2.44

   2.4380355 = sum of:
     2.4380355 = product of:
       4.876071 = sum of:
         4.876071 = weight(author_txt:morse in 1447) [ClassicSimilarity], result of:
           4.876071 = score(doc=1447,freq=1.0), product of:
             0.81081814 = queryWeight, product of:
               1.1769909 = boost
               9.622026 = idf(docFreq=7, maxDocs=44421)
               0.071595185 = queryNorm
             6.0137663 = fieldWeight in 1447, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.622026 = idf(docFreq=7, maxDocs=44421)
               0.625 = fieldNorm(doc=1447)
       0.5 = coord(1/2)
   

2. Morse, P.M.: Browsing and search theory (1973) 2.44

   2.4380355 = sum of:
     2.4380355 = product of:
       4.876071 = sum of:
         4.876071 = weight(author_txt:morse in 4339) [ClassicSimilarity], result of:
           4.876071 = score(doc=4339,freq=1.0), product of:
             0.81081814 = queryWeight, product of:
               1.1769909 = boost
               9.622026 = idf(docFreq=7, maxDocs=44421)
               0.071595185 = queryNorm
             6.0137663 = fieldWeight in 4339, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.622026 = idf(docFreq=7, maxDocs=44421)
               0.625 = fieldNorm(doc=4339)
       0.5 = coord(1/2)
   

3. Morse, T.: Mapping relationships : examining bibliographic relationships in sheet maps from Tillett to RDA (2012) 2.44

   2.4380355 = sum of:
     2.4380355 = product of:
       4.876071 = sum of:
         4.876071 = weight(author_txt:morse in 2896) [ClassicSimilarity], result of:
           4.876071 = score(doc=2896,freq=1.0), product of:
             0.81081814 = queryWeight, product of:
               1.1769909 = boost
               9.622026 = idf(docFreq=7, maxDocs=44421)
               0.071595185 = queryNorm
             6.0137663 = fieldWeight in 2896, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.622026 = idf(docFreq=7, maxDocs=44421)
               0.625 = fieldNorm(doc=2896)
       0.5 = coord(1/2)
   

4. Kantor, P.B.: ¬The Adaptive Network Library Interface : a historical overview and interim report (1993) 1.50

   1.4952738 = sum of:
     1.4952738 = product of:
       2.9905477 = sum of:
         2.9905477 = weight(author_txt:kantor in 6975) [ClassicSimilarity], result of:
           2.9905477 = score(doc=6975,freq=1.0), product of:
             0.5852983 = queryWeight, product of:
               8.175107 = idf(docFreq=33, maxDocs=44421)
               0.071595185 = queryNorm
             5.1094418 = fieldWeight in 6975, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.175107 = idf(docFreq=33, maxDocs=44421)
               0.625 = fieldNorm(doc=6975)
       0.5 = coord(1/2)
   

5. Kantor, P.B.: ¬The logic of weighted queries (1981) 1.50

   1.4952738 = sum of:
     1.4952738 = product of:
       2.9905477 = sum of:
         2.9905477 = weight(author_txt:kantor in 7621) [ClassicSimilarity], result of:
           2.9905477 = score(doc=7621,freq=1.0), product of:
             0.5852983 = queryWeight, product of:
               8.175107 = idf(docFreq=33, maxDocs=44421)
               0.071595185 = queryNorm
             5.1094418 = fieldWeight in 7621, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.175107 = idf(docFreq=33, maxDocs=44421)
               0.625 = fieldNorm(doc=7621)
       0.5 = coord(1/2)
   

Similar documents (content)

1. Wacholder, N.; Kelly, D.; Kantor, P.; Rittman, R.; Sun, Y.; Bai, B.; Small, S.; Yamrom, B.; Strzalkowski, T.: ¬A model for quantitative evaluation of an end-to-end question-answering system (2007) 0.68

   0.68405217 = sum of:
     0.68405217 = product of:
       1.4251087 = sum of:
         0.06772296 = weight(abstract_txt:conclude in 1435) [ClassicSimilarity], result of:
           0.06772296 = score(doc=1435,freq=1.0), product of:
             0.1369258 = queryWeight, product of:
               1.045987 = boost
               6.3308296 = idf(docFreq=214, maxDocs=44421)
               0.020677516 = queryNorm
             0.49459606 = fieldWeight in 1435, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.3308296 = idf(docFreq=214, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
         0.11044653 = weight(abstract_txt:answering in 1435) [ClassicSimilarity], result of:
           0.11044653 = score(doc=1435,freq=2.0), product of:
             0.15057495 = queryWeight, product of:
               1.0968822 = boost
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.020677516 = queryNorm
             0.7334987 = fieldWeight in 1435, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
         0.08696488 = weight(abstract_txt:judgments in 1435) [ClassicSimilarity], result of:
           0.08696488 = score(doc=1435,freq=1.0), product of:
             0.16176724 = queryWeight, product of:
               1.1369175 = boost
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.020677516 = queryNorm
             0.53759265 = fieldWeight in 1435, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
         0.022311013 = weight(abstract_txt:this in 1435) [ClassicSimilarity], result of:
           0.022311013 = score(doc=1435,freq=4.0), product of:
             0.059341878 = queryWeight, product of:
               1.1926826 = boost
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.020677516 = queryNorm
             0.37597418 = fieldWeight in 1435, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
         0.11524766 = weight(abstract_txt:surprisingly in 1435) [ClassicSimilarity], result of:
           0.11524766 = score(doc=1435,freq=1.0), product of:
             0.19517136 = queryWeight, product of:
               1.2487965 = boost
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.020677516 = queryNorm
             0.59049475 = fieldWeight in 1435, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
         0.17923564 = weight(abstract_txt:predetermined in 1435) [ClassicSimilarity], result of:
           0.17923564 = score(doc=1435,freq=1.0), product of:
             0.2619857 = queryWeight, product of:
               1.4468464 = boost
               8.757029 = idf(docFreq=18, maxDocs=44421)
               0.020677516 = queryNorm
             0.6841428 = fieldWeight in 1435, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.757029 = idf(docFreq=18, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
         0.033076897 = weight(abstract_txt:using in 1435) [ClassicSimilarity], result of:
           0.033076897 = score(doc=1435,freq=1.0), product of:
             0.12247617 = queryWeight, product of:
               1.7134446 = boost
               3.4568708 = idf(docFreq=3806, maxDocs=44421)
               0.020677516 = queryNorm
             0.27006802 = fieldWeight in 1435, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4568708 = idf(docFreq=3806, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
         0.118355125 = weight(abstract_txt:comparing in 1435) [ClassicSimilarity], result of:
           0.118355125 = score(doc=1435,freq=1.0), product of:
             0.25030106 = queryWeight, product of:
               2.0 = boost
               6.0524936 = idf(docFreq=283, maxDocs=44421)
               0.020677516 = queryNorm
             0.47285107 = fieldWeight in 1435, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.0524936 = idf(docFreq=283, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
         0.14395684 = weight(abstract_txt:evaluation in 1435) [ClassicSimilarity], result of:
           0.14395684 = score(doc=1435,freq=4.0), product of:
             0.20566954 = queryWeight, product of:
               2.22039 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.020677516 = queryNorm
             0.69994247 = fieldWeight in 1435, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
         0.14581335 = weight(abstract_txt:method in 1435) [ClassicSimilarity], result of:
           0.14581335 = score(doc=1435,freq=4.0), product of:
             0.207434 = queryWeight, product of:
               2.2298942 = boost
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.020677516 = queryNorm
             0.7029385 = fieldWeight in 1435, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
         0.27484804 = weight(abstract_txt:interactive in 1435) [ClassicSimilarity], result of:
           0.27484804 = score(doc=1435,freq=4.0), product of:
             0.3165266 = queryWeight, product of:
               2.7545419 = boost
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.020677516 = queryNorm
             0.8683253 = fieldWeight in 1435, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
         0.12712981 = weight(abstract_txt:systems in 1435) [ClassicSimilarity], result of:
           0.12712981 = score(doc=1435,freq=4.0), product of:
             0.23851919 = queryWeight, product of:
               3.38159 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.020677516 = queryNorm
             0.5329961 = fieldWeight in 1435, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.078125 = fieldNorm(doc=1435)
       0.48 = coord(12/25)
   

2. López-Ostenero, F.; Peinado, V.; Gonzalo, J.; Verdejo, F.: Interactive question answering : Is Cross-Language harder than monolingual searching? (2008) 0.23

   0.22554207 = sum of:
     0.22554207 = product of:
       0.70481896 = sum of:
         0.059177563 = weight(abstract_txt:takes in 3023) [ClassicSimilarity], result of:
           0.059177563 = score(doc=3023,freq=1.0), product of:
             0.12515053 = queryWeight, product of:
               6.0524936 = idf(docFreq=283, maxDocs=44421)
               0.020677516 = queryNorm
             0.47285107 = fieldWeight in 3023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.0524936 = idf(docFreq=283, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.07809749 = weight(abstract_txt:answering in 3023) [ClassicSimilarity], result of:
           0.07809749 = score(doc=3023,freq=1.0), product of:
             0.15057495 = queryWeight, product of:
               1.0968822 = boost
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.020677516 = queryNorm
             0.5186619 = fieldWeight in 3023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.6388726 = idf(docFreq=157, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.01577627 = weight(abstract_txt:this in 3023) [ClassicSimilarity], result of:
           0.01577627 = score(doc=3023,freq=2.0), product of:
             0.059341878 = queryWeight, product of:
               1.1926826 = boost
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.020677516 = queryNorm
             0.26585388 = fieldWeight in 3023, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.033076897 = weight(abstract_txt:using in 3023) [ClassicSimilarity], result of:
           0.033076897 = score(doc=3023,freq=1.0), product of:
             0.12247617 = queryWeight, product of:
               1.7134446 = boost
               3.4568708 = idf(docFreq=3806, maxDocs=44421)
               0.020677516 = queryNorm
             0.27006802 = fieldWeight in 3023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4568708 = idf(docFreq=3806, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.16247104 = weight(abstract_txt:cross in 3023) [ClassicSimilarity], result of:
           0.16247104 = score(doc=3023,freq=3.0), product of:
             0.2143622 = queryWeight, product of:
               1.8508565 = boost
               5.6011486 = idf(docFreq=445, maxDocs=44421)
               0.020677516 = queryNorm
             0.75792766 = fieldWeight in 3023, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.6011486 = idf(docFreq=445, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.07197842 = weight(abstract_txt:evaluation in 3023) [ClassicSimilarity], result of:
           0.07197842 = score(doc=3023,freq=1.0), product of:
             0.20566954 = queryWeight, product of:
               2.22039 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.020677516 = queryNorm
             0.34997123 = fieldWeight in 3023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.19434692 = weight(abstract_txt:interactive in 3023) [ClassicSimilarity], result of:
           0.19434692 = score(doc=3023,freq=2.0), product of:
             0.3165266 = queryWeight, product of:
               2.7545419 = boost
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.020677516 = queryNorm
             0.6139987 = fieldWeight in 3023, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
         0.08989435 = weight(abstract_txt:systems in 3023) [ClassicSimilarity], result of:
           0.08989435 = score(doc=3023,freq=2.0), product of:
             0.23851919 = queryWeight, product of:
               3.38159 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.020677516 = queryNorm
             0.37688518 = fieldWeight in 3023, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.078125 = fieldNorm(doc=3023)
       0.32 = coord(8/25)
   

3. Syn, S.Y.; Spring, M.B.: Finding subject terms for classificatory metadata from user-generated social tags (2013) 0.21

   0.20988567 = sum of:
     0.20988567 = product of:
       0.6558927 = sum of:
         0.069571905 = weight(abstract_txt:judgments in 1745) [ClassicSimilarity], result of:
           0.069571905 = score(doc=1745,freq=1.0), product of:
             0.16176724 = queryWeight, product of:
               1.1369175 = boost
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.020677516 = queryNorm
             0.43007413 = fieldWeight in 1745, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.0625 = fieldNorm(doc=1745)
         0.008924405 = weight(abstract_txt:this in 1745) [ClassicSimilarity], result of:
           0.008924405 = score(doc=1745,freq=1.0), product of:
             0.059341878 = queryWeight, product of:
               1.1926826 = boost
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.020677516 = queryNorm
             0.15038967 = fieldWeight in 1745, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.0625 = fieldNorm(doc=1745)
         0.14858782 = weight(abstract_txt:measurements in 1745) [ClassicSimilarity], result of:
           0.14858782 = score(doc=1745,freq=2.0), product of:
             0.21293488 = queryWeight, product of:
               1.3043888 = boost
               7.894805 = idf(docFreq=44, maxDocs=44421)
               0.020677516 = queryNorm
             0.69780874 = fieldWeight in 1745, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.894805 = idf(docFreq=44, maxDocs=44421)
               0.0625 = fieldNorm(doc=1745)
         0.12517156 = weight(abstract_txt:remove in 1745) [ClassicSimilarity], result of:
           0.12517156 = score(doc=1745,freq=1.0), product of:
             0.23929764 = queryWeight, product of:
               1.3827792 = boost
               8.369263 = idf(docFreq=27, maxDocs=44421)
               0.020677516 = queryNorm
             0.5230789 = fieldWeight in 1745, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.369263 = idf(docFreq=27, maxDocs=44421)
               0.0625 = fieldNorm(doc=1745)
         0.045832694 = weight(abstract_txt:using in 1745) [ClassicSimilarity], result of:
           0.045832694 = score(doc=1745,freq=3.0), product of:
             0.12247617 = queryWeight, product of:
               1.7134446 = boost
               3.4568708 = idf(docFreq=3806, maxDocs=44421)
               0.020677516 = queryNorm
             0.37421724 = fieldWeight in 1745, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.4568708 = idf(docFreq=3806, maxDocs=44421)
               0.0625 = fieldNorm(doc=1745)
         0.07504216 = weight(abstract_txt:cross in 1745) [ClassicSimilarity], result of:
           0.07504216 = score(doc=1745,freq=1.0), product of:
             0.2143622 = queryWeight, product of:
               1.8508565 = boost
               5.6011486 = idf(docFreq=445, maxDocs=44421)
               0.020677516 = queryNorm
             0.3500718 = fieldWeight in 1745, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.6011486 = idf(docFreq=445, maxDocs=44421)
               0.0625 = fieldNorm(doc=1745)
         0.0946841 = weight(abstract_txt:comparing in 1745) [ClassicSimilarity], result of:
           0.0946841 = score(doc=1745,freq=1.0), product of:
             0.25030106 = queryWeight, product of:
               2.0 = boost
               6.0524936 = idf(docFreq=283, maxDocs=44421)
               0.020677516 = queryNorm
             0.37828085 = fieldWeight in 1745, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.0524936 = idf(docFreq=283, maxDocs=44421)
               0.0625 = fieldNorm(doc=1745)
         0.08807811 = weight(abstract_txt:systems in 1745) [ClassicSimilarity], result of:
           0.08807811 = score(doc=1745,freq=3.0), product of:
             0.23851919 = queryWeight, product of:
               3.38159 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.020677516 = queryNorm
             0.36927053 = fieldWeight in 1745, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.0625 = fieldNorm(doc=1745)
       0.32 = coord(8/25)
   

4. Petrelli, D.: On the role of user-centred evaluation in the advancement of interactive information retrieval (2008) 0.14

   0.13515246 = sum of:
     0.13515246 = product of:
       0.48268735 = sum of:
         0.06339028 = weight(abstract_txt:indeed in 3026) [ClassicSimilarity], result of:
           0.06339028 = score(doc=3026,freq=1.0), product of:
             0.15203716 = queryWeight, product of:
               1.1021953 = boost
               6.6710296 = idf(docFreq=152, maxDocs=44421)
               0.020677516 = queryNorm
             0.41693935 = fieldWeight in 3026, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.6710296 = idf(docFreq=152, maxDocs=44421)
               0.0625 = fieldNorm(doc=3026)
         0.012621015 = weight(abstract_txt:this in 3026) [ClassicSimilarity], result of:
           0.012621015 = score(doc=3026,freq=2.0), product of:
             0.059341878 = queryWeight, product of:
               1.1926826 = boost
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.020677516 = queryNorm
             0.21268311 = fieldWeight in 3026, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.0625 = fieldNorm(doc=3026)
         0.12997684 = weight(abstract_txt:cross in 3026) [ClassicSimilarity], result of:
           0.12997684 = score(doc=3026,freq=3.0), product of:
             0.2143622 = queryWeight, product of:
               1.8508565 = boost
               5.6011486 = idf(docFreq=445, maxDocs=44421)
               0.020677516 = queryNorm
             0.60634214 = fieldWeight in 3026, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.6011486 = idf(docFreq=445, maxDocs=44421)
               0.0625 = fieldNorm(doc=3026)
         0.05758274 = weight(abstract_txt:evaluation in 3026) [ClassicSimilarity], result of:
           0.05758274 = score(doc=3026,freq=1.0), product of:
             0.20566954 = queryWeight, product of:
               2.22039 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.020677516 = queryNorm
             0.279977 = fieldWeight in 3026, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0625 = fieldNorm(doc=3026)
         0.05832534 = weight(abstract_txt:method in 3026) [ClassicSimilarity], result of:
           0.05832534 = score(doc=3026,freq=1.0), product of:
             0.207434 = queryWeight, product of:
               2.2298942 = boost
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.020677516 = queryNorm
             0.2811754 = fieldWeight in 3026, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.0625 = fieldNorm(doc=3026)
         0.10993922 = weight(abstract_txt:interactive in 3026) [ClassicSimilarity], result of:
           0.10993922 = score(doc=3026,freq=1.0), product of:
             0.3165266 = queryWeight, product of:
               2.7545419 = boost
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.020677516 = queryNorm
             0.34733012 = fieldWeight in 3026, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.557282 = idf(docFreq=465, maxDocs=44421)
               0.0625 = fieldNorm(doc=3026)
         0.05085192 = weight(abstract_txt:systems in 3026) [ClassicSimilarity], result of:
           0.05085192 = score(doc=3026,freq=1.0), product of:
             0.23851919 = queryWeight, product of:
               3.38159 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.020677516 = queryNorm
             0.21319844 = fieldWeight in 3026, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.0625 = fieldNorm(doc=3026)
       0.28 = coord(7/25)
   

5. Zhou, D.; Lawless, S.; Wu, X.; Zhao, W.; Liu, J.: ¬A study of user profile representation for personalized cross-language information retrieval (2016) 0.13

   0.1331172 = sum of:
     0.1331172 = product of:
       0.47541857 = sum of:
         0.012621015 = weight(abstract_txt:this in 4167) [ClassicSimilarity], result of:
           0.012621015 = score(doc=4167,freq=2.0), product of:
             0.059341878 = queryWeight, product of:
               1.1926826 = boost
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.020677516 = queryNorm
             0.21268311 = fieldWeight in 4167, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4062347 = idf(docFreq=10885, maxDocs=44421)
               0.0625 = fieldNorm(doc=4167)
         0.026461517 = weight(abstract_txt:using in 4167) [ClassicSimilarity], result of:
           0.026461517 = score(doc=4167,freq=1.0), product of:
             0.12247617 = queryWeight, product of:
               1.7134446 = boost
               3.4568708 = idf(docFreq=3806, maxDocs=44421)
               0.020677516 = queryNorm
             0.21605442 = fieldWeight in 4167, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4568708 = idf(docFreq=3806, maxDocs=44421)
               0.0625 = fieldNorm(doc=4167)
         0.12997684 = weight(abstract_txt:cross in 4167) [ClassicSimilarity], result of:
           0.12997684 = score(doc=4167,freq=3.0), product of:
             0.2143622 = queryWeight, product of:
               1.8508565 = boost
               5.6011486 = idf(docFreq=445, maxDocs=44421)
               0.020677516 = queryNorm
             0.60634214 = fieldWeight in 4167, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.6011486 = idf(docFreq=445, maxDocs=44421)
               0.0625 = fieldNorm(doc=4167)
         0.0946841 = weight(abstract_txt:comparing in 4167) [ClassicSimilarity], result of:
           0.0946841 = score(doc=4167,freq=1.0), product of:
             0.25030106 = queryWeight, product of:
               2.0 = boost
               6.0524936 = idf(docFreq=283, maxDocs=44421)
               0.020677516 = queryNorm
             0.37828085 = fieldWeight in 4167, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.0524936 = idf(docFreq=283, maxDocs=44421)
               0.0625 = fieldNorm(doc=4167)
         0.08143429 = weight(abstract_txt:evaluation in 4167) [ClassicSimilarity], result of:
           0.08143429 = score(doc=4167,freq=2.0), product of:
             0.20566954 = queryWeight, product of:
               2.22039 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.020677516 = queryNorm
             0.39594725 = fieldWeight in 4167, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0625 = fieldNorm(doc=4167)
         0.05832534 = weight(abstract_txt:method in 4167) [ClassicSimilarity], result of:
           0.05832534 = score(doc=4167,freq=1.0), product of:
             0.207434 = queryWeight, product of:
               2.2298942 = boost
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.020677516 = queryNorm
             0.2811754 = fieldWeight in 4167, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.0625 = fieldNorm(doc=4167)
         0.07191548 = weight(abstract_txt:systems in 4167) [ClassicSimilarity], result of:
           0.07191548 = score(doc=4167,freq=2.0), product of:
             0.23851919 = queryWeight, product of:
               3.38159 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.020677516 = queryNorm
             0.30150813 = fieldWeight in 4167, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.0625 = fieldNorm(doc=4167)
       0.28 = coord(7/25)