Document (#31345)

Author

Wu, S.
McClean, S.I.

Title

Improving high accuracy retrieval by eliminating the uneven correlation effect in data fusion

Source

Journal of the American Society for Information Science and Technology. 57(2006) no.14, S.1962-1973

Year

2006

Abstract

The aim of this research is twofold. On the one hand, high accuracy retrieval has been a concern of the information retrieval community for some time. We aim to investigate this issue via data fusion. On the other hand, the correlation among component results has been proven harmful to data fusion, but it has not been taken into account in data fusion algorithms. In the hope of achieving better performance, we propose a group of algorithms to eliminate the effect of uneven correlation among component results by assigning different weights to all component results or their combinations. Then the linear combination method or a variation is used for fusion. Extensive experimentation is carried out to evaluate the performances of these algorithms with six groups of component results, which are the top 10 systems submitted to Text REtrieval Conference (TREC) 6, 7, 8, 9, 2001, and 2002. The experimental results show that all eight data fusion methods involved outperform the best component system on average. Therefore, we demonstrate that the data fusion technique in general is effective with accurate retrieval results. The experimental results also demonstrate that all six methods presented in this article are effective for eliminating the effect of uneven correlation among component results. All of them outperform CombSum and five of them outperform CombMNZ on average.

Similar documents (content)

1. Beitzel, S.M.; Jensen, E.C.; Chowdhury, A.; Grossman, D.; Frieder, O; Goharian, N.: Fusion of effective retrieval strategies in the same information retrieval system (2004) 0.28

   0.28428534 = sum of:
     0.28428534 = product of:
       1.0153048 = sum of:
         0.03753272 = weight(abstract_txt:effective in 3502) [ClassicSimilarity], result of:
           0.03753272 = score(doc=3502,freq=2.0), product of:
             0.07022137 = queryWeight, product of:
               1.2220523 = boost
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.011878013 = queryNorm
             0.5344914 = fieldWeight in 3502, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.078125 = fieldNorm(doc=3502)
         0.03629965 = weight(abstract_txt:demonstrate in 3502) [ClassicSimilarity], result of:
           0.03629965 = score(doc=3502,freq=1.0), product of:
             0.08652486 = queryWeight, product of:
               1.3565181 = boost
               5.3699656 = idf(docFreq=561, maxDocs=44421)
               0.011878013 = queryNorm
             0.41952854 = fieldWeight in 3502, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.3699656 = idf(docFreq=561, maxDocs=44421)
               0.078125 = fieldNorm(doc=3502)
         0.023481077 = weight(abstract_txt:been in 3502) [ClassicSimilarity], result of:
           0.023481077 = score(doc=3502,freq=2.0), product of:
             0.05879924 = queryWeight, product of:
               1.3695776 = boost
               3.614442 = idf(docFreq=3251, maxDocs=44421)
               0.011878013 = queryNorm
             0.3993432 = fieldWeight in 3502, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.614442 = idf(docFreq=3251, maxDocs=44421)
               0.078125 = fieldNorm(doc=3502)
         0.06964655 = weight(abstract_txt:retrieval in 3502) [ClassicSimilarity], result of:
           0.06964655 = score(doc=3502,freq=8.0), product of:
             0.090661384 = queryWeight, product of:
               2.1955147 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.011878013 = queryNorm
             0.7682052 = fieldWeight in 3502, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.078125 = fieldNorm(doc=3502)
         0.04498738 = weight(abstract_txt:data in 3502) [ClassicSimilarity], result of:
           0.04498738 = score(doc=3502,freq=3.0), product of:
             0.09983122 = queryWeight, product of:
               2.5237656 = boost
               3.3302255 = idf(docFreq=4320, maxDocs=44421)
               0.011878013 = queryNorm
             0.45063436 = fieldWeight in 3502, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.3302255 = idf(docFreq=4320, maxDocs=44421)
               0.078125 = fieldNorm(doc=3502)
         0.03947103 = weight(abstract_txt:results in 3502) [ClassicSimilarity], result of:
           0.03947103 = score(doc=3502,freq=1.0), product of:
             0.14523733 = queryWeight, product of:
               3.5149915 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.011878013 = queryNorm
             0.2717692 = fieldWeight in 3502, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.078125 = fieldNorm(doc=3502)
         0.76388633 = weight(abstract_txt:fusion in 3502) [ClassicSimilarity], result of:
           0.76388633 = score(doc=3502,freq=4.0), product of:
             0.630804 = queryWeight, product of:
               6.8523006 = boost
               7.750224 = idf(docFreq=51, maxDocs=44421)
               0.011878013 = queryNorm
             1.2109725 = fieldWeight in 3502, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.750224 = idf(docFreq=51, maxDocs=44421)
               0.078125 = fieldNorm(doc=3502)
       0.28 = coord(7/25)
   

2. Larsen, B.; Ingwersen, P.; Lund, B.: Data fusion according to the principle of polyrepresentation (2009) 0.23

   0.22864974 = sum of:
     0.22864974 = product of:
       0.95270723 = sum of:
         0.016507998 = weight(abstract_txt:methods in 3752) [ClassicSimilarity], result of:
           0.016507998 = score(doc=3752,freq=2.0), product of:
             0.051514193 = queryWeight, product of:
               1.0466913 = boost
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.011878013 = queryNorm
             0.32045534 = fieldWeight in 3752, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3752)
         0.038542304 = weight(abstract_txt:retrieval in 3752) [ClassicSimilarity], result of:
           0.038542304 = score(doc=3752,freq=5.0), product of:
             0.090661384 = queryWeight, product of:
               2.1955147 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.011878013 = queryNorm
             0.42512372 = fieldWeight in 3752, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3752)
         0.04065492 = weight(abstract_txt:data in 3752) [ClassicSimilarity], result of:
           0.04065492 = score(doc=3752,freq=5.0), product of:
             0.09983122 = queryWeight, product of:
               2.5237656 = boost
               3.3302255 = idf(docFreq=4320, maxDocs=44421)
               0.011878013 = queryNorm
             0.40723652 = fieldWeight in 3752, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.3302255 = idf(docFreq=4320, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3752)
         0.11055901 = weight(abstract_txt:outperform in 3752) [ClassicSimilarity], result of:
           0.11055901 = score(doc=3752,freq=1.0), product of:
             0.26397762 = queryWeight, product of:
               2.9019127 = boost
               7.6584163 = idf(docFreq=56, maxDocs=44421)
               0.011878013 = queryNorm
             0.41881964 = fieldWeight in 3752, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.6584163 = idf(docFreq=56, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3752)
         0.039074324 = weight(abstract_txt:results in 3752) [ClassicSimilarity], result of:
           0.039074324 = score(doc=3752,freq=2.0), product of:
             0.14523733 = queryWeight, product of:
               3.5149915 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.011878013 = queryNorm
             0.26903775 = fieldWeight in 3752, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3752)
         0.7073687 = weight(abstract_txt:fusion in 3752) [ClassicSimilarity], result of:
           0.7073687 = score(doc=3752,freq=7.0), product of:
             0.630804 = queryWeight, product of:
               6.8523006 = boost
               7.750224 = idf(docFreq=51, maxDocs=44421)
               0.011878013 = queryNorm
             1.1213763 = fieldWeight in 3752, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               7.750224 = idf(docFreq=51, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3752)
       0.24 = coord(6/25)
   

3. Liu, X.; Yu, S.; Janssens, F.; Glänzel, W.; Moreau, Y.; Moor, B.de: Weighted hybrid clustering by combining text mining and bibliometrics on a large-scale journal database (2010) 0.23

   0.22663511 = sum of:
     0.22663511 = product of:
       0.7082347 = sum of:
         0.028882982 = weight(abstract_txt:methods in 451) [ClassicSimilarity], result of:
           0.028882982 = score(doc=451,freq=3.0), product of:
             0.051514193 = queryWeight, product of:
               1.0466913 = boost
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.011878013 = queryNorm
             0.5606801 = fieldWeight in 451, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.078125 = fieldNorm(doc=451)
         0.03629965 = weight(abstract_txt:demonstrate in 451) [ClassicSimilarity], result of:
           0.03629965 = score(doc=451,freq=1.0), product of:
             0.08652486 = queryWeight, product of:
               1.3565181 = boost
               5.3699656 = idf(docFreq=561, maxDocs=44421)
               0.011878013 = queryNorm
             0.41952854 = fieldWeight in 451, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.3699656 = idf(docFreq=561, maxDocs=44421)
               0.078125 = fieldNorm(doc=451)
         0.03685104 = weight(abstract_txt:experimental in 451) [ClassicSimilarity], result of:
           0.03685104 = score(doc=451,freq=1.0), product of:
             0.087398864 = queryWeight, product of:
               1.3633522 = boost
               5.397019 = idf(docFreq=546, maxDocs=44421)
               0.011878013 = queryNorm
             0.4216421 = fieldWeight in 451, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.397019 = idf(docFreq=546, maxDocs=44421)
               0.078125 = fieldNorm(doc=451)
         0.05596093 = weight(abstract_txt:effect in 451) [ClassicSimilarity], result of:
           0.05596093 = score(doc=451,freq=1.0), product of:
             0.13217814 = queryWeight, product of:
               2.0534334 = boost
               5.419201 = idf(docFreq=534, maxDocs=44421)
               0.011878013 = queryNorm
             0.42337507 = fieldWeight in 451, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.419201 = idf(docFreq=534, maxDocs=44421)
               0.078125 = fieldNorm(doc=451)
         0.092093885 = weight(abstract_txt:algorithms in 451) [ClassicSimilarity], result of:
           0.092093885 = score(doc=451,freq=2.0), product of:
             0.14623351 = queryWeight, product of:
               2.1598532 = boost
               5.7000527 = idf(docFreq=403, maxDocs=44421)
               0.011878013 = queryNorm
             0.6297728 = fieldWeight in 451, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.7000527 = idf(docFreq=403, maxDocs=44421)
               0.078125 = fieldNorm(doc=451)
         0.03673204 = weight(abstract_txt:data in 451) [ClassicSimilarity], result of:
           0.03673204 = score(doc=451,freq=2.0), product of:
             0.09983122 = queryWeight, product of:
               2.5237656 = boost
               3.3302255 = idf(docFreq=4320, maxDocs=44421)
               0.011878013 = queryNorm
             0.3679414 = fieldWeight in 451, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.3302255 = idf(docFreq=4320, maxDocs=44421)
               0.078125 = fieldNorm(doc=451)
         0.03947103 = weight(abstract_txt:results in 451) [ClassicSimilarity], result of:
           0.03947103 = score(doc=451,freq=1.0), product of:
             0.14523733 = queryWeight, product of:
               3.5149915 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.011878013 = queryNorm
             0.2717692 = fieldWeight in 451, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.078125 = fieldNorm(doc=451)
         0.38194317 = weight(abstract_txt:fusion in 451) [ClassicSimilarity], result of:
           0.38194317 = score(doc=451,freq=1.0), product of:
             0.630804 = queryWeight, product of:
               6.8523006 = boost
               7.750224 = idf(docFreq=51, maxDocs=44421)
               0.011878013 = queryNorm
             0.6054863 = fieldWeight in 451, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.750224 = idf(docFreq=51, maxDocs=44421)
               0.078125 = fieldNorm(doc=451)
       0.32 = coord(8/25)
   

4. Wu, S.; Li, J.; Zeng, X.; Bi, Y.: Adaptive data fusion methods in information retrieval (2014) 0.18

   0.18428934 = sum of:
     0.18428934 = product of:
       0.9214467 = sum of:
         0.033351194 = weight(abstract_txt:methods in 2500) [ClassicSimilarity], result of:
           0.033351194 = score(doc=2500,freq=4.0), product of:
             0.051514193 = queryWeight, product of:
               1.0466913 = boost
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.011878013 = queryNorm
             0.6474176 = fieldWeight in 2500, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.078125 = fieldNorm(doc=2500)
         0.023481077 = weight(abstract_txt:been in 2500) [ClassicSimilarity], result of:
           0.023481077 = score(doc=2500,freq=2.0), product of:
             0.05879924 = queryWeight, product of:
               1.3695776 = boost
               3.614442 = idf(docFreq=3251, maxDocs=44421)
               0.011878013 = queryNorm
             0.3993432 = fieldWeight in 2500, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.614442 = idf(docFreq=3251, maxDocs=44421)
               0.078125 = fieldNorm(doc=2500)
         0.042649627 = weight(abstract_txt:retrieval in 2500) [ClassicSimilarity], result of:
           0.042649627 = score(doc=2500,freq=3.0), product of:
             0.090661384 = queryWeight, product of:
               2.1955147 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.011878013 = queryNorm
             0.4704277 = fieldWeight in 2500, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.078125 = fieldNorm(doc=2500)
         0.058078453 = weight(abstract_txt:data in 2500) [ClassicSimilarity], result of:
           0.058078453 = score(doc=2500,freq=5.0), product of:
             0.09983122 = queryWeight, product of:
               2.5237656 = boost
               3.3302255 = idf(docFreq=4320, maxDocs=44421)
               0.011878013 = queryNorm
             0.5817664 = fieldWeight in 2500, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.3302255 = idf(docFreq=4320, maxDocs=44421)
               0.078125 = fieldNorm(doc=2500)
         0.76388633 = weight(abstract_txt:fusion in 2500) [ClassicSimilarity], result of:
           0.76388633 = score(doc=2500,freq=4.0), product of:
             0.630804 = queryWeight, product of:
               6.8523006 = boost
               7.750224 = idf(docFreq=51, maxDocs=44421)
               0.011878013 = queryNorm
             1.2109725 = fieldWeight in 2500, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.750224 = idf(docFreq=51, maxDocs=44421)
               0.078125 = fieldNorm(doc=2500)
       0.2 = coord(5/25)
   

5. Wu, M.; Hawking, D.; Turpin, A.; Scholer, F.: Using anchor text for homepage and topic distillation search tasks (2012) 0.17

   0.1662116 = sum of:
     0.1662116 = product of:
       0.59361285 = sum of:
         0.023106385 = weight(abstract_txt:methods in 1257) [ClassicSimilarity], result of:
           0.023106385 = score(doc=1257,freq=3.0), product of:
             0.051514193 = queryWeight, product of:
               1.0466913 = boost
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.011878013 = queryNorm
             0.44854406 = fieldWeight in 1257, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.0625 = fieldNorm(doc=1257)
         0.021231713 = weight(abstract_txt:effective in 1257) [ClassicSimilarity], result of:
           0.021231713 = score(doc=1257,freq=1.0), product of:
             0.07022137 = queryWeight, product of:
               1.2220523 = boost
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.011878013 = queryNorm
             0.302354 = fieldWeight in 1257, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.0625 = fieldNorm(doc=1257)
         0.029480834 = weight(abstract_txt:experimental in 1257) [ClassicSimilarity], result of:
           0.029480834 = score(doc=1257,freq=1.0), product of:
             0.087398864 = queryWeight, product of:
               1.3633522 = boost
               5.397019 = idf(docFreq=546, maxDocs=44421)
               0.011878013 = queryNorm
             0.33731368 = fieldWeight in 1257, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.397019 = idf(docFreq=546, maxDocs=44421)
               0.0625 = fieldNorm(doc=1257)
         0.0132829035 = weight(abstract_txt:been in 1257) [ClassicSimilarity], result of:
           0.0132829035 = score(doc=1257,freq=1.0), product of:
             0.05879924 = queryWeight, product of:
               1.3695776 = boost
               3.614442 = idf(docFreq=3251, maxDocs=44421)
               0.011878013 = queryNorm
             0.22590263 = fieldWeight in 1257, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.614442 = idf(docFreq=3251, maxDocs=44421)
               0.0625 = fieldNorm(doc=1257)
         0.019699018 = weight(abstract_txt:retrieval in 1257) [ClassicSimilarity], result of:
           0.019699018 = score(doc=1257,freq=1.0), product of:
             0.090661384 = queryWeight, product of:
               2.1955147 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.011878013 = queryNorm
             0.21728125 = fieldWeight in 1257, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0625 = fieldNorm(doc=1257)
         0.054692663 = weight(abstract_txt:results in 1257) [ClassicSimilarity], result of:
           0.054692663 = score(doc=1257,freq=3.0), product of:
             0.14523733 = queryWeight, product of:
               3.5149915 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.011878013 = queryNorm
             0.37657443 = fieldWeight in 1257, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.0625 = fieldNorm(doc=1257)
         0.43211934 = weight(abstract_txt:fusion in 1257) [ClassicSimilarity], result of:
           0.43211934 = score(doc=1257,freq=2.0), product of:
             0.630804 = queryWeight, product of:
               6.8523006 = boost
               7.750224 = idf(docFreq=51, maxDocs=44421)
               0.011878013 = queryNorm
             0.6850295 = fieldWeight in 1257, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.750224 = idf(docFreq=51, maxDocs=44421)
               0.0625 = fieldNorm(doc=1257)
       0.28 = coord(7/25)