Document (#33017)

Author

Trotman, A.

Title

Choosing document structure weights

Source

Information processing and management. 41(2005) no.2, S.243-264

Year

2005

Abstract

Existing ranking schemes assume all term occurrences in a given document are of equal influence. Intuitively, terms occurring in some places should have a greater influence than those elsewhere. An occurrence in an abstract may be more important than an occurrence in the body text. Although this observation is not new, there remains the issue of finding good weights for each structure. Vector space, probability, and Okapi BM25 ranking are extended to include structure weighting. Weights are then selected for the TREC WSJ collection using a genetic algorithm. The learned weights are then tested on an evaluation set of queries. Structure weighted vector space inner product and structure weighted probabilistic retrieval show an about 5% improvement in mean average precision over their unstructured counterparts. Structure weighted BM25 shows nearly no improvement. Analysis suggests BM25 cannot be improved using structure weighting.

Similar documents (content)

1. Moura, E.S. de; Fernandes, D.; Ribeiro-Neto, B.; Silva, A.S. da; Gonçalves, M.A.: Using structural information to improve search in Web collections (2010) 0.31

   0.30706197 = sum of:
     0.30706197 = product of:
       1.0966499 = sum of:
         0.06739831 = weight(abstract_txt:occurrences in 119) [ClassicSimilarity], result of:
           0.06739831 = score(doc=119,freq=1.0), product of:
             0.11619143 = queryWeight, product of:
               1.0403079 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.015042755 = queryNorm
             0.5800626 = fieldWeight in 119, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.078125 = fieldNorm(doc=119)
         0.026076905 = weight(abstract_txt:document in 119) [ClassicSimilarity], result of:
           0.026076905 = score(doc=119,freq=1.0), product of:
             0.07773003 = queryWeight, product of:
               1.2033286 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.015042755 = queryNorm
             0.33548045 = fieldWeight in 119, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.078125 = fieldNorm(doc=119)
         0.032276377 = weight(abstract_txt:then in 119) [ClassicSimilarity], result of:
           0.032276377 = score(doc=119,freq=1.0), product of:
             0.089606866 = queryWeight, product of:
               1.291994 = boost
               4.6105576 = idf(docFreq=1200, maxDocs=44421)
               0.015042755 = queryNorm
             0.3601998 = fieldWeight in 119, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6105576 = idf(docFreq=1200, maxDocs=44421)
               0.078125 = fieldNorm(doc=119)
         0.1414133 = weight(abstract_txt:ranking in 119) [ClassicSimilarity], result of:
           0.1414133 = score(doc=119,freq=6.0), product of:
             0.13203667 = queryWeight, product of:
               1.5683287 = boost
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.015042755 = queryNorm
             1.0710154 = fieldWeight in 119, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               5.5966744 = idf(docFreq=447, maxDocs=44421)
               0.078125 = fieldNorm(doc=119)
         0.11170823 = weight(abstract_txt:weighting in 119) [ClassicSimilarity], result of:
           0.11170823 = score(doc=119,freq=1.0), product of:
             0.20502566 = queryWeight, product of:
               1.9543122 = boost
               6.9740796 = idf(docFreq=112, maxDocs=44421)
               0.015042755 = queryNorm
             0.54485 = fieldWeight in 119, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.9740796 = idf(docFreq=112, maxDocs=44421)
               0.078125 = fieldNorm(doc=119)
         0.62237275 = weight(abstract_txt:bm25 in 119) [ClassicSimilarity], result of:
           0.62237275 = score(doc=119,freq=3.0), product of:
             0.51141703 = queryWeight, product of:
               3.7802722 = boost
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.015042755 = queryNorm
             1.2169574 = fieldWeight in 119, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.078125 = fieldNorm(doc=119)
         0.095403984 = weight(abstract_txt:structure in 119) [ClassicSimilarity], result of:
           0.095403984 = score(doc=119,freq=1.0), product of:
             0.28021097 = queryWeight, product of:
               4.274314 = boost
               4.3580413 = idf(docFreq=1545, maxDocs=44421)
               0.015042755 = queryNorm
             0.34047198 = fieldWeight in 119, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.3580413 = idf(docFreq=1545, maxDocs=44421)
               0.078125 = fieldNorm(doc=119)
       0.28 = coord(7/25)
   

2. Dang, E.K.F.; Luk, R.W.P.; Allan, J.; Ho, K.S.; Chung, K.F.L.; Lee, D.L.: ¬A new context-dependent term weight computed by boost and discount using relevance information (2010) 0.30

   0.30420977 = sum of:
     0.30420977 = product of:
       1.2675407 = sum of:
         0.03613322 = weight(abstract_txt:document in 120) [ClassicSimilarity], result of:
           0.03613322 = score(doc=120,freq=3.0), product of:
             0.07773003 = queryWeight, product of:
               1.2033286 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.015042755 = queryNorm
             0.46485534 = fieldWeight in 120, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.0625 = fieldNorm(doc=120)
         0.08660414 = weight(abstract_txt:improvement in 120) [ClassicSimilarity], result of:
           0.08660414 = score(doc=120,freq=2.0), product of:
             0.15935802 = queryWeight, product of:
               1.7229663 = boost
               6.148508 = idf(docFreq=257, maxDocs=44421)
               0.015042755 = queryNorm
             0.54345644 = fieldWeight in 120, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.148508 = idf(docFreq=257, maxDocs=44421)
               0.0625 = fieldNorm(doc=120)
         0.08163976 = weight(abstract_txt:occurrence in 120) [ClassicSimilarity], result of:
           0.08163976 = score(doc=120,freq=1.0), product of:
             0.1930305 = queryWeight, product of:
               1.8962815 = boost
               6.7669935 = idf(docFreq=138, maxDocs=44421)
               0.015042755 = queryNorm
             0.4229371 = fieldWeight in 120, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.7669935 = idf(docFreq=138, maxDocs=44421)
               0.0625 = fieldNorm(doc=120)
         0.08936658 = weight(abstract_txt:weighting in 120) [ClassicSimilarity], result of:
           0.08936658 = score(doc=120,freq=1.0), product of:
             0.20502566 = queryWeight, product of:
               1.9543122 = boost
               6.9740796 = idf(docFreq=112, maxDocs=44421)
               0.015042755 = queryNorm
             0.43587998 = fieldWeight in 120, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.9740796 = idf(docFreq=112, maxDocs=44421)
               0.0625 = fieldNorm(doc=120)
         0.4065322 = weight(abstract_txt:bm25 in 120) [ClassicSimilarity], result of:
           0.4065322 = score(doc=120,freq=2.0), product of:
             0.51141703 = queryWeight, product of:
               3.7802722 = boost
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.015042755 = queryNorm
             0.7949133 = fieldWeight in 120, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.0625 = fieldNorm(doc=120)
         0.56726485 = weight(abstract_txt:weights in 120) [ClassicSimilarity], result of:
           0.56726485 = score(doc=120,freq=8.0), product of:
             0.44278753 = queryWeight, product of:
               4.0616503 = boost
               7.2471204 = idf(docFreq=85, maxDocs=44421)
               0.015042755 = queryNorm
             1.281122 = fieldWeight in 120, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               7.2471204 = idf(docFreq=85, maxDocs=44421)
               0.0625 = fieldNorm(doc=120)
       0.24 = coord(6/25)
   

3. Wolfram, D.; Zhang, J.: ¬The influence of indexing practices and weighting algorithms on document spaces (2008) 0.27

   0.2670216 = sum of:
     0.2670216 = product of:
       0.8344425 = sum of:
         0.089470275 = weight(abstract_txt:occurring in 2963) [ClassicSimilarity], result of:
           0.089470275 = score(doc=2963,freq=3.0), product of:
             0.11291724 = queryWeight, product of:
               1.0255456 = boost
               7.319441 = idf(docFreq=79, maxDocs=44421)
               0.015042755 = queryNorm
             0.7923527 = fieldWeight in 2963, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.319441 = idf(docFreq=79, maxDocs=44421)
               0.0625 = fieldNorm(doc=2963)
         0.015541039 = weight(abstract_txt:than in 2963) [ClassicSimilarity], result of:
           0.015541039 = score(doc=2963,freq=1.0), product of:
             0.063877046 = queryWeight, product of:
               1.090843 = boost
               3.8927383 = idf(docFreq=2461, maxDocs=44421)
               0.015042755 = queryNorm
             0.24329615 = fieldWeight in 2963, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.8927383 = idf(docFreq=2461, maxDocs=44421)
               0.0625 = fieldNorm(doc=2963)
         0.051100094 = weight(abstract_txt:document in 2963) [ClassicSimilarity], result of:
           0.051100094 = score(doc=2963,freq=6.0), product of:
             0.07773003 = queryWeight, product of:
               1.2033286 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.015042755 = queryNorm
             0.6574048 = fieldWeight in 2963, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.0625 = fieldNorm(doc=2963)
         0.08423365 = weight(abstract_txt:influence in 2963) [ClassicSimilarity], result of:
           0.08423365 = score(doc=2963,freq=5.0), product of:
             0.115263514 = queryWeight, product of:
               1.4653311 = boost
               5.229121 = idf(docFreq=646, maxDocs=44421)
               0.015042755 = queryNorm
             0.7307919 = fieldWeight in 2963, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               5.229121 = idf(docFreq=646, maxDocs=44421)
               0.0625 = fieldNorm(doc=2963)
         0.082665615 = weight(abstract_txt:space in 2963) [ClassicSimilarity], result of:
           0.082665615 = score(doc=2963,freq=4.0), product of:
             0.122618146 = queryWeight, product of:
               1.5113574 = boost
               5.393369 = idf(docFreq=548, maxDocs=44421)
               0.015042755 = queryNorm
             0.67417115 = fieldWeight in 2963, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.393369 = idf(docFreq=548, maxDocs=44421)
               0.0625 = fieldNorm(doc=2963)
         0.073011965 = weight(abstract_txt:vector in 2963) [ClassicSimilarity], result of:
           0.073011965 = score(doc=2963,freq=1.0), product of:
             0.17917915 = queryWeight, product of:
               1.826979 = boost
               6.519684 = idf(docFreq=177, maxDocs=44421)
               0.015042755 = queryNorm
             0.40748024 = fieldWeight in 2963, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.519684 = idf(docFreq=177, maxDocs=44421)
               0.0625 = fieldNorm(doc=2963)
         0.15478745 = weight(abstract_txt:weighting in 2963) [ClassicSimilarity], result of:
           0.15478745 = score(doc=2963,freq=3.0), product of:
             0.20502566 = queryWeight, product of:
               1.9543122 = boost
               6.9740796 = idf(docFreq=112, maxDocs=44421)
               0.015042755 = queryNorm
             0.75496626 = fieldWeight in 2963, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.9740796 = idf(docFreq=112, maxDocs=44421)
               0.0625 = fieldNorm(doc=2963)
         0.28363243 = weight(abstract_txt:weights in 2963) [ClassicSimilarity], result of:
           0.28363243 = score(doc=2963,freq=2.0), product of:
             0.44278753 = queryWeight, product of:
               4.0616503 = boost
               7.2471204 = idf(docFreq=85, maxDocs=44421)
               0.015042755 = queryNorm
             0.640561 = fieldWeight in 2963, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.2471204 = idf(docFreq=85, maxDocs=44421)
               0.0625 = fieldNorm(doc=2963)
       0.32 = coord(8/25)
   

4. Dang, E.K.F.; Luk, R.W.P.; Allan, J.: Beyond bag-of-words : bigram-enhanced context-dependent term weights (2014) 0.21

   0.21090338 = sum of:
     0.21090338 = product of:
       0.8787641 = sum of:
         0.031616565 = weight(abstract_txt:document in 2283) [ClassicSimilarity], result of:
           0.031616565 = score(doc=2283,freq=3.0), product of:
             0.07773003 = queryWeight, product of:
               1.2033286 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.015042755 = queryNorm
             0.4067484 = fieldWeight in 2283, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.0546875 = fieldNorm(doc=2283)
         0.053583585 = weight(abstract_txt:improvement in 2283) [ClassicSimilarity], result of:
           0.053583585 = score(doc=2283,freq=1.0), product of:
             0.15935802 = queryWeight, product of:
               1.7229663 = boost
               6.148508 = idf(docFreq=257, maxDocs=44421)
               0.015042755 = queryNorm
             0.33624655 = fieldWeight in 2283, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.148508 = idf(docFreq=257, maxDocs=44421)
               0.0546875 = fieldNorm(doc=2283)
         0.07143479 = weight(abstract_txt:occurrence in 2283) [ClassicSimilarity], result of:
           0.07143479 = score(doc=2283,freq=1.0), product of:
             0.1930305 = queryWeight, product of:
               1.8962815 = boost
               6.7669935 = idf(docFreq=138, maxDocs=44421)
               0.015042755 = queryNorm
             0.37006995 = fieldWeight in 2283, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.7669935 = idf(docFreq=138, maxDocs=44421)
               0.0546875 = fieldNorm(doc=2283)
         0.07819576 = weight(abstract_txt:weighting in 2283) [ClassicSimilarity], result of:
           0.07819576 = score(doc=2283,freq=1.0), product of:
             0.20502566 = queryWeight, product of:
               1.9543122 = boost
               6.9740796 = idf(docFreq=112, maxDocs=44421)
               0.015042755 = queryNorm
             0.38139498 = fieldWeight in 2283, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.9740796 = idf(docFreq=112, maxDocs=44421)
               0.0546875 = fieldNorm(doc=2283)
         0.25152898 = weight(abstract_txt:bm25 in 2283) [ClassicSimilarity], result of:
           0.25152898 = score(doc=2283,freq=1.0), product of:
             0.51141703 = queryWeight, product of:
               3.7802722 = boost
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.015042755 = queryNorm
             0.49182755 = fieldWeight in 2283, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.0546875 = fieldNorm(doc=2283)
         0.39240444 = weight(abstract_txt:weights in 2283) [ClassicSimilarity], result of:
           0.39240444 = score(doc=2283,freq=5.0), product of:
             0.44278753 = queryWeight, product of:
               4.0616503 = boost
               7.2471204 = idf(docFreq=85, maxDocs=44421)
               0.015042755 = queryNorm
             0.88621384 = fieldWeight in 2283, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.2471204 = idf(docFreq=85, maxDocs=44421)
               0.0546875 = fieldNorm(doc=2283)
       0.24 = coord(6/25)
   

5. Kim, W.; Wilbur, W.J.: Corpus-based statistical screening for content-bearing terms (2001) 0.17

   0.17478715 = sum of:
     0.17478715 = product of:
       0.48551983 = sum of:
         0.037787057 = weight(abstract_txt:equal in 188) [ClassicSimilarity], result of:
           0.037787057 = score(doc=188,freq=1.0), product of:
             0.11105446 = queryWeight, product of:
               1.0170512 = boost
               7.2588162 = idf(docFreq=84, maxDocs=44421)
               0.015042755 = queryNorm
             0.34025702 = fieldWeight in 188, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.2588162 = idf(docFreq=84, maxDocs=44421)
               0.046875 = fieldNorm(doc=188)
         0.05478913 = weight(abstract_txt:occurring in 188) [ClassicSimilarity], result of:
           0.05478913 = score(doc=188,freq=2.0), product of:
             0.11291724 = queryWeight, product of:
               1.0255456 = boost
               7.319441 = idf(docFreq=79, maxDocs=44421)
               0.015042755 = queryNorm
             0.48521495 = fieldWeight in 188, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.319441 = idf(docFreq=79, maxDocs=44421)
               0.046875 = fieldNorm(doc=188)
         0.057189357 = weight(abstract_txt:occurrences in 188) [ClassicSimilarity], result of:
           0.057189357 = score(doc=188,freq=2.0), product of:
             0.11619143 = queryWeight, product of:
               1.0403079 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.015042755 = queryNorm
             0.49219942 = fieldWeight in 188, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.046875 = fieldNorm(doc=188)
         0.01165578 = weight(abstract_txt:than in 188) [ClassicSimilarity], result of:
           0.01165578 = score(doc=188,freq=1.0), product of:
             0.063877046 = queryWeight, product of:
               1.090843 = boost
               3.8927383 = idf(docFreq=2461, maxDocs=44421)
               0.015042755 = queryNorm
             0.18247211 = fieldWeight in 188, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.8927383 = idf(docFreq=2461, maxDocs=44421)
               0.046875 = fieldNorm(doc=188)
         0.03832507 = weight(abstract_txt:document in 188) [ClassicSimilarity], result of:
           0.03832507 = score(doc=188,freq=6.0), product of:
             0.07773003 = queryWeight, product of:
               1.2033286 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.015042755 = queryNorm
             0.4930536 = fieldWeight in 188, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.046875 = fieldNorm(doc=188)
         0.019365825 = weight(abstract_txt:then in 188) [ClassicSimilarity], result of:
           0.019365825 = score(doc=188,freq=1.0), product of:
             0.089606866 = queryWeight, product of:
               1.291994 = boost
               4.6105576 = idf(docFreq=1200, maxDocs=44421)
               0.015042755 = queryNorm
             0.21611989 = fieldWeight in 188, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6105576 = idf(docFreq=1200, maxDocs=44421)
               0.046875 = fieldNorm(doc=188)
         0.05475897 = weight(abstract_txt:vector in 188) [ClassicSimilarity], result of:
           0.05475897 = score(doc=188,freq=1.0), product of:
             0.17917915 = queryWeight, product of:
               1.826979 = boost
               6.519684 = idf(docFreq=177, maxDocs=44421)
               0.015042755 = queryNorm
             0.30561018 = fieldWeight in 188, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.519684 = idf(docFreq=177, maxDocs=44421)
               0.046875 = fieldNorm(doc=188)
         0.061229818 = weight(abstract_txt:occurrence in 188) [ClassicSimilarity], result of:
           0.061229818 = score(doc=188,freq=1.0), product of:
             0.1930305 = queryWeight, product of:
               1.8962815 = boost
               6.7669935 = idf(docFreq=138, maxDocs=44421)
               0.015042755 = queryNorm
             0.3172028 = fieldWeight in 188, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.7669935 = idf(docFreq=138, maxDocs=44421)
               0.046875 = fieldNorm(doc=188)
         0.1504188 = weight(abstract_txt:weights in 188) [ClassicSimilarity], result of:
           0.1504188 = score(doc=188,freq=1.0), product of:
             0.44278753 = queryWeight, product of:
               4.0616503 = boost
               7.2471204 = idf(docFreq=85, maxDocs=44421)
               0.015042755 = queryNorm
             0.33970878 = fieldWeight in 188, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.2471204 = idf(docFreq=85, maxDocs=44421)
               0.046875 = fieldNorm(doc=188)
       0.36 = coord(9/25)