Document (#32946)

Author

Nomoto, T.

Title

Discriminative sentence compression with conditional random fields

Source

Information processing and management. 43(2007) no.6, S.1571-1587

Year

2007

Abstract

The paper focuses on a particular approach to automatic sentence compression which makes use of a discriminative sequence classifier known as Conditional Random Fields (CRF). We devise several features for CRF that allow it to incorporate information on nonlinear relations among words. Along with that, we address the issue of data paucity by collecting data from RSS feeds available on the Internet, and turning them into training data for use with CRF, drawing on techniques from biology and information retrieval. We also discuss a recursive application of CRF on the syntactic structure of a sentence as a way of improving the readability of the compression it generates. Experiments found that our approach works reasonably well compared to the state-of-the-art system [Knight, K., & Marcu, D. (2002). Summarization beyond sentence extraction: A probabilistic approach to sentence compression. Artificial Intelligence 139, 91-107.].

Theme

Automatisches Abstracting

Similar documents (content)

1. Cannane, A.; Williams, H.E.: General-purpose compression for efficient retrieval (2001) 0.24

   0.23962568 = sum of:
     0.23962568 = product of:
       0.9984403 = sum of:
         0.011287417 = weight(abstract_txt:that in 5705) [ClassicSimilarity], result of:
           0.011287417 = score(doc=5705,freq=3.0), product of:
             0.035203982 = queryWeight, product of:
               1.0420328 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.014257987 = queryNorm
             0.320629 = fieldWeight in 5705, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.078125 = fieldNorm(doc=5705)
         0.0076517113 = weight(abstract_txt:with in 5705) [ClassicSimilarity], result of:
           0.0076517113 = score(doc=5705,freq=1.0), product of:
             0.039180942 = queryWeight, product of:
               1.0993171 = boost
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.014257987 = queryNorm
             0.19529167 = fieldWeight in 5705, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.078125 = fieldNorm(doc=5705)
         0.018192463 = weight(abstract_txt:data in 5705) [ClassicSimilarity], result of:
           0.018192463 = score(doc=5705,freq=1.0), product of:
             0.06979594 = queryWeight, product of:
               1.4672381 = boost
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.014257987 = queryNorm
             0.26065218 = fieldWeight in 5705, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.078125 = fieldNorm(doc=5705)
         0.025736354 = weight(abstract_txt:approach in 5705) [ClassicSimilarity], result of:
           0.025736354 = score(doc=5705,freq=1.0), product of:
             0.08795635 = queryWeight, product of:
               1.6470968 = boost
               3.745328 = idf(docFreq=2839, maxDocs=44218)
               0.014257987 = queryNorm
             0.29260373 = fieldWeight in 5705, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.745328 = idf(docFreq=2839, maxDocs=44218)
               0.078125 = fieldNorm(doc=5705)
         0.09102171 = weight(abstract_txt:random in 5705) [ClassicSimilarity], result of:
           0.09102171 = score(doc=5705,freq=1.0), product of:
             0.17836188 = queryWeight, product of:
               1.9150976 = boost
               6.532101 = idf(docFreq=174, maxDocs=44218)
               0.014257987 = queryNorm
             0.5103204 = fieldWeight in 5705, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.532101 = idf(docFreq=174, maxDocs=44218)
               0.078125 = fieldNorm(doc=5705)
         0.84455067 = weight(abstract_txt:compression in 5705) [ClassicSimilarity], result of:
           0.84455067 = score(doc=5705,freq=9.0), product of:
             0.4770366 = queryWeight, product of:
               4.4292555 = boost
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.014257987 = queryNorm
             1.7704107 = fieldWeight in 5705, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.078125 = fieldNorm(doc=5705)
       0.24 = coord(6/25)
   

2. Zajic, D.; Dorr, B.J.; Lin, J.; Schwartz, R.: Multi-candidate reduction : sentence compression as a tool for document summarization tasks (2007) 0.24

   0.23931971 = sum of:
     0.23931971 = product of:
       1.1965985 = sum of:
         0.007820153 = weight(abstract_txt:that in 944) [ClassicSimilarity], result of:
           0.007820153 = score(doc=944,freq=1.0), product of:
             0.035203982 = queryWeight, product of:
               1.0420328 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.014257987 = queryNorm
             0.22213829 = fieldWeight in 944, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.09375 = fieldNorm(doc=944)
         0.1231489 = weight(abstract_txt:summarization in 944) [ClassicSimilarity], result of:
           0.1231489 = score(doc=944,freq=3.0), product of:
             0.10632969 = queryWeight, product of:
               1.0455678 = boost
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.014257987 = queryNorm
             1.1581799 = fieldWeight in 944, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.09375 = fieldNorm(doc=944)
         0.04367604 = weight(abstract_txt:approach in 944) [ClassicSimilarity], result of:
           0.04367604 = score(doc=944,freq=2.0), product of:
             0.08795635 = queryWeight, product of:
               1.6470968 = boost
               3.745328 = idf(docFreq=2839, maxDocs=44218)
               0.014257987 = queryNorm
             0.49656498 = fieldWeight in 944, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.745328 = idf(docFreq=2839, maxDocs=44218)
               0.09375 = fieldNorm(doc=944)
         0.47775006 = weight(abstract_txt:compression in 944) [ClassicSimilarity], result of:
           0.47775006 = score(doc=944,freq=2.0), product of:
             0.4770366 = queryWeight, product of:
               4.4292555 = boost
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.014257987 = queryNorm
             1.0014956 = fieldWeight in 944, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.09375 = fieldNorm(doc=944)
         0.54420334 = weight(abstract_txt:sentence in 944) [ClassicSimilarity], result of:
           0.54420334 = score(doc=944,freq=3.0), product of:
             0.48962656 = queryWeight, product of:
               5.01698 = boost
               6.8448567 = idf(docFreq=127, maxDocs=44218)
               0.014257987 = queryNorm
             1.1114662 = fieldWeight in 944, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.8448567 = idf(docFreq=127, maxDocs=44218)
               0.09375 = fieldNorm(doc=944)
       0.2 = coord(5/25)
   

3. Zajic, D.M.; Dorr, B.J.; Lin, J.: Single-document and multi-document summarization techniques for email threads using sentence compression (2008) 0.20

   0.19579244 = sum of:
     0.19579244 = product of:
       0.9789622 = sum of:
         0.009216138 = weight(abstract_txt:that in 2105) [ClassicSimilarity], result of:
           0.009216138 = score(doc=2105,freq=2.0), product of:
             0.035203982 = queryWeight, product of:
               1.0420328 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.014257987 = queryNorm
             0.26179248 = fieldWeight in 2105, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.078125 = fieldNorm(doc=2105)
         0.1567609 = weight(abstract_txt:summarization in 2105) [ClassicSimilarity], result of:
           0.1567609 = score(doc=2105,freq=7.0), product of:
             0.10632969 = queryWeight, product of:
               1.0455678 = boost
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.014257987 = queryNorm
             1.474291 = fieldWeight in 2105, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.078125 = fieldNorm(doc=2105)
         0.044576675 = weight(abstract_txt:approach in 2105) [ClassicSimilarity], result of:
           0.044576675 = score(doc=2105,freq=3.0), product of:
             0.08795635 = queryWeight, product of:
               1.6470968 = boost
               3.745328 = idf(docFreq=2839, maxDocs=44218)
               0.014257987 = queryNorm
             0.5068045 = fieldWeight in 2105, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.745328 = idf(docFreq=2839, maxDocs=44218)
               0.078125 = fieldNorm(doc=2105)
         0.398125 = weight(abstract_txt:compression in 2105) [ClassicSimilarity], result of:
           0.398125 = score(doc=2105,freq=2.0), product of:
             0.4770366 = queryWeight, product of:
               4.4292555 = boost
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.014257987 = queryNorm
             0.8345796 = fieldWeight in 2105, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.078125 = fieldNorm(doc=2105)
         0.37028348 = weight(abstract_txt:sentence in 2105) [ClassicSimilarity], result of:
           0.37028348 = score(doc=2105,freq=2.0), product of:
             0.48962656 = queryWeight, product of:
               5.01698 = boost
               6.8448567 = idf(docFreq=127, maxDocs=44218)
               0.014257987 = queryNorm
             0.75625694 = fieldWeight in 2105, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.8448567 = idf(docFreq=127, maxDocs=44218)
               0.078125 = fieldNorm(doc=2105)
       0.2 = coord(5/25)
   

4. Yeh, J.-Y.; Ke, H.-R.; Yang, W.-P.; Meng, I.-H.: Text summarization using a trainable summarizer and latent semantic analysis (2005) 0.19

   0.1915509 = sum of:
     0.1915509 = product of:
       0.7981287 = sum of:
         0.067033775 = weight(abstract_txt:summarization in 1003) [ClassicSimilarity], result of:
           0.067033775 = score(doc=1003,freq=2.0), product of:
             0.10632969 = queryWeight, product of:
               1.0455678 = boost
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.014257987 = queryNorm
             0.6304333 = fieldWeight in 1003, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.0625 = fieldNorm(doc=1003)
         0.0061213695 = weight(abstract_txt:with in 1003) [ClassicSimilarity], result of:
           0.0061213695 = score(doc=1003,freq=1.0), product of:
             0.039180942 = queryWeight, product of:
               1.0993171 = boost
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.014257987 = queryNorm
             0.15623334 = fieldWeight in 1003, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.0625 = fieldNorm(doc=1003)
         0.01455397 = weight(abstract_txt:data in 1003) [ClassicSimilarity], result of:
           0.01455397 = score(doc=1003,freq=1.0), product of:
             0.06979594 = queryWeight, product of:
               1.4672381 = boost
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.014257987 = queryNorm
             0.20852174 = fieldWeight in 1003, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.0625 = fieldNorm(doc=1003)
         0.029117363 = weight(abstract_txt:approach in 1003) [ClassicSimilarity], result of:
           0.029117363 = score(doc=1003,freq=2.0), product of:
             0.08795635 = queryWeight, product of:
               1.6470968 = boost
               3.745328 = idf(docFreq=2839, maxDocs=44218)
               0.014257987 = queryNorm
             0.33104333 = fieldWeight in 1003, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.745328 = idf(docFreq=2839, maxDocs=44218)
               0.0625 = fieldNorm(doc=1003)
         0.3185 = weight(abstract_txt:compression in 1003) [ClassicSimilarity], result of:
           0.3185 = score(doc=1003,freq=2.0), product of:
             0.4770366 = queryWeight, product of:
               4.4292555 = boost
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.014257987 = queryNorm
             0.6676637 = fieldWeight in 1003, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.0625 = fieldNorm(doc=1003)
         0.36280224 = weight(abstract_txt:sentence in 1003) [ClassicSimilarity], result of:
           0.36280224 = score(doc=1003,freq=3.0), product of:
             0.48962656 = queryWeight, product of:
               5.01698 = boost
               6.8448567 = idf(docFreq=127, maxDocs=44218)
               0.014257987 = queryNorm
             0.74097747 = fieldWeight in 1003, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.8448567 = idf(docFreq=127, maxDocs=44218)
               0.0625 = fieldNorm(doc=1003)
       0.24 = coord(6/25)
   

5. Finegan-Dollak, C.; Radev, D.R.: Sentence simplification, compression, and disaggregation for summarization of sophisticated documents (2016) 0.17

   0.16673557 = sum of:
     0.16673557 = product of:
       0.8336779 = sum of:
         0.009029934 = weight(abstract_txt:that in 3122) [ClassicSimilarity], result of:
           0.009029934 = score(doc=3122,freq=3.0), product of:
             0.035203982 = queryWeight, product of:
               1.0420328 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.014257987 = queryNorm
             0.2565032 = fieldWeight in 3122, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=3122)
         0.010602523 = weight(abstract_txt:with in 3122) [ClassicSimilarity], result of:
           0.010602523 = score(doc=3122,freq=3.0), product of:
             0.039180942 = queryWeight, product of:
               1.0993171 = boost
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.014257987 = queryNorm
             0.27060407 = fieldWeight in 3122, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.0625 = fieldNorm(doc=3122)
         0.061161894 = weight(abstract_txt:generates in 3122) [ClassicSimilarity], result of:
           0.061161894 = score(doc=3122,freq=1.0), product of:
             0.1260248 = queryWeight, product of:
               1.1382899 = boost
               7.7650614 = idf(docFreq=50, maxDocs=44218)
               0.014257987 = queryNorm
             0.48531634 = fieldWeight in 3122, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.7650614 = idf(docFreq=50, maxDocs=44218)
               0.0625 = fieldNorm(doc=3122)
         0.39008126 = weight(abstract_txt:compression in 3122) [ClassicSimilarity], result of:
           0.39008126 = score(doc=3122,freq=3.0), product of:
             0.4770366 = queryWeight, product of:
               4.4292555 = boost
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.014257987 = queryNorm
             0.8177177 = fieldWeight in 3122, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.0625 = fieldNorm(doc=3122)
         0.36280224 = weight(abstract_txt:sentence in 3122) [ClassicSimilarity], result of:
           0.36280224 = score(doc=3122,freq=3.0), product of:
             0.48962656 = queryWeight, product of:
               5.01698 = boost
               6.8448567 = idf(docFreq=127, maxDocs=44218)
               0.014257987 = queryNorm
             0.74097747 = fieldWeight in 3122, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.8448567 = idf(docFreq=127, maxDocs=44218)
               0.0625 = fieldNorm(doc=3122)
       0.2 = coord(5/25)