Document (#33357)

Author
Rooney, N.
Patterson, D.
Galushka, M.
Dobrynin, V.
Smirnova, E.
Title
¬An investigation into the stability of contextual document clustering
Source
Journal of the American Society for Information Science and Technology. 59(2008) no.2, S.256-266
Year
2008
Abstract
In this article, we assess the effectiveness of Contextual Document Clustering (CDC) as a means of indexing within a dynamic and rapidly changing environment. We simulate a dynamic environment, by splitting two chronologically ordered datasets into time-ordered segments and assessing how the technique performs under two different scenarios. The first is when new documents are added incrementally without reclustering [incremental CDC (iCDC)], and the second is when reclustering is performed [nonincremental CDC (nCDC)]. The datasets are very large, are independent of each other, and belong to two very different domains. We show that CDC itself is effective at clustering very large document corpora, and that, significantly, it lends itself to a very simple, efficient incremental document addition process that is seen to be very stable over time despite the size of the corpus growing considerably. It was seen to be effective at incrementally clustering new documents even when the corpus grew to six times its original size. This is in contrast to what other researchers have found when applying similar simple incremental approaches to document clustering. The stability of iCDC is accounted for by the unique manner in which CDC discovers cluster themes.
Theme
Automatisches Klassifizieren

Similar documents (author)

  1. Patterson, E.L.: ¬The bibliographic control of microforms (1992) 6.19 
    6.1935673 = sum of:
  6.1935673 = weight(author_txt:patterson in 1353) [ClassicSimilarity], result of:
    6.1935673 = fieldWeight in 1353, product of:
      1.0 = tf(freq=1.0), with freq of:
        1.0 = termFreq=1.0
      9.909708 = idf(docFreq=5, maxDocs=44421)
      0.625 = fieldNorm(doc=1353)
  2. Patterson, C.D.: Origins of systematic serials control : remembering Carolyn Ulrich (1988) 6.19 
    6.1935673 = sum of:
  6.1935673 = weight(author_txt:patterson in 2543) [ClassicSimilarity], result of:
    6.1935673 = fieldWeight in 2543, product of:
      1.0 = tf(freq=1.0), with freq of:
        1.0 = termFreq=1.0
      9.909708 = idf(docFreq=5, maxDocs=44421)
      0.625 = fieldNorm(doc=2543)
  3. Rohlfing, H.; Schappacher, N.; Patterson, S.J.: ¬Das Zentralarchiv für Mathematiker-Nachlässe an der Niedersächsischen Staats- und Universitätsbibliothek (2003) 3.72 
    3.7161405 = sum of:
  3.7161405 = weight(author_txt:patterson in 3321) [ClassicSimilarity], result of:
    3.7161405 = fieldWeight in 3321, product of:
      1.0 = tf(freq=1.0), with freq of:
        1.0 = termFreq=1.0
      9.909708 = idf(docFreq=5, maxDocs=44421)
      0.375 = fieldNorm(doc=3321)
  4. Barry, E.; Bedoya, J.K.; Groom, C.; Patterson, L.: Virtual reference in UK academic libraries : the virtual enquiry project 2008-2009 (2010) 3.10 
    3.0967836 = sum of:
  3.0967836 = weight(author_txt:patterson in 3966) [ClassicSimilarity], result of:
    3.0967836 = fieldWeight in 3966, product of:
      1.0 = tf(freq=1.0), with freq of:
        1.0 = termFreq=1.0
      9.909708 = idf(docFreq=5, maxDocs=44421)
      0.3125 = fieldNorm(doc=3966)

Similar documents (content)

  1. Can, F.: Incremental clustering for dynamic information processing (1993) 0.32 
    0.3169005 = sum of:
  0.3169005 = product of:
    0.99031407 = sum of:
      0.03343303 = weight(abstract_txt:large in 6626) [ClassicSimilarity], result of:
        0.03343303 = score(doc=6626,freq=1.0), product of:
          0.08027708 = queryWeight, product of:
            1.0615872 = boost
            4.4423513 = idf(docFreq=1420, maxDocs=44421)
            0.017022485 = queryNorm
          0.41647044 = fieldWeight in 6626, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            4.4423513 = idf(docFreq=1420, maxDocs=44421)
            0.09375 = fieldNorm(doc=6626)
      0.037932117 = weight(abstract_txt:environment in 6626) [ClassicSimilarity], result of:
        0.037932117 = score(doc=6626,freq=1.0), product of:
          0.087326445 = queryWeight, product of:
            1.1072172 = boost
            4.6332955 = idf(docFreq=1173, maxDocs=44421)
            0.017022485 = queryNorm
          0.43437147 = fieldWeight in 6626, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            4.6332955 = idf(docFreq=1173, maxDocs=44421)
            0.09375 = fieldNorm(doc=6626)
      0.043176174 = weight(abstract_txt:effective in 6626) [ClassicSimilarity], result of:
        0.043176174 = score(doc=6626,freq=1.0), product of:
          0.09520005 = queryWeight, product of:
            1.156055 = boost
            4.837664 = idf(docFreq=956, maxDocs=44421)
            0.017022485 = queryNorm
          0.45353103 = fieldWeight in 6626, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            4.837664 = idf(docFreq=956, maxDocs=44421)
            0.09375 = fieldNorm(doc=6626)
      0.073028676 = weight(abstract_txt:dynamic in 6626) [ClassicSimilarity], result of:
        0.073028676 = score(doc=6626,freq=1.0), product of:
          0.13514605 = queryWeight, product of:
            1.3774048 = boost
            5.7639313 = idf(docFreq=378, maxDocs=44421)
            0.017022485 = queryNorm
          0.54036856 = fieldWeight in 6626, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            5.7639313 = idf(docFreq=378, maxDocs=44421)
            0.09375 = fieldNorm(doc=6626)
      0.07549334 = weight(abstract_txt:document in 6626) [ClassicSimilarity], result of:
        0.07549334 = score(doc=6626,freq=1.0), product of:
          0.18752542 = queryWeight, product of:
            2.565429 = boost
            4.29415 = idf(docFreq=1647, maxDocs=44421)
            0.017022485 = queryNorm
          0.40257657 = fieldWeight in 6626, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            4.29415 = idf(docFreq=1647, maxDocs=44421)
            0.09375 = fieldNorm(doc=6626)
      0.28892758 = weight(abstract_txt:incremental in 6626) [ClassicSimilarity], result of:
        0.28892758 = score(doc=6626,freq=1.0), product of:
          0.38698798 = queryWeight, product of:
            2.8546593 = boost
            7.963798 = idf(docFreq=41, maxDocs=44421)
            0.017022485 = queryNorm
          0.74660605 = fieldWeight in 6626, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            7.963798 = idf(docFreq=41, maxDocs=44421)
            0.09375 = fieldNorm(doc=6626)
      0.11373197 = weight(abstract_txt:very in 6626) [ClassicSimilarity], result of:
        0.11373197 = score(doc=6626,freq=1.0), product of:
          0.24643901 = queryWeight, product of:
            2.94093 = boost
            4.922683 = idf(docFreq=878, maxDocs=44421)
            0.017022485 = queryNorm
          0.4615015 = fieldWeight in 6626, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            4.922683 = idf(docFreq=878, maxDocs=44421)
            0.09375 = fieldNorm(doc=6626)
      0.32459116 = weight(abstract_txt:clustering in 6626) [ClassicSimilarity], result of:
        0.32459116 = score(doc=6626,freq=2.0), product of:
          0.3935521 = queryWeight, product of:
            3.7164736 = boost
            6.2208285 = idf(docFreq=239, maxDocs=44421)
            0.017022485 = queryNorm
          0.8247731 = fieldWeight in 6626, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            6.2208285 = idf(docFreq=239, maxDocs=44421)
            0.09375 = fieldNorm(doc=6626)
    0.32 = coord(8/25)
  2. Cai, X.; Li, W.: Enhancing sentence-level clustering with integrated and interactive frameworks for theme-based summarization (2011) 0.14 
    0.13962418 = sum of:
  0.13962418 = product of:
    0.8726511 = sum of:
      0.08536623 = weight(abstract_txt:discovers in 770) [ClassicSimilarity], result of:
        0.08536623 = score(doc=770,freq=1.0), product of:
          0.15597296 = queryWeight, product of:
            1.0463321 = boost
            8.757029 = idf(docFreq=18, maxDocs=44421)
            0.017022485 = queryNorm
          0.5473143 = fieldWeight in 770, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            8.757029 = idf(docFreq=18, maxDocs=44421)
            0.0625 = fieldNorm(doc=770)
      0.07138496 = weight(abstract_txt:datasets in 770) [ClassicSimilarity], result of:
        0.07138496 = score(doc=770,freq=1.0), product of:
          0.17442405 = queryWeight, product of:
            1.5648154 = boost
            6.548176 = idf(docFreq=172, maxDocs=44421)
            0.017022485 = queryNorm
          0.409261 = fieldWeight in 770, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            6.548176 = idf(docFreq=172, maxDocs=44421)
            0.0625 = fieldNorm(doc=770)
      0.12328011 = weight(abstract_txt:document in 770) [ClassicSimilarity], result of:
        0.12328011 = score(doc=770,freq=6.0), product of:
          0.18752542 = queryWeight, product of:
            2.565429 = boost
            4.29415 = idf(docFreq=1647, maxDocs=44421)
            0.017022485 = queryNorm
          0.6574048 = fieldWeight in 770, product of:
            2.4494898 = tf(freq=6.0), with freq of:
              6.0 = termFreq=6.0
            4.29415 = idf(docFreq=1647, maxDocs=44421)
            0.0625 = fieldNorm(doc=770)
      0.5926198 = weight(abstract_txt:clustering in 770) [ClassicSimilarity], result of:
        0.5926198 = score(doc=770,freq=15.0), product of:
          0.3935521 = queryWeight, product of:
            3.7164736 = boost
            6.2208285 = idf(docFreq=239, maxDocs=44421)
            0.017022485 = queryNorm
          1.5058229 = fieldWeight in 770, product of:
            3.8729835 = tf(freq=15.0), with freq of:
              15.0 = termFreq=15.0
            6.2208285 = idf(docFreq=239, maxDocs=44421)
            0.0625 = fieldNorm(doc=770)
    0.16 = coord(4/25)
  3. Ruocco, A.S.; Frieder, O.: Clustering and classification of large document bases in a parallel environment (1997) 0.11 
    0.113512665 = sum of:
  0.113512665 = product of:
    0.7094542 = sum of:
      0.03343303 = weight(abstract_txt:large in 2661) [ClassicSimilarity], result of:
        0.03343303 = score(doc=2661,freq=1.0), product of:
          0.08027708 = queryWeight, product of:
            1.0615872 = boost
            4.4423513 = idf(docFreq=1420, maxDocs=44421)
            0.017022485 = queryNorm
          0.41647044 = fieldWeight in 2661, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            4.4423513 = idf(docFreq=1420, maxDocs=44421)
            0.09375 = fieldNorm(doc=2661)
      0.086221546 = weight(abstract_txt:corpus in 2661) [ClassicSimilarity], result of:
        0.086221546 = score(doc=2661,freq=1.0), product of:
          0.15096802 = queryWeight, product of:
            1.4558022 = boost
            6.0919957 = idf(docFreq=272, maxDocs=44421)
            0.017022485 = queryNorm
          0.5711246 = fieldWeight in 2661, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            6.0919957 = idf(docFreq=272, maxDocs=44421)
            0.09375 = fieldNorm(doc=2661)
      0.1307583 = weight(abstract_txt:document in 2661) [ClassicSimilarity], result of:
        0.1307583 = score(doc=2661,freq=3.0), product of:
          0.18752542 = queryWeight, product of:
            2.565429 = boost
            4.29415 = idf(docFreq=1647, maxDocs=44421)
            0.017022485 = queryNorm
          0.697283 = fieldWeight in 2661, product of:
            1.7320508 = tf(freq=3.0), with freq of:
              3.0 = termFreq=3.0
            4.29415 = idf(docFreq=1647, maxDocs=44421)
            0.09375 = fieldNorm(doc=2661)
      0.45904127 = weight(abstract_txt:clustering in 2661) [ClassicSimilarity], result of:
        0.45904127 = score(doc=2661,freq=4.0), product of:
          0.3935521 = queryWeight, product of:
            3.7164736 = boost
            6.2208285 = idf(docFreq=239, maxDocs=44421)
            0.017022485 = queryNorm
          1.1664053 = fieldWeight in 2661, product of:
            2.0 = tf(freq=4.0), with freq of:
              4.0 = termFreq=4.0
            6.2208285 = idf(docFreq=239, maxDocs=44421)
            0.09375 = fieldNorm(doc=2661)
    0.16 = coord(4/25)
  4. Zhan, J.; Loh, H.T.: Using latent semantic indexing to improve the accuracy of document clustering (2007) 0.11 
    0.1128046 = sum of:
  0.1128046 = product of:
    0.7050288 = sum of:
      0.02786086 = weight(abstract_txt:large in 1264) [ClassicSimilarity], result of:
        0.02786086 = score(doc=1264,freq=1.0), product of:
          0.08027708 = queryWeight, product of:
            1.0615872 = boost
            4.4423513 = idf(docFreq=1420, maxDocs=44421)
            0.017022485 = queryNorm
          0.3470587 = fieldWeight in 1264, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            4.4423513 = idf(docFreq=1420, maxDocs=44421)
            0.078125 = fieldNorm(doc=1264)
      0.045300256 = weight(abstract_txt:when in 1264) [ClassicSimilarity], result of:
        0.045300256 = score(doc=1264,freq=1.0), product of:
          0.13985318 = queryWeight, product of:
            1.9815775 = boost
            4.1460857 = idf(docFreq=1910, maxDocs=44421)
            0.017022485 = queryNorm
          0.32391295 = fieldWeight in 1264, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            4.1460857 = idf(docFreq=1910, maxDocs=44421)
            0.078125 = fieldNorm(doc=1264)
      0.12582223 = weight(abstract_txt:document in 1264) [ClassicSimilarity], result of:
        0.12582223 = score(doc=1264,freq=4.0), product of:
          0.18752542 = queryWeight, product of:
            2.565429 = boost
            4.29415 = idf(docFreq=1647, maxDocs=44421)
            0.017022485 = queryNorm
          0.6709609 = fieldWeight in 1264, product of:
            2.0 = tf(freq=4.0), with freq of:
              4.0 = termFreq=4.0
            4.29415 = idf(docFreq=1647, maxDocs=44421)
            0.078125 = fieldNorm(doc=1264)
      0.5060454 = weight(abstract_txt:clustering in 1264) [ClassicSimilarity], result of:
        0.5060454 = score(doc=1264,freq=7.0), product of:
          0.3935521 = queryWeight, product of:
            3.7164736 = boost
            6.2208285 = idf(docFreq=239, maxDocs=44421)
            0.017022485 = queryNorm
          1.285841 = fieldWeight in 1264, product of:
            2.6457512 = tf(freq=7.0), with freq of:
              7.0 = termFreq=7.0
            6.2208285 = idf(docFreq=239, maxDocs=44421)
            0.078125 = fieldNorm(doc=1264)
    0.16 = coord(4/25)
  5. Zamir, O.; Etzioni, O.: Grouper : a dynamic clustering interface to Web search results (1999) 0.09 
    0.0888303 = sum of:
  0.0888303 = product of:
    0.5551894 = sum of:
      0.035980143 = weight(abstract_txt:effective in 207) [ClassicSimilarity], result of:
        0.035980143 = score(doc=207,freq=1.0), product of:
          0.09520005 = queryWeight, product of:
            1.156055 = boost
            4.837664 = idf(docFreq=956, maxDocs=44421)
            0.017022485 = queryNorm
          0.3779425 = fieldWeight in 207, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            4.837664 = idf(docFreq=956, maxDocs=44421)
            0.078125 = fieldNorm(doc=207)
      0.04770508 = weight(abstract_txt:simple in 207) [ClassicSimilarity], result of:
        0.04770508 = score(doc=207,freq=1.0), product of:
          0.11489602 = queryWeight, product of:
            1.2700254 = boost
            5.314588 = idf(docFreq=593, maxDocs=44421)
            0.017022485 = queryNorm
          0.4152022 = fieldWeight in 207, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            5.314588 = idf(docFreq=593, maxDocs=44421)
            0.078125 = fieldNorm(doc=207)
      0.08896975 = weight(abstract_txt:document in 207) [ClassicSimilarity], result of:
        0.08896975 = score(doc=207,freq=2.0), product of:
          0.18752542 = queryWeight, product of:
            2.565429 = boost
            4.29415 = idf(docFreq=1647, maxDocs=44421)
            0.017022485 = queryNorm
          0.47444102 = fieldWeight in 207, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            4.29415 = idf(docFreq=1647, maxDocs=44421)
            0.078125 = fieldNorm(doc=207)
      0.38253438 = weight(abstract_txt:clustering in 207) [ClassicSimilarity], result of:
        0.38253438 = score(doc=207,freq=4.0), product of:
          0.3935521 = queryWeight, product of:
            3.7164736 = boost
            6.2208285 = idf(docFreq=239, maxDocs=44421)
            0.017022485 = queryNorm
          0.9720045 = fieldWeight in 207, product of:
            2.0 = tf(freq=4.0), with freq of:
              4.0 = termFreq=4.0
            6.2208285 = idf(docFreq=239, maxDocs=44421)
            0.078125 = fieldNorm(doc=207)
    0.16 = coord(4/25)