Document (#28684)

Author

Kostoff, R.N.
Block, J.A.

Title

Factor matrix text filtering and clustering

Source

Journal of the American Society for Information Science and Technology. 56(2005) no.9, S.946-968

Year

2005

Abstract

The presence of trivial words in text databases can affect record or concept (words/phrases) clustering adversely. Additionally, the determination of whether a word/phrase is trivial is context-dependent. Our objective in the present article is to demonstrate a context-dependent trivial word filter to improve clustering quality. Factor analysis was used as a context-dependent trivial word filter for subsequent term clustering. Medline records for Raynaud's Phenomenon were used as the database, and words were extracted from the record abstracts. A factor matrix of these words was generated, and the words that had low factor loadings across all factors were identified, and eliminated. The remaining words, which had high factor loading values for at least one factor and therefore were influential in determining the theme of that factor, were input to the clustering algorithm. Both quantitative and qualitative analyses were used to show that factor matrix filtering leads to higher quality clusters and subsequent taxonomies.

Theme

Computerlinguistik

Similar documents (author)

1. Kostoff, R.N.: Citation analysis cross field normalization : a new paradigm (1997) 2.20

   2.1964777 = sum of:
     2.1964777 = product of:
       4.3929553 = sum of:
         4.3929553 = weight(author_txt:kostoff in 1464) [ClassicSimilarity], result of:
           4.3929553 = score(doc=1464,freq=1.0), product of:
             0.7304832 = queryWeight, product of:
               1.0342292 = boost
               9.622026 = idf(docFreq=7, maxDocs=44421)
               0.073405206 = queryNorm
             6.0137663 = fieldWeight in 1464, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.622026 = idf(docFreq=7, maxDocs=44421)
               0.625 = fieldNorm(doc=1464)
       0.5 = coord(1/2)
   

2. Kostoff, R.N.: ¬The use and misuse of citation analysis in research evaluation (1998) 2.20

   2.1964777 = sum of:
     2.1964777 = product of:
       4.3929553 = sum of:
         4.3929553 = weight(author_txt:kostoff in 5129) [ClassicSimilarity], result of:
           4.3929553 = score(doc=5129,freq=1.0), product of:
             0.7304832 = queryWeight, product of:
               1.0342292 = boost
               9.622026 = idf(docFreq=7, maxDocs=44421)
               0.073405206 = queryNorm
             6.0137663 = fieldWeight in 5129, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.622026 = idf(docFreq=7, maxDocs=44421)
               0.625 = fieldNorm(doc=5129)
       0.5 = coord(1/2)
   

3. Kostoff, R.N.: Where is the research in the research literature? (2012) 2.20

   2.1964777 = sum of:
     2.1964777 = product of:
       4.3929553 = sum of:
         4.3929553 = weight(author_txt:kostoff in 1374) [ClassicSimilarity], result of:
           4.3929553 = score(doc=1374,freq=1.0), product of:
             0.7304832 = queryWeight, product of:
               1.0342292 = boost
               9.622026 = idf(docFreq=7, maxDocs=44421)
               0.073405206 = queryNorm
             6.0137663 = fieldWeight in 1374, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.622026 = idf(docFreq=7, maxDocs=44421)
               0.625 = fieldNorm(doc=1374)
       0.5 = coord(1/2)
   

4. Block, B.: Quality control makes WorldCat ever better (1999) 1.99

   1.9855297 = sum of:
     1.9855297 = product of:
       3.9710593 = sum of:
         3.9710593 = weight(author_txt:block in 5398) [ClassicSimilarity], result of:
           3.9710593 = score(doc=5398,freq=1.0), product of:
             0.68293065 = queryWeight, product of:
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.073405206 = queryNorm
             5.814733 = fieldWeight in 5398, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.625 = fieldNorm(doc=5398)
       0.5 = coord(1/2)
   

5. Block, B.: Kooperative Neukatalogisierung : Eine neue Facette in der Zusammenlegung der Verbünde (2007) 1.99

   1.9855297 = sum of:
     1.9855297 = product of:
       3.9710593 = sum of:
         3.9710593 = weight(author_txt:block in 394) [ClassicSimilarity], result of:
           3.9710593 = score(doc=394,freq=1.0), product of:
             0.68293065 = queryWeight, product of:
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.073405206 = queryNorm
             5.814733 = fieldWeight in 394, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.625 = fieldNorm(doc=394)
       0.5 = coord(1/2)
   

Similar documents (content)

1. Cai, X.; Li, W.: Enhancing sentence-level clustering with integrated and interactive frameworks for theme-based summarization (2011) 0.23

   0.22541879 = sum of:
     0.22541879 = product of:
       0.8050671 = sum of:
         0.014475003 = weight(abstract_txt:text in 770) [ClassicSimilarity], result of:
           0.014475003 = score(doc=770,freq=1.0), product of:
             0.05731423 = queryWeight, product of:
               1.1646378 = boost
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.012178545 = queryNorm
             0.25255513 = fieldWeight in 770, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.0625 = fieldNorm(doc=770)
         0.02197797 = weight(abstract_txt:quality in 770) [ClassicSimilarity], result of:
           0.02197797 = score(doc=770,freq=1.0), product of:
             0.075713806 = queryWeight, product of:
               1.3385888 = boost
               4.6444306 = idf(docFreq=1160, maxDocs=44421)
               0.012178545 = queryNorm
             0.2902769 = fieldWeight in 770, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6444306 = idf(docFreq=1160, maxDocs=44421)
               0.0625 = fieldNorm(doc=770)
         0.012451252 = weight(abstract_txt:used in 770) [ClassicSimilarity], result of:
           0.012451252 = score(doc=770,freq=1.0), product of:
             0.059341125 = queryWeight, product of:
               1.4513868 = boost
               3.3572001 = idf(docFreq=4205, maxDocs=44421)
               0.012178545 = queryNorm
             0.20982501 = fieldWeight in 770, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3572001 = idf(docFreq=4205, maxDocs=44421)
               0.0625 = fieldNorm(doc=770)
         0.03786177 = weight(abstract_txt:context in 770) [ClassicSimilarity], result of:
           0.03786177 = score(doc=770,freq=2.0), product of:
             0.09885631 = queryWeight, product of:
               1.8733007 = boost
               4.333128 = idf(docFreq=1584, maxDocs=44421)
               0.012178545 = queryNorm
             0.38299802 = fieldWeight in 770, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.333128 = idf(docFreq=1584, maxDocs=44421)
               0.0625 = fieldNorm(doc=770)
         0.10583898 = weight(abstract_txt:word in 770) [ClassicSimilarity], result of:
           0.10583898 = score(doc=770,freq=4.0), product of:
             0.15570082 = queryWeight, product of:
               2.3509898 = boost
               5.4380693 = idf(docFreq=524, maxDocs=44421)
               0.012178545 = queryNorm
             0.67975867 = fieldWeight in 770, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.4380693 = idf(docFreq=524, maxDocs=44421)
               0.0625 = fieldNorm(doc=770)
         0.51135266 = weight(abstract_txt:clustering in 770) [ClassicSimilarity], result of:
           0.51135266 = score(doc=770,freq=15.0), product of:
             0.33958355 = queryWeight, product of:
               4.4823213 = boost
               6.2208285 = idf(docFreq=239, maxDocs=44421)
               0.012178545 = queryNorm
             1.5058229 = fieldWeight in 770, product of:
               3.8729835 = tf(freq=15.0), with freq of:
                 15.0 = termFreq=15.0
               6.2208285 = idf(docFreq=239, maxDocs=44421)
               0.0625 = fieldNorm(doc=770)
         0.10110951 = weight(abstract_txt:words in 770) [ClassicSimilarity], result of:
           0.10110951 = score(doc=770,freq=1.0), product of:
             0.30205438 = queryWeight, product of:
               4.6308727 = boost
               5.355831 = idf(docFreq=569, maxDocs=44421)
               0.012178545 = queryNorm
             0.33473945 = fieldWeight in 770, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.355831 = idf(docFreq=569, maxDocs=44421)
               0.0625 = fieldNorm(doc=770)
       0.28 = coord(7/25)
   

2. Kalish, M.: Immersion multimedia for adult Chiricahua language learners (2005) 0.17

   0.16733254 = sum of:
     0.16733254 = product of:
       0.59761626 = sum of:
         0.0054406286 = weight(abstract_txt:that in 916) [ClassicSimilarity], result of:
           0.0054406286 = score(doc=916,freq=1.0), product of:
             0.029446917 = queryWeight, product of:
               1.0224103 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.012178545 = queryNorm
             0.18476056 = fieldWeight in 916, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.078125 = fieldNorm(doc=916)
         0.018093754 = weight(abstract_txt:text in 916) [ClassicSimilarity], result of:
           0.018093754 = score(doc=916,freq=1.0), product of:
             0.05731423 = queryWeight, product of:
               1.1646378 = boost
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.012178545 = queryNorm
             0.3156939 = fieldWeight in 916, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.078125 = fieldNorm(doc=916)
         0.047327213 = weight(abstract_txt:context in 916) [ClassicSimilarity], result of:
           0.047327213 = score(doc=916,freq=2.0), product of:
             0.09885631 = queryWeight, product of:
               1.8733007 = boost
               4.333128 = idf(docFreq=1584, maxDocs=44421)
               0.012178545 = queryNorm
             0.47874752 = fieldWeight in 916, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.333128 = idf(docFreq=1584, maxDocs=44421)
               0.078125 = fieldNorm(doc=916)
         0.093549326 = weight(abstract_txt:word in 916) [ClassicSimilarity], result of:
           0.093549326 = score(doc=916,freq=2.0), product of:
             0.15570082 = queryWeight, product of:
               2.3509898 = boost
               5.4380693 = idf(docFreq=524, maxDocs=44421)
               0.012178545 = queryNorm
             0.60082746 = fieldWeight in 916, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.4380693 = idf(docFreq=524, maxDocs=44421)
               0.078125 = fieldNorm(doc=916)
         0.040581338 = weight(abstract_txt:were in 916) [ClassicSimilarity], result of:
           0.040581338 = score(doc=916,freq=1.0), product of:
             0.14163417 = queryWeight, product of:
               3.171059 = boost
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.012178545 = queryNorm
             0.28652224 = fieldWeight in 916, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.078125 = fieldNorm(doc=916)
         0.17873806 = weight(abstract_txt:words in 916) [ClassicSimilarity], result of:
           0.17873806 = score(doc=916,freq=2.0), product of:
             0.30205438 = queryWeight, product of:
               4.6308727 = boost
               5.355831 = idf(docFreq=569, maxDocs=44421)
               0.012178545 = queryNorm
             0.5917413 = fieldWeight in 916, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.355831 = idf(docFreq=569, maxDocs=44421)
               0.078125 = fieldNorm(doc=916)
         0.21388593 = weight(abstract_txt:factor in 916) [ClassicSimilarity], result of:
           0.21388593 = score(doc=916,freq=1.0), product of:
             0.47211903 = queryWeight, product of:
               6.6852155 = boost
               5.7988343 = idf(docFreq=365, maxDocs=44421)
               0.012178545 = queryNorm
             0.45303392 = fieldWeight in 916, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7988343 = idf(docFreq=365, maxDocs=44421)
               0.078125 = fieldNorm(doc=916)
       0.28 = coord(7/25)
   

3. Leydesdorff, L.; Zhou, P.: Co-word analysis using the Chinese character set (2008) 0.15

   0.15475577 = sum of:
     0.15475577 = product of:
       0.77377886 = sum of:
         0.0065287547 = weight(abstract_txt:that in 2970) [ClassicSimilarity], result of:
           0.0065287547 = score(doc=2970,freq=1.0), product of:
             0.029446917 = queryWeight, product of:
               1.0224103 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.012178545 = queryNorm
             0.22171268 = fieldWeight in 2970, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.09375 = fieldNorm(doc=2970)
         0.13748886 = weight(abstract_txt:word in 2970) [ClassicSimilarity], result of:
           0.13748886 = score(doc=2970,freq=3.0), product of:
             0.15570082 = queryWeight, product of:
               2.3509898 = boost
               5.4380693 = idf(docFreq=524, maxDocs=44421)
               0.012178545 = queryNorm
             0.8830324 = fieldWeight in 2970, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.4380693 = idf(docFreq=524, maxDocs=44421)
               0.09375 = fieldNorm(doc=2970)
         0.15861246 = weight(abstract_txt:matrix in 2970) [ClassicSimilarity], result of:
           0.15861246 = score(doc=2970,freq=1.0), product of:
             0.24700807 = queryWeight, product of:
               2.961153 = boost
               6.849437 = idf(docFreq=127, maxDocs=44421)
               0.012178545 = queryNorm
             0.6421347 = fieldWeight in 2970, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.849437 = idf(docFreq=127, maxDocs=44421)
               0.09375 = fieldNorm(doc=2970)
         0.21448566 = weight(abstract_txt:words in 2970) [ClassicSimilarity], result of:
           0.21448566 = score(doc=2970,freq=2.0), product of:
             0.30205438 = queryWeight, product of:
               4.6308727 = boost
               5.355831 = idf(docFreq=569, maxDocs=44421)
               0.012178545 = queryNorm
             0.71008956 = fieldWeight in 2970, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.355831 = idf(docFreq=569, maxDocs=44421)
               0.09375 = fieldNorm(doc=2970)
         0.25666314 = weight(abstract_txt:factor in 2970) [ClassicSimilarity], result of:
           0.25666314 = score(doc=2970,freq=1.0), product of:
             0.47211903 = queryWeight, product of:
               6.6852155 = boost
               5.7988343 = idf(docFreq=365, maxDocs=44421)
               0.012178545 = queryNorm
             0.54364073 = fieldWeight in 2970, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7988343 = idf(docFreq=365, maxDocs=44421)
               0.09375 = fieldNorm(doc=2970)
       0.2 = coord(5/25)
   

4. Lund, K.; Burgess, C.; Atchley, R.A.: Semantic and associative priming in high-dimensional semantic space (1995) 0.15

   0.14751989 = sum of:
     0.14751989 = product of:
       0.73759943 = sum of:
         0.010771895 = weight(abstract_txt:that in 3151) [ClassicSimilarity], result of:
           0.010771895 = score(doc=3151,freq=2.0), product of:
             0.029446917 = queryWeight, product of:
               1.0224103 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.012178545 = queryNorm
             0.36580724 = fieldWeight in 3151, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.109375 = fieldNorm(doc=3151)
         0.13096906 = weight(abstract_txt:word in 3151) [ClassicSimilarity], result of:
           0.13096906 = score(doc=3151,freq=2.0), product of:
             0.15570082 = queryWeight, product of:
               2.3509898 = boost
               5.4380693 = idf(docFreq=524, maxDocs=44421)
               0.012178545 = queryNorm
             0.84115845 = fieldWeight in 3151, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.4380693 = idf(docFreq=524, maxDocs=44421)
               0.109375 = fieldNorm(doc=3151)
         0.32051232 = weight(abstract_txt:matrix in 3151) [ClassicSimilarity], result of:
           0.32051232 = score(doc=3151,freq=3.0), product of:
             0.24700807 = queryWeight, product of:
               2.961153 = boost
               6.849437 = idf(docFreq=127, maxDocs=44421)
               0.012178545 = queryNorm
             1.2975783 = fieldWeight in 3151, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.849437 = idf(docFreq=127, maxDocs=44421)
               0.109375 = fieldNorm(doc=3151)
         0.09840452 = weight(abstract_txt:were in 3151) [ClassicSimilarity], result of:
           0.09840452 = score(doc=3151,freq=3.0), product of:
             0.14163417 = queryWeight, product of:
               3.171059 = boost
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.012178545 = queryNorm
             0.6947795 = fieldWeight in 3151, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.109375 = fieldNorm(doc=3151)
         0.17694165 = weight(abstract_txt:words in 3151) [ClassicSimilarity], result of:
           0.17694165 = score(doc=3151,freq=1.0), product of:
             0.30205438 = queryWeight, product of:
               4.6308727 = boost
               5.355831 = idf(docFreq=569, maxDocs=44421)
               0.012178545 = queryNorm
             0.58579403 = fieldWeight in 3151, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.355831 = idf(docFreq=569, maxDocs=44421)
               0.109375 = fieldNorm(doc=3151)
       0.2 = coord(5/25)
   

5. Lee, G.E.; Sun, A.: Understanding the stability of medical concept embeddings (2021) 0.14

   0.14285062 = sum of:
     0.14285062 = product of:
       0.59521097 = sum of:
         0.004352503 = weight(abstract_txt:that in 1160) [ClassicSimilarity], result of:
           0.004352503 = score(doc=1160,freq=1.0), product of:
             0.029446917 = queryWeight, product of:
               1.0224103 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.012178545 = queryNorm
             0.14780845 = fieldWeight in 1160, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=1160)
         0.02197797 = weight(abstract_txt:quality in 1160) [ClassicSimilarity], result of:
           0.02197797 = score(doc=1160,freq=1.0), product of:
             0.075713806 = queryWeight, product of:
               1.3385888 = boost
               4.6444306 = idf(docFreq=1160, maxDocs=44421)
               0.012178545 = queryNorm
             0.2902769 = fieldWeight in 1160, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6444306 = idf(docFreq=1160, maxDocs=44421)
               0.0625 = fieldNorm(doc=1160)
         0.03786177 = weight(abstract_txt:context in 1160) [ClassicSimilarity], result of:
           0.03786177 = score(doc=1160,freq=2.0), product of:
             0.09885631 = queryWeight, product of:
               1.8733007 = boost
               4.333128 = idf(docFreq=1584, maxDocs=44421)
               0.012178545 = queryNorm
             0.38299802 = fieldWeight in 1160, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.333128 = idf(docFreq=1584, maxDocs=44421)
               0.0625 = fieldNorm(doc=1160)
         0.09165924 = weight(abstract_txt:word in 1160) [ClassicSimilarity], result of:
           0.09165924 = score(doc=1160,freq=3.0), product of:
             0.15570082 = queryWeight, product of:
               2.3509898 = boost
               5.4380693 = idf(docFreq=524, maxDocs=44421)
               0.012178545 = queryNorm
             0.58868825 = fieldWeight in 1160, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.4380693 = idf(docFreq=524, maxDocs=44421)
               0.0625 = fieldNorm(doc=1160)
         0.14299044 = weight(abstract_txt:words in 1160) [ClassicSimilarity], result of:
           0.14299044 = score(doc=1160,freq=2.0), product of:
             0.30205438 = queryWeight, product of:
               4.6308727 = boost
               5.355831 = idf(docFreq=569, maxDocs=44421)
               0.012178545 = queryNorm
             0.47339305 = fieldWeight in 1160, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.355831 = idf(docFreq=569, maxDocs=44421)
               0.0625 = fieldNorm(doc=1160)
         0.29636905 = weight(abstract_txt:factor in 1160) [ClassicSimilarity], result of:
           0.29636905 = score(doc=1160,freq=3.0), product of:
             0.47211903 = queryWeight, product of:
               6.6852155 = boost
               5.7988343 = idf(docFreq=365, maxDocs=44421)
               0.012178545 = queryNorm
             0.62774223 = fieldWeight in 1160, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.7988343 = idf(docFreq=365, maxDocs=44421)
               0.0625 = fieldNorm(doc=1160)
       0.24 = coord(6/25)