Document (#10929)

Author

Akman, K.I.

Title

¬A new text compression technique based on natural language structure

Source

Journal of information science. 21(1995) no.2, S.87-94

Year

1995

Abstract

Describes a new data compression technique which utilizes some of the common structural characteristics of languages. The proposed algorithm partitions words into their roots and suffixes which are then replaced by shorter bit representations. The method used 3 dictionaries in the from of binary search trees and 1 character array. The first 2 dictionaries are for roots, and the third one is for suffixes. The character array is used for both searching compressible words and coding incompressible words. The number of bits in representing a substring depends on the number of the entries in the dictionary in which the substring is found. The proposed algorithm is implemented in the Turkish language and tested using 3 different text groups with different lenghts. Results indicate a compression factor of up to 47 per cent

Theme

Computerlinguistik

Similar documents (content)

1. Ucoluk, G.; Toroslu, I.H.: ¬A genetic algorithm approach for verification of the syllable-based text compression technique (1997) 0.23

   0.23150826 = sum of:
     0.23150826 = product of:
       0.7234633 = sum of:
         0.06411296 = weight(abstract_txt:coding in 3601) [ClassicSimilarity], result of:
           0.06411296 = score(doc=3601,freq=1.0), product of:
             0.12140046 = queryWeight, product of:
               6.759825 = idf(docFreq=139, maxDocs=44421)
               0.017959112 = queryNorm
             0.52811134 = fieldWeight in 3601, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.759825 = idf(docFreq=139, maxDocs=44421)
               0.078125 = fieldNorm(doc=3601)
         0.047441516 = weight(abstract_txt:text in 3601) [ClassicSimilarity], result of:
           0.047441516 = score(doc=3601,freq=3.0), product of:
             0.086762436 = queryWeight, product of:
               1.1955582 = boost
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.017959112 = queryNorm
             0.5467979 = fieldWeight in 3601, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.078125 = fieldNorm(doc=3601)
         0.030122163 = weight(abstract_txt:language in 3601) [ClassicSimilarity], result of:
           0.030122163 = score(doc=3601,freq=1.0), product of:
             0.09243942 = queryWeight, product of:
               1.234052 = boost
               4.1709876 = idf(docFreq=1863, maxDocs=44421)
               0.017959112 = queryNorm
             0.3258584 = fieldWeight in 3601, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1709876 = idf(docFreq=1863, maxDocs=44421)
               0.078125 = fieldNorm(doc=3601)
         0.015404057 = weight(abstract_txt:which in 3601) [ClassicSimilarity], result of:
           0.015404057 = score(doc=3601,freq=1.0), product of:
             0.06766856 = queryWeight, product of:
               1.2931352 = boost
               2.9137893 = idf(docFreq=6552, maxDocs=44421)
               0.017959112 = queryNorm
             0.2276398 = fieldWeight in 3601, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.9137893 = idf(docFreq=6552, maxDocs=44421)
               0.078125 = fieldNorm(doc=3601)
         0.14775091 = weight(abstract_txt:turkish in 3601) [ClassicSimilarity], result of:
           0.14775091 = score(doc=3601,freq=1.0), product of:
             0.21180843 = queryWeight, product of:
               1.3208742 = boost
               8.928879 = idf(docFreq=15, maxDocs=44421)
               0.017959112 = queryNorm
             0.69756866 = fieldWeight in 3601, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.928879 = idf(docFreq=15, maxDocs=44421)
               0.078125 = fieldNorm(doc=3601)
         0.072684266 = weight(abstract_txt:technique in 3601) [ClassicSimilarity], result of:
           0.072684266 = score(doc=3601,freq=1.0), product of:
             0.16630043 = queryWeight, product of:
               1.6552039 = boost
               5.5944448 = idf(docFreq=448, maxDocs=44421)
               0.017959112 = queryNorm
             0.437066 = fieldWeight in 3601, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5944448 = idf(docFreq=448, maxDocs=44421)
               0.078125 = fieldNorm(doc=3601)
         0.07707968 = weight(abstract_txt:algorithm in 3601) [ClassicSimilarity], result of:
           0.07707968 = score(doc=3601,freq=1.0), product of:
             0.17293905 = queryWeight, product of:
               1.6879181 = boost
               5.7050157 = idf(docFreq=401, maxDocs=44421)
               0.017959112 = queryNorm
             0.44570434 = fieldWeight in 3601, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7050157 = idf(docFreq=401, maxDocs=44421)
               0.078125 = fieldNorm(doc=3601)
         0.26886776 = weight(abstract_txt:compression in 3601) [ClassicSimilarity], result of:
           0.26886776 = score(doc=3601,freq=1.0), product of:
             0.45532626 = queryWeight, product of:
               3.3543766 = boost
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.017959112 = queryNorm
             0.59049475 = fieldWeight in 3601, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.078125 = fieldNorm(doc=3601)
       0.32 = coord(8/25)
   

2. Cheng, K.-S.; Young, G.H.; Wong, K.-F.: ¬A study on word-based and integral-bit Chinese text compression algorithms (1999) 0.21

   0.20514548 = sum of:
     0.20514548 = product of:
       1.0257274 = sum of:
         0.12693718 = weight(abstract_txt:coding in 4056) [ClassicSimilarity], result of:
           0.12693718 = score(doc=4056,freq=2.0), product of:
             0.12140046 = queryWeight, product of:
               6.759825 = idf(docFreq=139, maxDocs=44421)
               0.017959112 = queryNorm
             1.0456071 = fieldWeight in 4056, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.759825 = idf(docFreq=139, maxDocs=44421)
               0.109375 = fieldNorm(doc=4056)
         0.03834652 = weight(abstract_txt:text in 4056) [ClassicSimilarity], result of:
           0.03834652 = score(doc=4056,freq=1.0), product of:
             0.086762436 = queryWeight, product of:
               1.1955582 = boost
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.017959112 = queryNorm
             0.44197148 = fieldWeight in 4056, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.109375 = fieldNorm(doc=4056)
         0.021565678 = weight(abstract_txt:which in 4056) [ClassicSimilarity], result of:
           0.021565678 = score(doc=4056,freq=1.0), product of:
             0.06766856 = queryWeight, product of:
               1.2931352 = boost
               2.9137893 = idf(docFreq=6552, maxDocs=44421)
               0.017959112 = queryNorm
             0.3186957 = fieldWeight in 4056, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.9137893 = idf(docFreq=6552, maxDocs=44421)
               0.109375 = fieldNorm(doc=4056)
         0.1869083 = weight(abstract_txt:algorithm in 4056) [ClassicSimilarity], result of:
           0.1869083 = score(doc=4056,freq=3.0), product of:
             0.17293905 = queryWeight, product of:
               1.6879181 = boost
               5.7050157 = idf(docFreq=401, maxDocs=44421)
               0.017959112 = queryNorm
             1.0807756 = fieldWeight in 4056, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.7050157 = idf(docFreq=401, maxDocs=44421)
               0.109375 = fieldNorm(doc=4056)
         0.6519697 = weight(abstract_txt:compression in 4056) [ClassicSimilarity], result of:
           0.6519697 = score(doc=4056,freq=3.0), product of:
             0.45532626 = queryWeight, product of:
               3.3543766 = boost
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.017959112 = queryNorm
             1.4318737 = fieldWeight in 4056, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.109375 = fieldNorm(doc=4056)
       0.2 = coord(5/25)
   

3. Cannane, A.; Williams, H.E.: General-purpose compression for efficient retrieval (2001) 0.20

   0.2035489 = sum of:
     0.2035489 = product of:
       1.0177444 = sum of:
         0.027390374 = weight(abstract_txt:text in 6705) [ClassicSimilarity], result of:
           0.027390374 = score(doc=6705,freq=1.0), product of:
             0.086762436 = queryWeight, product of:
               1.1955582 = boost
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.017959112 = queryNorm
             0.3156939 = fieldWeight in 6705, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.078125 = fieldNorm(doc=6705)
         0.015404057 = weight(abstract_txt:which in 6705) [ClassicSimilarity], result of:
           0.015404057 = score(doc=6705,freq=1.0), product of:
             0.06766856 = queryWeight, product of:
               1.2931352 = boost
               2.9137893 = idf(docFreq=6552, maxDocs=44421)
               0.017959112 = queryNorm
             0.2276398 = fieldWeight in 6705, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.9137893 = idf(docFreq=6552, maxDocs=44421)
               0.078125 = fieldNorm(doc=6705)
         0.072684266 = weight(abstract_txt:technique in 6705) [ClassicSimilarity], result of:
           0.072684266 = score(doc=6705,freq=1.0), product of:
             0.16630043 = queryWeight, product of:
               1.6552039 = boost
               5.5944448 = idf(docFreq=448, maxDocs=44421)
               0.017959112 = queryNorm
             0.437066 = fieldWeight in 6705, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5944448 = idf(docFreq=448, maxDocs=44421)
               0.078125 = fieldNorm(doc=6705)
         0.0956624 = weight(abstract_txt:words in 6705) [ClassicSimilarity], result of:
           0.0956624 = score(doc=6705,freq=1.0), product of:
             0.22862534 = queryWeight, product of:
               2.3769095 = boost
               5.355831 = idf(docFreq=569, maxDocs=44421)
               0.017959112 = queryNorm
             0.4184243 = fieldWeight in 6705, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.355831 = idf(docFreq=569, maxDocs=44421)
               0.078125 = fieldNorm(doc=6705)
         0.8066034 = weight(abstract_txt:compression in 6705) [ClassicSimilarity], result of:
           0.8066034 = score(doc=6705,freq=9.0), product of:
             0.45532626 = queryWeight, product of:
               3.3543766 = boost
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.017959112 = queryNorm
             1.7714844 = fieldWeight in 6705, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.078125 = fieldNorm(doc=6705)
       0.2 = coord(5/25)
   

4. Wang, F.L.; Yang, C.C.: Mining Web data for Chinese segmentation (2007) 0.15

   0.1530708 = sum of:
     0.1530708 = product of:
       0.5466814 = sum of:
         0.016278245 = weight(abstract_txt:different in 1604) [ClassicSimilarity], result of:
           0.016278245 = score(doc=1604,freq=1.0), product of:
             0.07116692 = queryWeight, product of:
               1.0827892 = boost
               3.6597328 = idf(docFreq=3107, maxDocs=44421)
               0.017959112 = queryNorm
             0.2287333 = fieldWeight in 1604, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6597328 = idf(docFreq=3107, maxDocs=44421)
               0.0625 = fieldNorm(doc=1604)
         0.0219123 = weight(abstract_txt:text in 1604) [ClassicSimilarity], result of:
           0.0219123 = score(doc=1604,freq=1.0), product of:
             0.086762436 = queryWeight, product of:
               1.1955582 = boost
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.017959112 = queryNorm
             0.25255513 = fieldWeight in 1604, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.0625 = fieldNorm(doc=1604)
         0.04173849 = weight(abstract_txt:language in 1604) [ClassicSimilarity], result of:
           0.04173849 = score(doc=1604,freq=3.0), product of:
             0.09243942 = queryWeight, product of:
               1.234052 = boost
               4.1709876 = idf(docFreq=1863, maxDocs=44421)
               0.017959112 = queryNorm
             0.45152265 = fieldWeight in 1604, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.1709876 = idf(docFreq=1863, maxDocs=44421)
               0.0625 = fieldNorm(doc=1604)
         0.03249486 = weight(abstract_txt:proposed in 1604) [ClassicSimilarity], result of:
           0.03249486 = score(doc=1604,freq=1.0), product of:
             0.11282784 = queryWeight, product of:
               1.3633676 = boost
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.017959112 = queryNorm
             0.28800395 = fieldWeight in 1604, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.0625 = fieldNorm(doc=1604)
         0.15104473 = weight(abstract_txt:algorithm in 1604) [ClassicSimilarity], result of:
           0.15104473 = score(doc=1604,freq=6.0), product of:
             0.17293905 = queryWeight, product of:
               1.6879181 = boost
               5.7050157 = idf(docFreq=401, maxDocs=44421)
               0.017959112 = queryNorm
             0.8733986 = fieldWeight in 1604, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               5.7050157 = idf(docFreq=401, maxDocs=44421)
               0.0625 = fieldNorm(doc=1604)
         0.13015296 = weight(abstract_txt:character in 1604) [ClassicSimilarity], result of:
           0.13015296 = score(doc=1604,freq=2.0), product of:
             0.22585644 = queryWeight, product of:
               1.9289504 = boost
               6.519684 = idf(docFreq=177, maxDocs=44421)
               0.017959112 = queryNorm
             0.5762641 = fieldWeight in 1604, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.519684 = idf(docFreq=177, maxDocs=44421)
               0.0625 = fieldNorm(doc=1604)
         0.15305984 = weight(abstract_txt:words in 1604) [ClassicSimilarity], result of:
           0.15305984 = score(doc=1604,freq=4.0), product of:
             0.22862534 = queryWeight, product of:
               2.3769095 = boost
               5.355831 = idf(docFreq=569, maxDocs=44421)
               0.017959112 = queryNorm
             0.6694789 = fieldWeight in 1604, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.355831 = idf(docFreq=569, maxDocs=44421)
               0.0625 = fieldNorm(doc=1604)
       0.28 = coord(7/25)
   

5. Hmeidi, I.I.; Al-Shalabi, R.F.; Al-Taani, A.T.; Najadat, H.; Al-Hazaimeh, S.A.: ¬A novel approach to the extraction of roots from Arabic words using bigrams (2010) 0.14

   0.14394845 = sum of:
     0.14394845 = product of:
       0.5997852 = sum of:
         0.0219123 = weight(abstract_txt:text in 413) [ClassicSimilarity], result of:
           0.0219123 = score(doc=413,freq=1.0), product of:
             0.086762436 = queryWeight, product of:
               1.1955582 = boost
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.017959112 = queryNorm
             0.25255513 = fieldWeight in 413, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.0625 = fieldNorm(doc=413)
         0.02409773 = weight(abstract_txt:language in 413) [ClassicSimilarity], result of:
           0.02409773 = score(doc=413,freq=1.0), product of:
             0.09243942 = queryWeight, product of:
               1.234052 = boost
               4.1709876 = idf(docFreq=1863, maxDocs=44421)
               0.017959112 = queryNorm
             0.26068673 = fieldWeight in 413, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1709876 = idf(docFreq=1863, maxDocs=44421)
               0.0625 = fieldNorm(doc=413)
         0.05628275 = weight(abstract_txt:proposed in 413) [ClassicSimilarity], result of:
           0.05628275 = score(doc=413,freq=3.0), product of:
             0.11282784 = queryWeight, product of:
               1.3633676 = boost
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.017959112 = queryNorm
             0.49883747 = fieldWeight in 413, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.0625 = fieldNorm(doc=413)
         0.061663747 = weight(abstract_txt:algorithm in 413) [ClassicSimilarity], result of:
           0.061663747 = score(doc=413,freq=1.0), product of:
             0.17293905 = queryWeight, product of:
               1.6879181 = boost
               5.7050157 = idf(docFreq=401, maxDocs=44421)
               0.017959112 = queryNorm
             0.35656348 = fieldWeight in 413, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7050157 = idf(docFreq=401, maxDocs=44421)
               0.0625 = fieldNorm(doc=413)
         0.24836938 = weight(abstract_txt:roots in 413) [ClassicSimilarity], result of:
           0.24836938 = score(doc=413,freq=3.0), product of:
             0.3035508 = queryWeight, product of:
               2.2362509 = boost
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.017959112 = queryNorm
             0.8182135 = fieldWeight in 413, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.558333 = idf(docFreq=62, maxDocs=44421)
               0.0625 = fieldNorm(doc=413)
         0.18745926 = weight(abstract_txt:words in 413) [ClassicSimilarity], result of:
           0.18745926 = score(doc=413,freq=6.0), product of:
             0.22862534 = queryWeight, product of:
               2.3769095 = boost
               5.355831 = idf(docFreq=569, maxDocs=44421)
               0.017959112 = queryNorm
             0.81994087 = fieldWeight in 413, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               5.355831 = idf(docFreq=569, maxDocs=44421)
               0.0625 = fieldNorm(doc=413)
       0.24 = coord(6/25)