Document (#33021)

Author

Xu, J.
Weischedel, R.

Title

Empirical studies on the impact of lexical resources on CLIR performance

Source

Information processing and management. 41(2005) no.3, S.475-488

Year

2005

Abstract

In this paper, we compile and review several experiments measuring cross-lingual information retrieval (CLIR) performance as a function of the following resources: bilingual term lists, parallel corpora, machine translation (MT), and stemmers. Our CLIR system uses a simple probabilistic language model; the studies used TREC test corpora over Chinese, Spanish and Arabic. Our findings include: One can achieve an acceptable CLIR performance using only a bilingual term list (70-80% on Chinese and Arabic corpora). However, if a bilingual term list and parallel corpora are available, CLIR performance can rival monolingual performance. If no parallel corpus is available, pseudo-parallel texts produced by an MT system can partially overcome the lack of parallel text. While stemming is useful normally, with a very large parallel corpus for Arabic-English, stemming hurt performance in our empirical studies with Arabic, a highly inflected language.

Theme

Multilinguale Probleme

Similar documents (content)

1. Larkey, L.S.; Connell, M.E.: Structured queries, language modelling, and relevance modelling in cross-language information retrieval (2005) 0.33

   0.32904968 = sum of:
     0.32904968 = product of:
       1.0282803 = sum of:
         0.07170549 = weight(abstract_txt:pseudo in 2022) [ClassicSimilarity], result of:
           0.07170549 = score(doc=2022,freq=2.0), product of:
             0.10156055 = queryWeight, product of:
               1.11257 = boost
               7.9878955 = idf(docFreq=40, maxDocs=44421)
               0.01142787 = queryNorm
             0.70603687 = fieldWeight in 2022, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.9878955 = idf(docFreq=40, maxDocs=44421)
               0.0625 = fieldNorm(doc=2022)
         0.053220525 = weight(abstract_txt:lingual in 2022) [ClassicSimilarity], result of:
           0.053220525 = score(doc=2022,freq=1.0), product of:
             0.104894534 = queryWeight, product of:
               1.130684 = boost
               8.117949 = idf(docFreq=35, maxDocs=44421)
               0.01142787 = queryNorm
             0.5073718 = fieldWeight in 2022, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.117949 = idf(docFreq=35, maxDocs=44421)
               0.0625 = fieldNorm(doc=2022)
         0.035364103 = weight(abstract_txt:language in 2022) [ClassicSimilarity], result of:
           0.035364103 = score(doc=2022,freq=6.0), product of:
             0.055381928 = queryWeight, product of:
               1.1618869 = boost
               4.1709876 = idf(docFreq=1863, maxDocs=44421)
               0.01142787 = queryNorm
             0.6385495 = fieldWeight in 2022, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.1709876 = idf(docFreq=1863, maxDocs=44421)
               0.0625 = fieldNorm(doc=2022)
         0.063615665 = weight(abstract_txt:corpus in 2022) [ClassicSimilarity], result of:
           0.063615665 = score(doc=2022,freq=2.0), product of:
             0.118143395 = queryWeight, product of:
               1.6970106 = boost
               6.0919957 = idf(docFreq=272, maxDocs=44421)
               0.01142787 = queryNorm
             0.53846145 = fieldWeight in 2022, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.0919957 = idf(docFreq=272, maxDocs=44421)
               0.0625 = fieldNorm(doc=2022)
         0.1340536 = weight(abstract_txt:bilingual in 2022) [ClassicSimilarity], result of:
           0.1340536 = score(doc=2022,freq=1.0), product of:
             0.28006542 = queryWeight, product of:
               3.2000384 = boost
               7.6584163 = idf(docFreq=56, maxDocs=44421)
               0.01142787 = queryNorm
             0.47865102 = fieldWeight in 2022, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.6584163 = idf(docFreq=56, maxDocs=44421)
               0.0625 = fieldNorm(doc=2022)
         0.17291534 = weight(abstract_txt:arabic in 2022) [ClassicSimilarity], result of:
           0.17291534 = score(doc=2022,freq=1.0), product of:
             0.36526588 = queryWeight, product of:
               4.219873 = boost
               7.574333 = idf(docFreq=61, maxDocs=44421)
               0.01142787 = queryNorm
             0.47339582 = fieldWeight in 2022, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.574333 = idf(docFreq=61, maxDocs=44421)
               0.0625 = fieldNorm(doc=2022)
         0.22265421 = weight(abstract_txt:parallel in 2022) [ClassicSimilarity], result of:
           0.22265421 = score(doc=2022,freq=2.0), product of:
             0.39279062 = queryWeight, product of:
               5.35946 = boost
               6.4132004 = idf(docFreq=197, maxDocs=44421)
               0.01142787 = queryNorm
             0.56685215 = fieldWeight in 2022, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.4132004 = idf(docFreq=197, maxDocs=44421)
               0.0625 = fieldNorm(doc=2022)
         0.27475125 = weight(abstract_txt:clir in 2022) [ClassicSimilarity], result of:
           0.27475125 = score(doc=2022,freq=1.0), product of:
             0.5357759 = queryWeight, product of:
               5.714016 = boost
               8.20496 = idf(docFreq=32, maxDocs=44421)
               0.01142787 = queryNorm
             0.51281 = fieldWeight in 2022, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.20496 = idf(docFreq=32, maxDocs=44421)
               0.0625 = fieldNorm(doc=2022)
       0.32 = coord(8/25)
   

2. Yang, C.C.; Li, K.W.: Automatic construction of English/Chinese parallel corpora (2003) 0.26

   0.2616556 = sum of:
     0.2616556 = product of:
       0.8176738 = sum of:
         0.06585704 = weight(abstract_txt:lingual in 2683) [ClassicSimilarity], result of:
           0.06585704 = score(doc=2683,freq=2.0), product of:
             0.104894534 = queryWeight, product of:
               1.130684 = boost
               8.117949 = idf(docFreq=35, maxDocs=44421)
               0.01142787 = queryNorm
             0.6278405 = fieldWeight in 2683, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.117949 = idf(docFreq=35, maxDocs=44421)
               0.0546875 = fieldNorm(doc=2683)
         0.025265332 = weight(abstract_txt:language in 2683) [ClassicSimilarity], result of:
           0.025265332 = score(doc=2683,freq=4.0), product of:
             0.055381928 = queryWeight, product of:
               1.1618869 = boost
               4.1709876 = idf(docFreq=1863, maxDocs=44421)
               0.01142787 = queryNorm
             0.45620176 = fieldWeight in 2683, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.1709876 = idf(docFreq=1863, maxDocs=44421)
               0.0546875 = fieldNorm(doc=2683)
         0.06817384 = weight(abstract_txt:corpus in 2683) [ClassicSimilarity], result of:
           0.06817384 = score(doc=2683,freq=3.0), product of:
             0.118143395 = queryWeight, product of:
               1.6970106 = boost
               6.0919957 = idf(docFreq=272, maxDocs=44421)
               0.01142787 = queryNorm
             0.5770432 = fieldWeight in 2683, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.0919957 = idf(docFreq=272, maxDocs=44421)
               0.0546875 = fieldNorm(doc=2683)
         0.08701213 = weight(abstract_txt:chinese in 2683) [ClassicSimilarity], result of:
           0.08701213 = score(doc=2683,freq=4.0), product of:
             0.12630035 = queryWeight, product of:
               1.7546161 = boost
               6.2987905 = idf(docFreq=221, maxDocs=44421)
               0.01142787 = queryNorm
             0.6889302 = fieldWeight in 2683, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.2987905 = idf(docFreq=221, maxDocs=44421)
               0.0546875 = fieldNorm(doc=2683)
         0.1172969 = weight(abstract_txt:bilingual in 2683) [ClassicSimilarity], result of:
           0.1172969 = score(doc=2683,freq=1.0), product of:
             0.28006542 = queryWeight, product of:
               3.2000384 = boost
               7.6584163 = idf(docFreq=56, maxDocs=44421)
               0.01142787 = queryNorm
             0.41881964 = fieldWeight in 2683, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.6584163 = idf(docFreq=56, maxDocs=44421)
               0.0546875 = fieldNorm(doc=2683)
         0.051494114 = weight(abstract_txt:performance in 2683) [ClassicSimilarity], result of:
           0.051494114 = score(doc=2683,freq=1.0), product of:
             0.20382158 = queryWeight, product of:
               3.8606956 = boost
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.01142787 = queryNorm
             0.25264308 = fieldWeight in 2683, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.0546875 = fieldNorm(doc=2683)
         0.20775205 = weight(abstract_txt:corpora in 2683) [ClassicSimilarity], result of:
           0.20775205 = score(doc=2683,freq=3.0), product of:
             0.3128752 = queryWeight, product of:
               3.905534 = boost
               7.01012 = idf(docFreq=108, maxDocs=44421)
               0.01142787 = queryNorm
             0.6640093 = fieldWeight in 2683, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.01012 = idf(docFreq=108, maxDocs=44421)
               0.0546875 = fieldNorm(doc=2683)
         0.19482243 = weight(abstract_txt:parallel in 2683) [ClassicSimilarity], result of:
           0.19482243 = score(doc=2683,freq=2.0), product of:
             0.39279062 = queryWeight, product of:
               5.35946 = boost
               6.4132004 = idf(docFreq=197, maxDocs=44421)
               0.01142787 = queryNorm
             0.49599564 = fieldWeight in 2683, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.4132004 = idf(docFreq=197, maxDocs=44421)
               0.0546875 = fieldNorm(doc=2683)
       0.32 = coord(8/25)
   

3. Chen, J.: ¬A lexical knowledge base approach for English-Chinese cross-language information retrieval (2006) 0.24

   0.24418904 = sum of:
     0.24418904 = product of:
       0.8721037 = sum of:
         0.020417472 = weight(abstract_txt:language in 5923) [ClassicSimilarity], result of:
           0.020417472 = score(doc=5923,freq=2.0), product of:
             0.055381928 = queryWeight, product of:
               1.1618869 = boost
               4.1709876 = idf(docFreq=1863, maxDocs=44421)
               0.01142787 = queryNorm
             0.3686667 = fieldWeight in 5923, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1709876 = idf(docFreq=1863, maxDocs=44421)
               0.0625 = fieldNorm(doc=5923)
         0.021203859 = weight(abstract_txt:resources in 5923) [ClassicSimilarity], result of:
           0.021203859 = score(doc=5923,freq=2.0), product of:
             0.05679499 = queryWeight, product of:
               1.1766163 = boost
               4.2238636 = idf(docFreq=1767, maxDocs=44421)
               0.01142787 = queryNorm
             0.3733403 = fieldWeight in 5923, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.2238636 = idf(docFreq=1767, maxDocs=44421)
               0.0625 = fieldNorm(doc=5923)
         0.015868805 = weight(abstract_txt:available in 5923) [ClassicSimilarity], result of:
           0.015868805 = score(doc=5923,freq=1.0), product of:
             0.058984727 = queryWeight, product of:
               1.199084 = boost
               4.304519 = idf(docFreq=1630, maxDocs=44421)
               0.01142787 = queryNorm
             0.26903245 = fieldWeight in 5923, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.304519 = idf(docFreq=1630, maxDocs=44421)
               0.0625 = fieldNorm(doc=5923)
         0.028002791 = weight(abstract_txt:empirical in 5923) [ClassicSimilarity], result of:
           0.028002791 = score(doc=5923,freq=1.0), product of:
             0.0861346 = queryWeight, product of:
               1.449001 = boost
               5.2016807 = idf(docFreq=664, maxDocs=44421)
               0.01142787 = queryNorm
             0.32510504 = fieldWeight in 5923, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.2016807 = idf(docFreq=664, maxDocs=44421)
               0.0625 = fieldNorm(doc=5923)
         0.07031642 = weight(abstract_txt:chinese in 5923) [ClassicSimilarity], result of:
           0.07031642 = score(doc=5923,freq=2.0), product of:
             0.12630035 = queryWeight, product of:
               1.7546161 = boost
               6.2987905 = idf(docFreq=221, maxDocs=44421)
               0.01142787 = queryNorm
             0.5567397 = fieldWeight in 5923, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.2987905 = idf(docFreq=221, maxDocs=44421)
               0.0625 = fieldNorm(doc=5923)
         0.10193191 = weight(abstract_txt:performance in 5923) [ClassicSimilarity], result of:
           0.10193191 = score(doc=5923,freq=3.0), product of:
             0.20382158 = queryWeight, product of:
               3.8606956 = boost
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.01142787 = queryNorm
             0.5001036 = fieldWeight in 5923, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.0625 = fieldNorm(doc=5923)
         0.6143624 = weight(abstract_txt:clir in 5923) [ClassicSimilarity], result of:
           0.6143624 = score(doc=5923,freq=5.0), product of:
             0.5357759 = queryWeight, product of:
               5.714016 = boost
               8.20496 = idf(docFreq=32, maxDocs=44421)
               0.01142787 = queryNorm
             1.146678 = fieldWeight in 5923, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               8.20496 = idf(docFreq=32, maxDocs=44421)
               0.0625 = fieldNorm(doc=5923)
       0.28 = coord(7/25)
   

4. Perea-Ortega, J.M.; Martín-Valdivia, M.T.; Ureña-López, L.A.; Martínez-Cámara, E.: Improving polarity classification of bilingual parallel corpora combining machine learning and semantic orientation approaches (2013) 0.24

   0.23584723 = sum of:
     0.23584723 = product of:
       0.84231156 = sum of:
         0.014993393 = weight(abstract_txt:resources in 2045) [ClassicSimilarity], result of:
           0.014993393 = score(doc=2045,freq=1.0), product of:
             0.05679499 = queryWeight, product of:
               1.1766163 = boost
               4.2238636 = idf(docFreq=1767, maxDocs=44421)
               0.01142787 = queryNorm
             0.26399148 = fieldWeight in 2045, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.2238636 = idf(docFreq=1767, maxDocs=44421)
               0.0625 = fieldNorm(doc=2045)
         0.044983067 = weight(abstract_txt:corpus in 2045) [ClassicSimilarity], result of:
           0.044983067 = score(doc=2045,freq=1.0), product of:
             0.118143395 = queryWeight, product of:
               1.6970106 = boost
               6.0919957 = idf(docFreq=272, maxDocs=44421)
               0.01142787 = queryNorm
             0.38074973 = fieldWeight in 2045, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.0919957 = idf(docFreq=272, maxDocs=44421)
               0.0625 = fieldNorm(doc=2045)
         0.1340536 = weight(abstract_txt:bilingual in 2045) [ClassicSimilarity], result of:
           0.1340536 = score(doc=2045,freq=1.0), product of:
             0.28006542 = queryWeight, product of:
               3.2000384 = boost
               7.6584163 = idf(docFreq=56, maxDocs=44421)
               0.01142787 = queryNorm
             0.47865102 = fieldWeight in 2045, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.6584163 = idf(docFreq=56, maxDocs=44421)
               0.0625 = fieldNorm(doc=2045)
         0.058850415 = weight(abstract_txt:performance in 2045) [ClassicSimilarity], result of:
           0.058850415 = score(doc=2045,freq=1.0), product of:
             0.20382158 = queryWeight, product of:
               3.8606956 = boost
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.01142787 = queryNorm
             0.28873494 = fieldWeight in 2045, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.0625 = fieldNorm(doc=2045)
         0.19386153 = weight(abstract_txt:corpora in 2045) [ClassicSimilarity], result of:
           0.19386153 = score(doc=2045,freq=2.0), product of:
             0.3128752 = queryWeight, product of:
               3.905534 = boost
               7.01012 = idf(docFreq=108, maxDocs=44421)
               0.01142787 = queryNorm
             0.61961293 = fieldWeight in 2045, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.01012 = idf(docFreq=108, maxDocs=44421)
               0.0625 = fieldNorm(doc=2045)
         0.17291534 = weight(abstract_txt:arabic in 2045) [ClassicSimilarity], result of:
           0.17291534 = score(doc=2045,freq=1.0), product of:
             0.36526588 = queryWeight, product of:
               4.219873 = boost
               7.574333 = idf(docFreq=61, maxDocs=44421)
               0.01142787 = queryNorm
             0.47339582 = fieldWeight in 2045, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.574333 = idf(docFreq=61, maxDocs=44421)
               0.0625 = fieldNorm(doc=2045)
         0.22265421 = weight(abstract_txt:parallel in 2045) [ClassicSimilarity], result of:
           0.22265421 = score(doc=2045,freq=2.0), product of:
             0.39279062 = queryWeight, product of:
               5.35946 = boost
               6.4132004 = idf(docFreq=197, maxDocs=44421)
               0.01142787 = queryNorm
             0.56685215 = fieldWeight in 2045, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.4132004 = idf(docFreq=197, maxDocs=44421)
               0.0625 = fieldNorm(doc=2045)
       0.28 = coord(7/25)
   

5. Dadashkarimia, J.; Shakery, A.; Failia, H.; Zamani, H.: ¬An expectation-maximization algorithm for query translation based on pseudo-relevant documents (2017) 0.23

   0.23466371 = sum of:
     0.23466371 = product of:
       0.8380847 = sum of:
         0.076843314 = weight(abstract_txt:pseudo in 4296) [ClassicSimilarity], result of:
           0.076843314 = score(doc=4296,freq=3.0), product of:
             0.10156055 = queryWeight, product of:
               1.11257 = boost
               7.9878955 = idf(docFreq=40, maxDocs=44421)
               0.01142787 = queryNorm
             0.75662565 = fieldWeight in 4296, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.9878955 = idf(docFreq=40, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4296)
         0.025265332 = weight(abstract_txt:language in 4296) [ClassicSimilarity], result of:
           0.025265332 = score(doc=4296,freq=4.0), product of:
             0.055381928 = queryWeight, product of:
               1.1618869 = boost
               4.1709876 = idf(docFreq=1863, maxDocs=44421)
               0.01142787 = queryNorm
             0.45620176 = fieldWeight in 4296, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.1709876 = idf(docFreq=1863, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4296)
         0.057568833 = weight(abstract_txt:term in 4296) [ClassicSimilarity], result of:
           0.057568833 = score(doc=4296,freq=4.0), product of:
             0.109775834 = queryWeight, product of:
               2.0034516 = boost
               4.794713 = idf(docFreq=998, maxDocs=44421)
               0.01142787 = queryNorm
             0.52442175 = fieldWeight in 4296, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.794713 = idf(docFreq=998, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4296)
         0.1172969 = weight(abstract_txt:bilingual in 4296) [ClassicSimilarity], result of:
           0.1172969 = score(doc=4296,freq=1.0), product of:
             0.28006542 = queryWeight, product of:
               3.2000384 = boost
               7.6584163 = idf(docFreq=56, maxDocs=44421)
               0.01142787 = queryNorm
             0.41881964 = fieldWeight in 4296, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.6584163 = idf(docFreq=56, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4296)
         0.051494114 = weight(abstract_txt:performance in 4296) [ClassicSimilarity], result of:
           0.051494114 = score(doc=4296,freq=1.0), product of:
             0.20382158 = queryWeight, product of:
               3.8606956 = boost
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.01142787 = queryNorm
             0.25264308 = fieldWeight in 4296, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4296)
         0.16962884 = weight(abstract_txt:corpora in 4296) [ClassicSimilarity], result of:
           0.16962884 = score(doc=4296,freq=2.0), product of:
             0.3128752 = queryWeight, product of:
               3.905534 = boost
               7.01012 = idf(docFreq=108, maxDocs=44421)
               0.01142787 = queryNorm
             0.54216135 = fieldWeight in 4296, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.01012 = idf(docFreq=108, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4296)
         0.3399873 = weight(abstract_txt:clir in 4296) [ClassicSimilarity], result of:
           0.3399873 = score(doc=4296,freq=2.0), product of:
             0.5357759 = queryWeight, product of:
               5.714016 = boost
               8.20496 = idf(docFreq=32, maxDocs=44421)
               0.01142787 = queryNorm
             0.63457 = fieldWeight in 4296, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.20496 = idf(docFreq=32, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4296)
       0.28 = coord(7/25)