Document (#33337)

Author

Simeoni, F.
Yakici, M.
Neely, S.
Crestani, F.

Title

Metadata harvesting for content-based distributed information retrieval

Source

Journal of the American Society for Information Science and Technology. 59(2008) no.1, S.12-24

Year

2008

Abstract

We propose an approach to content-based Distributed Information Retrieval based on the periodic and incremental centralization of full-content indices of widely dispersed and autonomously managed document sources. Inspired by the success of the Open Archive Initiative's (OAI) Protocol for metadata harvesting, the approach occupies middle ground between content crawling and distributed retrieval. As in crawling, some data move toward the retrieval process, but it is statistics about the content rather than content itself; this grants more efficient use of network resources and wider scope of application. As in distributed retrieval, some processing is distributed along with the data, but it is indexing rather than retrieval; this reduces the costs of content provision while promoting the simplicity, effectiveness, and responsiveness of retrieval. Overall, we argue that the approach retains the good properties of centralized retrieval without renouncing to cost-effective, large-scale resource pooling. We discuss the requirements associated with the approach and identify two strategies to deploy it on top of the OAI infrastructure. In particular, we define a minimal extension of the OAI protocol which supports the coordinated harvesting of full-content indices and descriptive metadata for content resources. Finally, we report on the implementation of a proof-of-concept prototype service for multimodel content-based retrieval of distributed file collections.

Similar documents (author)

1. Crestani, F.: Combination of similarity measures for effective spoken document retrieval (2003) 5.44

   5.4410844 = sum of:
     5.4410844 = weight(author_txt:crestani in 5690) [ClassicSimilarity], result of:
       5.4410844 = fieldWeight in 5690, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.705735 = idf(docFreq=19, maxDocs=44421)
         0.625 = fieldNorm(doc=5690)
   

2. Crestani, F.; Lee, P.L.: Searching the web by constraining spreading activities (2000) 4.35

   4.3528676 = sum of:
     4.3528676 = weight(author_txt:crestani in 1394) [ClassicSimilarity], result of:
       4.3528676 = fieldWeight in 1394, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.705735 = idf(docFreq=19, maxDocs=44421)
         0.5 = fieldNorm(doc=1394)
   

3. Tombros, T.; Crestani, F.: Users' perception of relevance of spoken documents (2000) 4.35

   4.3528676 = sum of:
     4.3528676 = weight(author_txt:crestani in 5996) [ClassicSimilarity], result of:
       4.3528676 = fieldWeight in 5996, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.705735 = idf(docFreq=19, maxDocs=44421)
         0.5 = fieldNorm(doc=5996)
   

4. Crestani, F.; Du, H.: Written versus spoken queries : a qualitative and quantitative comparative analysis (2006) 4.35

   4.3528676 = sum of:
     4.3528676 = weight(author_txt:crestani in 47) [ClassicSimilarity], result of:
       4.3528676 = fieldWeight in 47, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.705735 = idf(docFreq=19, maxDocs=44421)
         0.5 = fieldNorm(doc=47)
   

5. Crestani, F.; Wu, S.: Testing the cluster hypothesis in distributed information retrieval (2006) 4.35

   4.3528676 = sum of:
     4.3528676 = weight(author_txt:crestani in 1984) [ClassicSimilarity], result of:
       4.3528676 = fieldWeight in 1984, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.705735 = idf(docFreq=19, maxDocs=44421)
         0.5 = fieldNorm(doc=1984)
   

Similar documents (content)

1. Van de Sompel, H.; Young, J.A.; Hickey, T.B.: Using the OAI-PMH ... differently (2003) 0.25

   0.25435746 = sum of:
     0.25435746 = product of:
       0.9084195 = sum of:
         0.018925661 = weight(abstract_txt:resources in 2191) [ClassicSimilarity], result of:
           0.018925661 = score(doc=2191,freq=1.0), product of:
             0.071690425 = queryWeight, product of:
               1.057566 = boost
               4.2238636 = idf(docFreq=1767, maxDocs=44421)
               0.016048845 = queryNorm
             0.26399148 = fieldWeight in 2191, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.2238636 = idf(docFreq=1767, maxDocs=44421)
               0.0625 = fieldNorm(doc=2191)
         0.12220078 = weight(abstract_txt:initiative's in 2191) [ClassicSimilarity], result of:
           0.12220078 = score(doc=2191,freq=1.0), product of:
             0.19730273 = queryWeight, product of:
               1.2405906 = boost
               9.909708 = idf(docFreq=5, maxDocs=44421)
               0.016048845 = queryNorm
             0.61935675 = fieldWeight in 2191, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.909708 = idf(docFreq=5, maxDocs=44421)
               0.0625 = fieldNorm(doc=2191)
         0.12801555 = weight(abstract_txt:protocol in 2191) [ClassicSimilarity], result of:
           0.12801555 = score(doc=2191,freq=3.0), product of:
             0.17778504 = queryWeight, product of:
               1.6654228 = boost
               6.651612 = idf(docFreq=155, maxDocs=44421)
               0.016048845 = queryNorm
             0.7200581 = fieldWeight in 2191, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.651612 = idf(docFreq=155, maxDocs=44421)
               0.0625 = fieldNorm(doc=2191)
         0.10718966 = weight(abstract_txt:metadata in 2191) [ClassicSimilarity], result of:
           0.10718966 = score(doc=2191,freq=6.0), product of:
             0.14349686 = queryWeight, product of:
               1.8324989 = boost
               4.87927 = idf(docFreq=917, maxDocs=44421)
               0.016048845 = queryNorm
             0.74698263 = fieldWeight in 2191, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.87927 = idf(docFreq=917, maxDocs=44421)
               0.0625 = fieldNorm(doc=2191)
         0.2360828 = weight(abstract_txt:harvesting in 2191) [ClassicSimilarity], result of:
           0.2360828 = score(doc=2191,freq=2.0), product of:
             0.35034057 = queryWeight, product of:
               2.863306 = boost
               7.62393 = idf(docFreq=58, maxDocs=44421)
               0.016048845 = queryNorm
             0.67386657 = fieldWeight in 2191, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.62393 = idf(docFreq=58, maxDocs=44421)
               0.0625 = fieldNorm(doc=2191)
         0.20432012 = weight(abstract_txt:distributed in 2191) [ClassicSimilarity], result of:
           0.20432012 = score(doc=2191,freq=2.0), product of:
             0.40086514 = queryWeight, product of:
               4.331484 = boost
               5.7665734 = idf(docFreq=377, maxDocs=44421)
               0.016048845 = queryNorm
             0.5096979 = fieldWeight in 2191, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.7665734 = idf(docFreq=377, maxDocs=44421)
               0.0625 = fieldNorm(doc=2191)
         0.09168497 = weight(abstract_txt:content in 2191) [ClassicSimilarity], result of:
           0.09168497 = score(doc=2191,freq=1.0), product of:
             0.35098013 = queryWeight, product of:
               5.232427 = boost
               4.1796083 = idf(docFreq=1847, maxDocs=44421)
               0.016048845 = queryNorm
             0.26122552 = fieldWeight in 2191, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1796083 = idf(docFreq=1847, maxDocs=44421)
               0.0625 = fieldNorm(doc=2191)
       0.28 = coord(7/25)
   

2. Nelson, M.L.; Harrison, T.L.; Rocker, J.A.: OAI and NASA's scientific and technical information (2003) 0.15

   0.15168245 = sum of:
     0.15168245 = product of:
       0.94801533 = sum of:
         0.13065533 = weight(abstract_txt:protocol in 4340) [ClassicSimilarity], result of:
           0.13065533 = score(doc=4340,freq=2.0), product of:
             0.17778504 = queryWeight, product of:
               1.6654228 = boost
               6.651612 = idf(docFreq=155, maxDocs=44421)
               0.016048845 = queryNorm
             0.7349062 = fieldWeight in 4340, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.651612 = idf(docFreq=155, maxDocs=44421)
               0.078125 = fieldNorm(doc=4340)
         0.09474316 = weight(abstract_txt:metadata in 4340) [ClassicSimilarity], result of:
           0.09474316 = score(doc=4340,freq=3.0), product of:
             0.14349686 = queryWeight, product of:
               1.8324989 = boost
               4.87927 = idf(docFreq=917, maxDocs=44421)
               0.016048845 = queryNorm
             0.66024554 = fieldWeight in 4340, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.87927 = idf(docFreq=917, maxDocs=44421)
               0.078125 = fieldNorm(doc=4340)
         0.3614265 = weight(abstract_txt:harvesting in 4340) [ClassicSimilarity], result of:
           0.3614265 = score(doc=4340,freq=3.0), product of:
             0.35034057 = queryWeight, product of:
               2.863306 = boost
               7.62393 = idf(docFreq=58, maxDocs=44421)
               0.016048845 = queryNorm
             1.0316433 = fieldWeight in 4340, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.62393 = idf(docFreq=58, maxDocs=44421)
               0.078125 = fieldNorm(doc=4340)
         0.36119035 = weight(abstract_txt:distributed in 4340) [ClassicSimilarity], result of:
           0.36119035 = score(doc=4340,freq=4.0), product of:
             0.40086514 = queryWeight, product of:
               4.331484 = boost
               5.7665734 = idf(docFreq=377, maxDocs=44421)
               0.016048845 = queryNorm
             0.9010271 = fieldWeight in 4340, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.7665734 = idf(docFreq=377, maxDocs=44421)
               0.078125 = fieldNorm(doc=4340)
       0.16 = coord(4/25)
   

3. Fu, T.; Abbasi, A.; Chen, H.: ¬A focused crawler for Dark Web forums (2010) 0.15

   0.14734519 = sum of:
     0.14734519 = product of:
       0.6139383 = sum of:
         0.085897654 = weight(abstract_txt:periodic in 458) [ClassicSimilarity], result of:
           0.085897654 = score(doc=458,freq=1.0), product of:
             0.15598087 = queryWeight, product of:
               1.1030561 = boost
               8.811096 = idf(docFreq=17, maxDocs=44421)
               0.016048845 = queryNorm
             0.5506935 = fieldWeight in 458, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.811096 = idf(docFreq=17, maxDocs=44421)
               0.0625 = fieldNorm(doc=458)
         0.016199145 = weight(abstract_txt:based in 458) [ClassicSimilarity], result of:
           0.016199145 = score(doc=458,freq=1.0), product of:
             0.08142639 = queryWeight, product of:
               1.5939493 = boost
               3.1830752 = idf(docFreq=5005, maxDocs=44421)
               0.016048845 = queryNorm
             0.1989422 = fieldWeight in 458, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.1830752 = idf(docFreq=5005, maxDocs=44421)
               0.0625 = fieldNorm(doc=458)
         0.04555387 = weight(abstract_txt:approach in 458) [ClassicSimilarity], result of:
           0.04555387 = score(doc=458,freq=3.0), product of:
             0.11248127 = queryWeight, product of:
               1.8734065 = boost
               3.741144 = idf(docFreq=2864, maxDocs=44421)
               0.016048845 = queryNorm
             0.4049907 = fieldWeight in 458, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.741144 = idf(docFreq=2864, maxDocs=44421)
               0.0625 = fieldNorm(doc=458)
         0.21378832 = weight(abstract_txt:crawling in 458) [ClassicSimilarity], result of:
           0.21378832 = score(doc=458,freq=2.0), product of:
             0.286466 = queryWeight, product of:
               2.1140413 = boost
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.016048845 = queryNorm
             0.7462956 = fieldWeight in 458, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.0625 = fieldNorm(doc=458)
         0.047485456 = weight(abstract_txt:retrieval in 458) [ClassicSimilarity], result of:
           0.047485456 = score(doc=458,freq=1.0), product of:
             0.21854374 = queryWeight, product of:
               3.916989 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.016048845 = queryNorm
             0.21728125 = fieldWeight in 458, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0625 = fieldNorm(doc=458)
         0.20501381 = weight(abstract_txt:content in 458) [ClassicSimilarity], result of:
           0.20501381 = score(doc=458,freq=5.0), product of:
             0.35098013 = queryWeight, product of:
               5.232427 = boost
               4.1796083 = idf(docFreq=1847, maxDocs=44421)
               0.016048845 = queryNorm
             0.584118 = fieldWeight in 458, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.1796083 = idf(docFreq=1847, maxDocs=44421)
               0.0625 = fieldNorm(doc=458)
       0.24 = coord(6/25)
   

4. Van de Sompel, H.; Nelson, M.L.; Lagoze, C.; Warner, S.: Resource harvesting within the OAI-PMH framework (2004) 0.14

   0.13983968 = sum of:
     0.13983968 = product of:
       0.69919837 = sum of:
         0.0634786 = weight(abstract_txt:resources in 5110) [ClassicSimilarity], result of:
           0.0634786 = score(doc=5110,freq=5.0), product of:
             0.071690425 = queryWeight, product of:
               1.057566 = boost
               4.2238636 = idf(docFreq=1767, maxDocs=44421)
               0.016048845 = queryNorm
             0.8854543 = fieldWeight in 5110, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.2238636 = idf(docFreq=1767, maxDocs=44421)
               0.09375 = fieldNorm(doc=5110)
         0.11086473 = weight(abstract_txt:protocol in 5110) [ClassicSimilarity], result of:
           0.11086473 = score(doc=5110,freq=1.0), product of:
             0.17778504 = queryWeight, product of:
               1.6654228 = boost
               6.651612 = idf(docFreq=155, maxDocs=44421)
               0.016048845 = queryNorm
             0.6235886 = fieldWeight in 5110, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.651612 = idf(docFreq=155, maxDocs=44421)
               0.09375 = fieldNorm(doc=5110)
         0.13127998 = weight(abstract_txt:metadata in 5110) [ClassicSimilarity], result of:
           0.13127998 = score(doc=5110,freq=4.0), product of:
             0.14349686 = queryWeight, product of:
               1.8324989 = boost
               4.87927 = idf(docFreq=917, maxDocs=44421)
               0.016048845 = queryNorm
             0.9148631 = fieldWeight in 5110, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.87927 = idf(docFreq=917, maxDocs=44421)
               0.09375 = fieldNorm(doc=5110)
         0.03945081 = weight(abstract_txt:approach in 5110) [ClassicSimilarity], result of:
           0.03945081 = score(doc=5110,freq=1.0), product of:
             0.11248127 = queryWeight, product of:
               1.8734065 = boost
               3.741144 = idf(docFreq=2864, maxDocs=44421)
               0.016048845 = queryNorm
             0.35073224 = fieldWeight in 5110, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.741144 = idf(docFreq=2864, maxDocs=44421)
               0.09375 = fieldNorm(doc=5110)
         0.35412422 = weight(abstract_txt:harvesting in 5110) [ClassicSimilarity], result of:
           0.35412422 = score(doc=5110,freq=2.0), product of:
             0.35034057 = queryWeight, product of:
               2.863306 = boost
               7.62393 = idf(docFreq=58, maxDocs=44421)
               0.016048845 = queryNorm
             1.0107999 = fieldWeight in 5110, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.62393 = idf(docFreq=58, maxDocs=44421)
               0.09375 = fieldNorm(doc=5110)
       0.2 = coord(5/25)
   

5. Suleman, H.; Fox, E.A.: Leveraging OAI harvesting to disseminate theses (2003) 0.14

   0.13954698 = sum of:
     0.13954698 = product of:
       0.69773483 = sum of:
         0.02024893 = weight(abstract_txt:based in 5779) [ClassicSimilarity], result of:
           0.02024893 = score(doc=5779,freq=1.0), product of:
             0.08142639 = queryWeight, product of:
               1.5939493 = boost
               3.1830752 = idf(docFreq=5005, maxDocs=44421)
               0.016048845 = queryNorm
             0.24867775 = fieldWeight in 5779, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.1830752 = idf(docFreq=5005, maxDocs=44421)
               0.078125 = fieldNorm(doc=5779)
         0.092387274 = weight(abstract_txt:protocol in 5779) [ClassicSimilarity], result of:
           0.092387274 = score(doc=5779,freq=1.0), product of:
             0.17778504 = queryWeight, product of:
               1.6654228 = boost
               6.651612 = idf(docFreq=155, maxDocs=44421)
               0.016048845 = queryNorm
             0.5196572 = fieldWeight in 5779, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.651612 = idf(docFreq=155, maxDocs=44421)
               0.078125 = fieldNorm(doc=5779)
         0.10939999 = weight(abstract_txt:metadata in 5779) [ClassicSimilarity], result of:
           0.10939999 = score(doc=5779,freq=4.0), product of:
             0.14349686 = queryWeight, product of:
               1.8324989 = boost
               4.87927 = idf(docFreq=917, maxDocs=44421)
               0.016048845 = queryNorm
             0.76238596 = fieldWeight in 5779, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.87927 = idf(docFreq=917, maxDocs=44421)
               0.078125 = fieldNorm(doc=5779)
         0.2951035 = weight(abstract_txt:harvesting in 5779) [ClassicSimilarity], result of:
           0.2951035 = score(doc=5779,freq=2.0), product of:
             0.35034057 = queryWeight, product of:
               2.863306 = boost
               7.62393 = idf(docFreq=58, maxDocs=44421)
               0.016048845 = queryNorm
             0.8423332 = fieldWeight in 5779, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.62393 = idf(docFreq=58, maxDocs=44421)
               0.078125 = fieldNorm(doc=5779)
         0.18059517 = weight(abstract_txt:distributed in 5779) [ClassicSimilarity], result of:
           0.18059517 = score(doc=5779,freq=1.0), product of:
             0.40086514 = queryWeight, product of:
               4.331484 = boost
               5.7665734 = idf(docFreq=377, maxDocs=44421)
               0.016048845 = queryNorm
             0.45051354 = fieldWeight in 5779, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7665734 = idf(docFreq=377, maxDocs=44421)
               0.078125 = fieldNorm(doc=5779)
       0.2 = coord(5/25)