Document (#28090)

Author

Cothey, V.

Title

Web-crawling reliability

Source

Journal of the American Society for Information Science and Technology. 55(2004) no.14, S.1228-1238

Year

2004

Abstract

In this article, I investigate the reliability, in the social science sense, of collecting informetric data about the World Wide Web by Web crawling. The investigation includes a critical examination of the practice of Web crawling and contrasts the results of content crawling with the results of link crawling. It is shown that Web crawling by search engines is intentionally biased and selective. I also report the results of a [arge-scale experimental simulation of Web crawling that illustrates the effects of different crawling policies an data collection. It is concluded that the reliability of Web crawling as a data collection technique is improved by fuller reporting of relevant crawling policies.

Footnote

Beitrag in einem Themenheft über Webometrics

Theme

Internet
Informetrie

Object

WWW

Similar documents (content)

1. Bidoki, A.M.Z.; Yazdani, N.: an intelligent ranking algorithm for web pages : DistanceRank (2008) 0.15

   0.14512129 = sum of:
     0.14512129 = product of:
       0.90700805 = sum of:
         0.018988544 = weight(abstract_txt:link in 3068) [ClassicSimilarity], result of:
           0.018988544 = score(doc=3068,freq=1.0), product of:
             0.042584863 = queryWeight, product of:
               1.0210208 = boost
               5.707506 = idf(docFreq=400, maxDocs=44421)
               0.007307592 = queryNorm
             0.44589892 = fieldWeight in 3068, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.707506 = idf(docFreq=400, maxDocs=44421)
               0.078125 = fieldNorm(doc=3068)
         0.005731213 = weight(abstract_txt:that in 3068) [ClassicSimilarity], result of:
           0.005731213 = score(doc=3068,freq=2.0), product of:
             0.021934222 = queryWeight, product of:
               1.269196 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.007307592 = queryNorm
             0.2612909 = fieldWeight in 3068, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.078125 = fieldNorm(doc=3068)
         0.012897464 = weight(abstract_txt:results in 3068) [ClassicSimilarity], result of:
           0.012897464 = score(doc=3068,freq=1.0), product of:
             0.04745742 = queryWeight, product of:
               1.8668939 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.007307592 = queryNorm
             0.2717692 = fieldWeight in 3068, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.078125 = fieldNorm(doc=3068)
         0.86939085 = weight(abstract_txt:crawling in 3068) [ClassicSimilarity], result of:
           0.86939085 = score(doc=3068,freq=2.0), product of:
             0.93195343 = queryWeight, product of:
               15.10442 = boost
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.007307592 = queryNorm
             0.93286943 = fieldWeight in 3068, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.078125 = fieldNorm(doc=3068)
       0.16 = coord(4/25)
   

2. Alqaraleh, S.; Ramadan, O.; Salamah, M.: Efficient watcher based web crawler design (2015) 0.12

   0.12074707 = sum of:
     0.12074707 = product of:
       1.0062256 = sum of:
         0.016137885 = weight(abstract_txt:sense in 2627) [ClassicSimilarity], result of:
           0.016137885 = score(doc=2627,freq=1.0), product of:
             0.04433689 = queryWeight, product of:
               1.0418124 = boost
               5.823732 = idf(docFreq=356, maxDocs=44421)
               0.007307592 = queryNorm
             0.36398324 = fieldWeight in 2627, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.823732 = idf(docFreq=356, maxDocs=44421)
               0.0625 = fieldNorm(doc=2627)
         0.0064841267 = weight(abstract_txt:that in 2627) [ClassicSimilarity], result of:
           0.0064841267 = score(doc=2627,freq=4.0), product of:
             0.021934222 = queryWeight, product of:
               1.269196 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.007307592 = queryNorm
             0.2956169 = fieldWeight in 2627, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=2627)
         0.98360354 = weight(abstract_txt:crawling in 2627) [ClassicSimilarity], result of:
           0.98360354 = score(doc=2627,freq=4.0), product of:
             0.93195343 = queryWeight, product of:
               15.10442 = boost
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.007307592 = queryNorm
             1.0554214 = fieldWeight in 2627, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.0625 = fieldNorm(doc=2627)
       0.12 = coord(3/25)
   

3. Fu, T.; Abbasi, A.; Chen, H.: ¬A focused crawler for Dark Web forums (2010) 0.12

   0.11774736 = sum of:
     0.11774736 = product of:
       0.735921 = sum of:
         0.014899493 = weight(abstract_txt:improved in 458) [ClassicSimilarity], result of:
           0.014899493 = score(doc=458,freq=1.0), product of:
             0.042038623 = queryWeight, product of:
               1.0144513 = boost
               5.6707826 = idf(docFreq=415, maxDocs=44421)
               0.007307592 = queryNorm
             0.3544239 = fieldWeight in 458, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.6707826 = idf(docFreq=415, maxDocs=44421)
               0.0625 = fieldNorm(doc=458)
         0.015190835 = weight(abstract_txt:link in 458) [ClassicSimilarity], result of:
           0.015190835 = score(doc=458,freq=1.0), product of:
             0.042584863 = queryWeight, product of:
               1.0210208 = boost
               5.707506 = idf(docFreq=400, maxDocs=44421)
               0.007307592 = queryNorm
             0.35671914 = fieldWeight in 458, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.707506 = idf(docFreq=400, maxDocs=44421)
               0.0625 = fieldNorm(doc=458)
         0.010317971 = weight(abstract_txt:results in 458) [ClassicSimilarity], result of:
           0.010317971 = score(doc=458,freq=1.0), product of:
             0.04745742 = queryWeight, product of:
               1.8668939 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.007307592 = queryNorm
             0.21741535 = fieldWeight in 458, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.0625 = fieldNorm(doc=458)
         0.6955127 = weight(abstract_txt:crawling in 458) [ClassicSimilarity], result of:
           0.6955127 = score(doc=458,freq=2.0), product of:
             0.93195343 = queryWeight, product of:
               15.10442 = boost
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.007307592 = queryNorm
             0.7462956 = fieldWeight in 458, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.0625 = fieldNorm(doc=458)
       0.16 = coord(4/25)
   

4. Marcondes, C.H.; Costa, L.C da.: ¬A model to represent and process scientific knowledge in biomedical articles with semantic Web technologies (2016) 0.11

   0.11154395 = sum of:
     0.11154395 = product of:
       0.55771977 = sum of:
         0.00458497 = weight(abstract_txt:that in 3829) [ClassicSimilarity], result of:
           0.00458497 = score(doc=3829,freq=2.0), product of:
             0.021934222 = queryWeight, product of:
               1.269196 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.007307592 = queryNorm
             0.20903271 = fieldWeight in 3829, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=3829)
         0.030659124 = weight(abstract_txt:reporting in 3829) [ClassicSimilarity], result of:
           0.030659124 = score(doc=3829,freq=1.0), product of:
             0.06801019 = queryWeight, product of:
               1.2903097 = boost
               7.212831 = idf(docFreq=88, maxDocs=44421)
               0.007307592 = queryNorm
             0.45080194 = fieldWeight in 3829, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.212831 = idf(docFreq=88, maxDocs=44421)
               0.0625 = fieldNorm(doc=3829)
         0.012802645 = weight(abstract_txt:data in 3829) [ClassicSimilarity], result of:
           0.012802645 = score(doc=3829,freq=2.0), product of:
             0.04349417 = queryWeight, product of:
               1.7872409 = boost
               3.3302255 = idf(docFreq=4320, maxDocs=44421)
               0.007307592 = queryNorm
             0.29435313 = fieldWeight in 3829, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.3302255 = idf(docFreq=4320, maxDocs=44421)
               0.0625 = fieldNorm(doc=3829)
         0.017871251 = weight(abstract_txt:results in 3829) [ClassicSimilarity], result of:
           0.017871251 = score(doc=3829,freq=3.0), product of:
             0.04745742 = queryWeight, product of:
               1.8668939 = boost
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.007307592 = queryNorm
             0.37657443 = fieldWeight in 3829, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.4786456 = idf(docFreq=3724, maxDocs=44421)
               0.0625 = fieldNorm(doc=3829)
         0.49180177 = weight(abstract_txt:crawling in 3829) [ClassicSimilarity], result of:
           0.49180177 = score(doc=3829,freq=1.0), product of:
             0.93195343 = queryWeight, product of:
               15.10442 = boost
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.007307592 = queryNorm
             0.5277107 = fieldWeight in 3829, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.0625 = fieldNorm(doc=3829)
       0.2 = coord(5/25)
   

5. Menczer, F.: Lexical and semantic clustering by Web links (2004) 0.11

   0.11039468 = sum of:
     0.11039468 = product of:
       0.9199557 = sum of:
         0.042459674 = weight(abstract_txt:link in 4090) [ClassicSimilarity], result of:
           0.042459674 = score(doc=4090,freq=5.0), product of:
             0.042584863 = queryWeight, product of:
               1.0210208 = boost
               5.707506 = idf(docFreq=400, maxDocs=44421)
               0.007307592 = queryNorm
             0.9970603 = fieldWeight in 4090, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               5.707506 = idf(docFreq=400, maxDocs=44421)
               0.078125 = fieldNorm(doc=4090)
         0.008105158 = weight(abstract_txt:that in 4090) [ClassicSimilarity], result of:
           0.008105158 = score(doc=4090,freq=4.0), product of:
             0.021934222 = queryWeight, product of:
               1.269196 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.007307592 = queryNorm
             0.3695211 = fieldWeight in 4090, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.078125 = fieldNorm(doc=4090)
         0.86939085 = weight(abstract_txt:crawling in 4090) [ClassicSimilarity], result of:
           0.86939085 = score(doc=4090,freq=2.0), product of:
             0.93195343 = queryWeight, product of:
               15.10442 = boost
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.007307592 = queryNorm
             0.93286943 = fieldWeight in 4090, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.443371 = idf(docFreq=25, maxDocs=44421)
               0.078125 = fieldNorm(doc=4090)
       0.12 = coord(3/25)